Custom topology

WombatOAM automatically generates the node names and node family names that are used in the dashboard. Family names are derived from the release name of the node monitored by WombatOAM, while the node names are simply the name of the Erlang node. Nodes for which the same family name is generated are listed together.

The release name and node name do not always logically identify where a given node belongs. For example, monitored nodes with the same version in multiple Riak clusters will belong to the same family, instead of being grouped in the clusters to which they belong. If a different naming and grouping arrangement would be more logical for the system you are monitoring, you can override the default naming in WombatOAM, using the methods described below.

To override WombatOAM's default naming, you have to specify the relevant options in the configuration, create a callback module, and optionally use a script for upgrading the current node structure. These steps are described in detail below.

Configuration options

The wombat.config file has the following configuration options for how nodes and node families are named in WombatOAM:

{set, wo_core, additional_paths, ["path/to/alias_callback"]}.

The additional_paths option specifies the location of the callback module used for naming nodes and node families. The path should end in a directory.

{set, wo_core, alias_callback, callback_module_name}.

The alias_callback option specifies the name of the callback module. This module should be placed in a directory specified by the additional_paths option. The callback module makes it possible to implement fairly complex logic to grouping nodes into families and naming them as required. Naming the nodes and families is performed separately, so the names cannot depend on each other.

{set, wo_core, node_family_rules, [Rule]}.

Where

Rule :: {FamilyName, [Condition]}.
FamilyName :: atom()|string().
Condition :: {application, loaded|loading|started|starting|running|start_p_false, App}
           | {application, isnot, loaded|loading|started|starting|running|start_p_false, App}
           | {environment, App, EnvVar, set|not_set}
           | {environment, App, EnvVar, equals, Value}.
App :: atom().
EnvVar :: atom().
Value :: term().

If specified and alias_callback is not defined or does not apply to a given node, then the family name is first tried to be derived based on the list of Rules in order. If no rule applies, then the default family name is used. The rules are applied in the order they are specified in the config and the first matching rule will be used. The list of Condition tuples define what conditions must apply to the node in order to belong to the given family FamilyName. If all conditions apply, then the node belongs to the family FamilyName. For example, the condition {application, loaded, mnesia} specifies that an application named mnesia is loaded on the node, while {environment, my_app, my_var, not_set} states that the application named my_app does not have the environment variable my_var set. To make sure that this environment variable is set to a specific value, for example, 5 the condition {environment, my_app, my_var, equals, 5} can be given. Rules in the form {application, isnot, State, App} are to be interpreted as the given application is not in the specified state on the node. Each rule can contain any number of conditions and by definition any node matches the empty condition list. Defining either-or conditions is not supported, but they can be specified by duplication. For example

{set, wo_core, node_family_rules, [
  {"Test nodes", [{application, running, test},
                 {environment, test, test_type, equals, unit}]},
  {"Test nodes", [{application, running, test},
                 {environment, test, test_type, equals, feature}]}]}.

which describes that nodes that have the test application running on them with the environment variable test_type set to either unit or feature belong to the node family "Test nodes".

{set, wo_core, automatic_node_restructure_at_startup, true}.

The automatic_node_restructure_at_startup option specifies whether the nodes should be restructured at startup according to the rules in the callback module. If the rules do not change and the callback functions run without any side effects, then no restructuring occurs. Existing nodes' UUIDs are never changed. If the restructuring is set to true, then it is advisable to not disable the node_info plugin, otherwise the node will be excluded from the node family at startup.

Alias callback interface

The alias callback module has to implement and export any of the following functions (a missing implementation will be substituted by the default implementation):

-spec generate_custom_family_name(Node :: atom(),
                                  ScriptId :: {string(), string()}
                                            | {error, term()}) ->
          {ok, string()} | {error, term()}.

Used for naming node families. The parameters are (1) the name of the node, and (2) the release name (either the name and version of the permanent release or the script_id - see the documentation of the release_handler and init:script_id/0). If an error occurs when capturing the release name, the following term is returned: {error, no_scriptid}. If the callback function is not implemented or exported, or if an error (crash) occurs in the function, the default naming logic is used instead. The callback should be free of side effects.

-spec generate_custom_node_name(node()) -> {ok, string()} | {error, term()}.

Used for naming the nodes. The parameter is the name of the node. If the callback function is not implemented or exported, or if an error (crash) occurs within the function, the default naming logic is used instead. The function should be free of side effects.

Example module

-module(alias_callback).

%% API
-export([generate_custom_family_name/2,
         generate_custom_node_name/1]).

%%%=============================================================================
%%% External functions
%%%=============================================================================

%%------------------------------------------------------------------------------
%% @doc
%% Generate a name for a given node-family.
%%
%% See http://www.erlang.org/doc/man/init.html#script_id-0 for more information
%% about script ids.
%% @end
%%------------------------------------------------------------------------------
-spec generate_custom_family_name(Node :: atom(),
                                  ScriptId :: {string(), string()}
                                            | {error, term()}) ->
           {ok, string()} | {error, term()}.
generate_custom_family_name(Node, {Name, Version}) ->
    case atom_to_list(Node) of
        "riak" ++ _ ->
            {ok, "Riak cluster"};
        "rabbitmq" ++ _ ->
            {ok, "Rabbit cluster"};
        _ ->
            %% Default behaviour
            {ok, Name ++ " " ++ Version}
    end.

%%------------------------------------------------------------------------------
%% @doc
%% Generate a name for a given node.
%% @end
%%------------------------------------------------------------------------------
-spec generate_custom_node_name(node()) -> {ok, string()} | {error, term()}.
generate_custom_node_name(Node) ->
    {ok, atom_to_list(Node)}.

Adding a new node to WombatOAM

If the configuration settings are in place, and the callback module is compiled and located in the path, WombatOAM will load the module at startup.

All new nodes will be added according to the logic implemented in the callback module. Existing nodes will not be affected. To restructure existing nodes, follow the workflow in the next section.

Restructuring existing nodes

Workflow

1. Prepare the callback module and add it to the wombat.config. The callback module will be loaded into the WombatOAM VM. This requires the configuration options to be present at startup. For changes to the configuration file to take effect, WombatOAM must be restarted. Do NOT set the automatic_node_restructure_at_startup option before performing a successful upgrade if the old WombatOAM version is lower than 1.14.0.

2. Upgrade WombatOAM to the new version containing the feature. Configuration changes will take effect after a WombatOAM restart. You can modify the configuration before performing the WombatOAM upgrade. If the node has already been upgraded, restart WombatOAM to ensure the configuration options are loaded.

3. Run the node_restructure command-line script to review the proposed modifications. The command-line script provides a fast feedback loop to enable modifying the callback module to mirror the desired node structure. The script automatically reloads the callback module.

4. Enable restructuring in WombatOAM configuration. The restructuring is only done at WombatOAM startup if the corresponding configuration option is enabled. If it is disabled, the restructuring is not performed, but new nodes are added according to the specified callback module.

5. Restart WombatOAM. On-the-fly restructuring would impose problems in some WombatOAM subsystems due to the concurrent nature of events in WombatOAM. Restarting WombatOAM and performing the restructuring during the initial phase eliminates the risk of clashing cached data between various subsystems.

Step 4 and 5 is done by the provided script.

Command-line script to preview and apply restructuring

The node_restructure.escript script lets you check whether the restructuring works as intended, reload the callback module, and commit changes. To run the script, you need escript in your PATH. If you have modified the port of EPMD for WombatOAM, you have to set the shell variable ERL_EPMD_PORT accordingly. E.g. set it to port 1234 in case of bash:

export ERL_EPMD_PORT=1234

To preview the changes, run the following: escript node_restructure.escript preview

$ escript node_restructure.escript preview
"Family1":
    'node1@domain' : "Node 1"
    'node2@domain' : "Node 2"
"Family2":
    'node3@domain' : "Node 3"
    'node4@domain' : "Node 4"
    'node5@domain' : "Node 5"

Previewing has no effect on the WombatOAM node: no data is modified, no beam files are changed, and new nodes will keep using the callback module that is already loaded.

To reload the callback module, run the script with reload option:

$ escript node_restructure.escript reload
Reloaded.

Reloading the callback module performs a code change in the WombatOAM VM. Its effect is permanent: the new callback module will be used to name and group new nodes added to WombatOAM. Nodes that are already present are not changed. The script also shows the existing nodes in the new structure.

If you are satisfied with the result, set the above parameter in wombat.config and restart WombatOAM. The restructuring is done during the startup, after which WombatOAM starts normally. The configuration changes and restart procedure is automated with the commit option. Before the actual changes are made you will be prompted for confirmation.

$ escript node_restructure.escript commit
New nodes will be added according to new rules!
Existing nodes would look like the following after committing:
"Family 1":
    'node1@domain' : "Node 1"
    'node2@domain' : "Node 2"
    'node3@domain' : "Node 3"
"Family 2":
    'node4@domain' : "Node 4"
    'node5@domain' : "Node 5"

+-------------------------+
|WombatOAM will be RESTARTED!|
+-------------------------+
The following config option will be added to the `wombat.config':
   {set, wo_core, automatic_node_restructure_at_startup, true}.
Are you sure to proceed? (yes/no) yes

Stopping wombat... ok
Starting wombat... ok
Changes are committed.

Note: if the automatic_node_restructure_at_startup is explicitly set to false in wombat.config, it will not be modified but a warning will be printed.

Reverting your changes

After the commit, the changes cannot be undone, but removing the callback module configuration option and committing again would restore the structure to system defaults. Preview is also available with system defaults.