There is a Ganglia aggregator running:
And three Ganglia agents working in unicast mode, to publish data to the aggregator.
From one of the Ganglia agents, I manually push new metrics using gmetric CLI utility, and I see it plotted on the Web dashboard connected to the aggregator.
If new metrics can be published in this way, then what is the use of "gmond collection group" and those metrics definitions in gmond.conf?
There is an important difference between the new metrics pushed by gmetric and gmond collection group metrics: the collection group metrics are collected by gmond itself by the output data of "modules".
Gmond reads the configuration file and loads all "modules" as specified in the configuration, each module periodically publishes latest metrics data. For example, load_module.so publishes load average data.
Then the "collection_group" section tells gmond to gather the metrics data from modules, at regular intervals.
Therefore, it is not necessary to define new metrics in gmond unless the new metrics are loaded into gmond.