1

I'm trying to turn our Icinga spaghetti into something more manageable and less repetitive. So far features like templates, object inheritance, multiple inheritance, hostgroups, servicegroups, etc have been meeting my needs, but I'm stuck on one aspect.


TL;DR: How do I tell Icinga that if that host is a member of a particular hostgroup I want to add a new contact group to all member hosts and services, even services defined against a different hostgroup that the host is also a member of?


I have two orthogonal sets of hostgroups - one for the host type, which is associated with services and service dependencies associated with the host type, e.g. "PostgreSQL servers". Another set of groups identifies areas of responsibility for nodes and who needs to care about each node.

I need to apply contact specifications to services according to the hostgroups identifying areas of responsibility - and want to do so without having to override each service for each (host-type, area-of-responsibility) hostgroup pair.

So, say I define a hostgroup "postgresql_servers" and associated service "postgresql_connection":

define hostgroup {
        hostgroup_name          postgres_servers
        alias                   All PostgreSQL servers
}

define service {
        use                     some_service_template
        hostgroup_name          postgres_servers
        contact_groups          support_engineers_notifications
        service_description     POSTGRES_CONNECTION
        check_command           check_dummy!2!"Passive check failed"
}

and then I have a hostgroup "servers_for_bob" and a contact "bob", which aren't yet associated with each other:

define hostgroup {
        hostgroup_name          servers_for_bob,
        alias                   These are for Bob
}

define contact {
    contact_name                  bob
    alias                         bob: Bob B.
    use                           some_contact_template
}

define contactgroup {
    contactgroup_name             team_bob
    members                       bob
}

Now, it's easy to make a host a member of both hostgroups:

define host {
        use             some_host_template
        host_name       buildingthings.example.com
        hostgroups      servers_for_bob, postgres_servers
        alias           The first thing bob built
}

... and in the process it will get all the services defined against postgres_servers, like POSTGRES_CONNECTION. But service notifications will only go to support_engineers_notifications as defined on the base service POSTGRES_CONNECTION.

Now I want to notify team_bob when the host has issues or any of its services has a problem. Without redeclaring all those services.

For the host its self I can use merge rules in object inheritance in contact_groups e.g.

define host {
        use             some_host_template
        host_name       buildingthings.example.com
        hostgroups      servers_for_bob, postgres_servers
        alias           The first thing bob built
        contact_groups  +team_bob
}

and for more hosts, can use multiple host template inheritance to reduce repetition.

However, AFAIK that won't cause the services that were implicitly defined by making the host a member of postgres_servers send notifications to team_bob too.

Here's the topology:

+--------------------------------------+       +-----------+                                    
|                                      |       | Contact:  |                                    
|    Hostgroup:                        |       | Team Bob  |                                    
|    postgresql_servers               <--????--+           |                                    
|                                      | ^^^^  |           |                                    
|                                      | how?  +-----------+                                    
|                                      |                                                        
|   +----------------------------------------+                                                  
|   |                                  |     |                                                  
|   |                                  |     |                                                  
|   |   +-------------------------+    |     |                                                  
|   |   |-------------------------|    |     |                                                  
|   |   ||                       ||    |     |                                                  
|   |   || Host buildingthings   ||    |     |                                                  
|   |   || hostgroups:           ||    |     |                                                  
|   |   ||   postgres_servers,   ||    |     |                                                  
|   |   ||   servers_for_bob     ||    |     |                                                 
|   |   ||                       ||    |     |                                                  
|   |   ||                       ||    |     |                                                  
|   |   ||                       ||    |     |                                                  
|   |   ||                       ||    |     |                                                  
|   |   ||                       ||    |     |                                                  
|   |   |-------------------------|    |     |                                                  
|   |   +-------------------------+    |     |                                                  
|   |                                  |     |                                                  
+--------------------------------------+     |      +----------------------+                    
    |                                        |      |                      |                    
    |                                        |      | Service:             |                    
    |                                       <-------+ POSTGRES_CONNECTION  |                    
    |                                        |      |                      |                    
    |      Hostgroup                         |      |                      |                    
    |      servers_for_bob                   |      +----------------------+                    
    |                                    ^   |                                                  
    +----------------------------------------+      +---------+------------+                    
                                         |          | More services...     |                    
                                         +----------+                      |                    
                                                    +----------------------+                    

(thanks http://asciiflow.com/)

How do I tell Icinga that if that host is a member of the hostgroup servers_for_bob I want to add the contact bob to all member hosts and services, even those defined implicitly via hostgroup inheritance?

I've seen vague mentions about using host- and service- escalations to work around this, but haven't been able to figure out how.

It seems like this would be a common requirement, but I'm not sure where to go from here. Help?

Craig Ringer
  • 10,553
  • 9
  • 38
  • 59
  • Nagios is not as elastic as everybody would like it to be. We deal with such problems with automation tools that give us the power to make every service or host unique. Maybe instead doing it all by hand look at puppet declarations for nagios http://docs.puppetlabs.com/references/latest/type.html#nagiosservice – 3h4x Jul 10 '14 at 06:19
  • Unfortuntely it's a distributed environment with servers under only limited central control, with spread responsibility. There's zero possibility of central configuration management of the member nodes - it's like if New Relic said "To use our monitoring systems you have to put your servers under control of our puppetmaster". Er. No. – Craig Ringer Jul 10 '14 at 06:37

2 Answers2

1

While it would seem that this is piece of cake, apparently it is not :). You most likely have only 2 solutions:

1) Make sure that each server definition template does not end up with servers in more servergroups, resulting in the possibility to put the contacts in the server templates.

2) Use some automation tool to generate your config and leave it as fluffy as it can get (it will be simple in the tool). We have been using Puppet to handle nagios config and, while it generates a large amount of configuration, it's ok because the code that generated it is based on some pretty simplistic templates.

Florin Asăvoaie
  • 6,932
  • 22
  • 35
  • Pity. Don't suppose it's possible with another tool you're familiar with like Zabbix? I'm getting a bit sick of how simple things require complex workarounds with Icinga/Nagios. Maybe I should just bite the bullet and see how "special" the Icinga source code is... – Craig Ringer Jul 10 '14 at 06:40
1

You can do it with escalation. We use it to send SMS to our NOC team.

define serviceescalation {
    service_description *
    host_name first_host, second_host
    first_notification 4
    last_notification 10
    notification_interval 20
    contacts NOC
}
3h4x
  • 491
  • 4
  • 7
  • Ah, that's interesting. It might be possible to adapt that approach to my needs - wildcard service descriptions might be the key. – Craig Ringer Jul 10 '14 at 10:46