Configuring Service Monitoring

Please read about the latest changes and upgrade.

Please note that your newly registered account needs to be manually activated by an admin. Just be patient, you cannot post to any forum before this happens. This is thanks to the unfortunate event of spammers recently. More details here.

  • Apologize for the venting, but one of the biggest short comings of Director is the lack of documentation. I'm finding that many of the older blog posts are out of date for the most recent version. The help on github is also either out of date or glosses over points, leaving the reader unclear.


    I have host notifications working (up and down), which took substantial searching and reading of blog posts and many hours to get working correctly. Now I'm trying to setup service monitoring and am also not clear how to proceed. I have a freshly configuring instance of Icinga2/IcingaWeb/Director. A couple questions:


    1. Can someone outline the high level steps for setting up service monitoring. (something like create service templates, apply to host or host template?)


    2. Do I have to create a template for each individual service, for example a template for ping4, a template for ssh, a template for host alive etc? I tried creating a default service template but Director seems to require that I include some check command in this default template?


    3. Do I need to create a generic/default service template that other service templates need to inherit from?


    4. Once Service templates are created should services be added to the host template or directly to hosts?


    5. When creating apply rules for service templates Do I have to specify both "Apply For" and "Assign Where?"


    Any help with these questions would be greatly appreciated. One of my biggest fears is that I'm going to have a poorly configured icinga2/Director instance as the foundation of my monitoring project.

  • Although I would agree that the Director documentation is sparse, this is very understandable because Director is still in it's early stages (Still only Version 1.2.0 as of today!). Knowing this, you cannot expect to have detailed, in-depth documentation, because Director's features are constantly changing. Another thing to note is that the Icinga developers are not native English speakers, so cut them some slack!


    A lot of learning Director comes from experimenting with its tools and finding out what works and what doesn't work and if they don't work, finding out how to fix it. Searching through the forums or the internet usually yields results for your problems and if they don't, simply create a thread.


    Now, on to your problems.


    By having the host notifications up, I don't see how it is much different for getting the services notifications up.


    1. Service templates work very similarly to Host templates. Like you said, you have to 1) create a service template, 2) create a service based on the template, 3) apply the service to a host. That is the high level explanation. Getting into the nitty-gritty is something you have to experiment with to learn how to ask more in-depth questions.


    2. Creating templates is all based on how you want Director to work for you. In my set up, I only have one service template that I use because I am only executing simple checks and do not need any other special attributes for my services. For example, you may want to create two separate service templates if you were to run a service on a host with an agent installed on it and a host without an agent installed on it.


    Templates only exist for you to speed up the process of creating more services. Every time you create a new service, you can choose to import that template to use the same configurations from the template so you can save a few clicks.


    3. As stated in the previous point, I only have one default service template because my checks are simply and do not need special attributes. This depends on your set up entirely.


    4. Services can be added to hosts directly, but you should learn how to use apply rules. There are many posts about apply rules on the forums. Apply rules allow you to deploy services on many hosts based on their attributes. Apply rules (in my experience) is synonymous with Assign where (because once you implement the apply rule, it ends up saying "assign where" in the preview). For example, say I create a service to check the status of the SSH service on a host. Well, since SSH exists only on Linux-based systems, I wouldn't want to use that check on every host in my environment (since I have Windows-based systems as well). I create an apply rule so that the service only executes on Linux systems (in my environment, I created host groups to help with this. If you want a brief explanation of that, I can oblige).


    5. "Apply for" is different that "Assign where". I do believe that the language here is somewhat confusing, something that should be changed in the future. You will only use the Apply For rule for if you have hosts with custom attributes (something you shouldn't be worrying about just yet). Instead, focus on the "Assign where" option. That is what I described in the previous point.


    Knowing this basic information should be enough to get you started with assigning services. If you'd like more explanation, let me know. The more specific your questions get, the better.

  • I tried following the instructions here, but they seem to be incomplete or Director has changed since they were written:


    https://github.com/Icinga/icin…24-Working-with-agents.md


    I want to use the standard icinga Agent based checks. I followed the instructions above but there appear to be lots of details missing.


    1. As instructed, I created an "Icinga Agent" host template.
    2. The host template requires a "Check command". This isn't mentioned in the documentation linked above and this is where I suspect I'm doing something wrong. Should I use "dummy" for the check command as would happen with a non Director icinga2 configuration using the agent?
    3. Inevitably after running the agent setup shell script on the monitored host and configuring (or trying to configure) icinga agent based checks, my services show as Pending with no change.

  • I would not create a template with only the agent set to yes.
    Templates are great if you have services that use the same checks, for different things, like service checks or wmi queries.


    @watermelon is right with his points, right now Director is in its early stages and documentation is not a priority.


    I plan to write a complete tutorial on how to use Icinga2, Director, influxdb, grafana, once I get approval for building the production system.

  • Although I would agree that the Director documentation is sparse, this is very understandable because Director is still in it's early stages (Still only Version 1.2.0 as of today!). Knowing this, you cannot expect to have detailed, in-depth documentation, because Director's features are constantly changing. Another thing to note is that the Icinga developers are not native English speakers, so cut them some slack!


    A lot of learning Director comes from experimenting with its tools and finding out what works and what doesn't work and if they don't work, finding out how to fix it. Searching through the forums or the internet usually yields results for your problems and if they don't, simply create a thread.


    Full ACK. While we agree on writing every little documentation piece in English, best practices and things you design and then use in production evolve over time.


    We encourage everyone to share their experiences, and to get started with their documentation contributions. There's hints over at https://www.icinga.com/community/get-involved/ as well as Markdown generally is not so hard to learn. The thing is rather to find the correct words, descriptions, headers and always care to explain it from a users point of view.


    Also keep in mind that developers might not know about problems with understanding things as they are used to their tools for months or years. Community members and contributors are literally the key to help them and help out to make the documentation great - for beginners and also for advanced users.


    TL;DR - once you've found your solution with the help of community members, please think about contributing documentation enhancements. That'll help others in the future, and maybe even you if you come back after a while.

  • I've had to set Director aside for now. I had intended to use it for a new project in a production environment, but it doesn't appear ready for that yet, or at least I don't have the time to invest in repeated trial and error. I'll leave my Director instance up and contribute where I can to the documentation once I figure things out. I'd be happy with even some high level guidance, not even "every little documentation piece". I'll try to help contribute to that.

  • I spent more time trying to configure services and monitoring today. Perhaps someone can correct me, but it seems that if you use Director, you lose the default monitoring built into Icinga2 and the Icinga agent (disk, http, icinga,load, ping4, ping6, ssh, etc, see the attached screenshot). If true, and these all have to be manually recreated with templates in Director, that would be unfortunate. Hopefully someone can tell me I'm wrong?

  • you are not entirely wrong. if you configure your checks by hand there are some default ones in conf.d/ that apply if they get the wanted parameters.
    Usually I delete those as I do not want them in my Director setup.

  • Those objects inside conf.d/ are meant to be examples. When it comes to your own tools which either render configuration files (Puppet, Ansible, etc.) or use the config package API (Director) I (and the docs) strongly suggest to use only one way. Inside the newly written Puppet module it is for instance using objects.d/ AFAIK.


    No matter what, the key to maintain Icinga 2 also is to learn how the configuration works. That'll help especially when you'll get a validation error, and that may happen even with the Icinga Director (e.g. missing host check_command).

  • I have the same 'problem' as Nothing2Crazy. I would like to use the service templates from the Director. Can someone share his configuration (files) so we not all have to search for examples how to write these scripts?

  • Imho you should've read and understood the monitoring basics chapter inside the Icinga 2 docs.
    https://docs.icinga.com/icinga…chapter/monitoring-basics


    Those questions are somewhat generic and I am answering them independent from any tool here.


    1. Can someone outline the high level steps for setting up service monitoring. (something like create service templates, apply to host or host template?)


    It is best practice to hide several object attributes into templates. That could be for instance the check_interval, retry_interval or max_check_attempts. Or a specific "notes" field.


    The example config underneath conf.d/ toys with that somehow already. You'll have the "generic-service" and "generic-host" templates. If you're going further, you might have different types of your monitoring hosts and services. One for "basic" checks, then "database" related, some could be "snmp", others "environmental".



    2. Do I have to create a template for each individual service, for example a template for ping4, a template for ssh, a template for host alive etc? I tried creating a default service template but Director seems to require that I include some check command in this default template?


    I would identify the same defaults and use only as much as needed. By grouping such services, it certainly will reduce the amount of templates created. But still you could go for a specific template for each service, although that doesn't make much sense - unless you have multiple service apply rules for each service, e.g. 10x ping4 apply with different assign where expression.



    3. Do I need to create a generic/default service template that other service templates need to inherit from?


    It's possible and also recommended to chain such, if you have for example specific requirements. Example: "generic-host" including the check_command, then the templates for "prod-host" and "dev-host". Both import the "generic-host" template and also set the custom attribute "production", one to true, the other to false.


    Then you'll define your host objects which import the relevant templates then. DEV boxes import for example the "dev-host" template then.


    Although, if you prefer just one level of templates, that'll also work. It is generally up to you, try it out in a test lab.



    4. Once Service templates are created should services be added to the host template or directly to hosts?


    That's probably a Director related question, but in case you are looking to solve this with Icinga 2 itself, there are common practices to use "assign where "dev-host" in host.templates" in your service apply rule. That could for instance create services which are always required (basically a service set as default). One example for a dev host - monitor cpu and memory if something is going crazy and send notifications to devs.


    On the other hand, if there are specific services - e.g. a mysql database check with a specific SQL query - those should probably be applied based on a group or custom attribute being set. Or by matching the hostname.



    5. When creating apply rules for service templates Do I have to specify both "Apply For" and "Assign Where?"


    You should have a read about "apply for" beforehand.


    https://docs.icinga.com/icinga…ng-basics#using-apply-for


    Then you'll recognise that apply for expects a host array or dictionary and does not require the assign where expression being specified. If the host does not provide such a value type, it will be silently ignored and no services are generated.


    If you prefer not to depend on the host to specify certain service attributes, and just want to apply a ping4 check, don't use "apply for". Just assign the service to a matching pattern. Common best practice would be to assign "ping4" to all hosts with the "address" attribute thus allowing exceptions e.g. when the custom attribute "no_ping" is set to false.



    After all I recommend to know about the Icinga 2 configuration objects and their attributes. It eases the way you're dealing with a configuration frontend. I've seen quite a lot of users accidentally enabling "volatile" on their services and then wondering why they get notified on every state change, even SOFT. That could easily be avoided by checking the docs for the attribute the first time you use it.