HA and plugins with tmp files

  • Hello,


    I recently installed a HA cluster. Only the master zone for now and two endpoints.


    Is there any way to deal with plugins for example snmp interface checks that stores information in temp files, the bandwidth calculation will be wrong if the check is scheduled on different nodes?


    I think it would be good to have some kind of api for plugin persistent storage (that in a cluster would be synced) if it doesn't exist already?


    Best Regards
    Magnus

  • Yes, you have different solutions,


    If you don't want to patch your plugins or you write them yourself,
    you can use an NFS share, or clustered filesystems like ocsfs2.


    If you write your plugins, or you can patch the default ones, you can store your persistence data
    in db like redis or mongodb for example.


    Defining an api in icinga for this would be partially pointless, because these are standard nagios plugins,
    and as this api would not be present in all nagios-compatible systems, the probability of it being used is low.

  • Hello,


    Thanks for the info.


    The HA feature is pretty crippled by this. I write most of the plugins myself so I will probably look into redis or something like that. Though that is another component that has to have HA.


    I still think this should be handled somehow by icinga. Perhaps the monitoring api should be updated so nagios/naemon and others start supporting this in a nicer way than having temp files. Also one of the monitoring solutions need to take lead and implement something and hopefully the rest will follow.


    My plugins are icinga only anyway since I use the icinga api to dynamically create services for devices depending on which features are enabled on them.


    Regards
    Magnus

  • Now that you say this, I've another idea that only uses the icinga api:
    Since you make your plugins yourself, you can store and retrieve the persistent data in the service object itself in a custom attribute.


    Something like:


    $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/objects/services/example.localdomain!myservice' -d '{ "attrs": { "vars.persistent_data" : { "key1": "value1" } } }'


    This way, the data will be replicated automatically to all the nodes who know this object.
    (Doc: https://docs.icinga.com/icinga…api-config-objects-modify)

  • Creating a new plugin api has been discussed for a long time. The main point is - who is willing to work on that. Just because core developers specify a new format does not necessarily mean that everyone agrees on that. Or is willing to rewrite all the plugins out there to support the new format. Imho - and that is purely my personal opinion - community members should get together and start working on such a proposal and specification best together with the monitoring plugins project. We as core developers may join for guidance then.


    Just to throw in some ideas - JSON support (which includes proper performance data formatting), thresholds with better range support, Inventory capabilities, libraries for many languagues.

  • Great ide I didn't think about that at all.


    Thanks a lot.

  • Creating a new plugin api has been discussed for a long time. The main point is - who is willing to work on that. Just because core developers specify a new format does not necessarily mean that everyone agrees on that. Or is willing to rewrite all the plugins out there to support the new format. Imho - and that is purely my personal opinion - community members should get together and start working on such a proposal and specification best together with the monitoring plugins project. We as core developers may join for guidance then.


    Just to throw in some ideas - JSON support (which includes proper performance data formatting), thresholds with better range support, Inventory capabilities, libraries for many languagues.

    And also a uniform api would be great.


    I monitor a lot of networking equipment for example checkpoint firewalls. If I for example through snmp detect that it is configured as a cluster I use the api to create a cluster service which uses a predefined check command for that.


    Previously in nagios I resorted to the ugly solution of creating configuration files in /etc/nagios3/conf.d/dynamic and issuing the RELOAD command to the command file which restarted nagios since they have implemented the reload command as a restart. When I read about the icinga api I really liked that and switched over to icinga2. I really like the better configuration syntax to.

  • I think I will go the redis route instead of storing the values as variables in the service objects.


    Redis was super easy to setup redundantly and also I am a little worried about the amount of config syncs icinga would get. Say I do 1000 service checks in 5 minutes that would mean 1000 config syncs.


    That aside I really liked the ide of having the information within icinga which I feel is where it should really be.


    Thanks anyway
    Magnus

  • actually i wrote 2 nagios plugins that use a mysql table for specific temporary data ( a snmp traffic script and a mysql logpos script in bash)
    Maybe i can upload these to github if someone needs them.