Hello and thank you for the answer,
Let me explain the logic behind the setup and please correct me if I am wrong.
Firstly, I decided not to perform check from the master and the reason is because each of every machine I am monitoring is a separate physical client ( web hosting), so if at some point a client needs more than one machine and a private network I will need to create satellites and create zones, so I decided to start from the beginning with zones, since I am using ansible it’s not much of an effort to create separate zone for each client and it’s not affecting the performance ( again everything I say is what I understood so far). However, as I understood so far, a satelite can monitor itself and the nodes in the zone, however having a satelite for one machine it doesn’t really have any benefit, it will just cost more.
Now in my setup, each of our client doesn’t really need zone yet because its just one machine one the same network ( but this will change soon with a new customer) so for me it made sense that a host would be a zone and a satellite as well monitoring itself and I didn’t really see any issues so far to be honest I am monitoring each and every one without much issues 3 months or so.
So today what happened, I did run the exact same playbook of Ansible that was supposed to install icinga2, connect the node to the master, put the scripts in the right places and work out of the box, didn’t work which was weird, after trying a lot of time to find an answer I read a comment pointing out that “icinga2 daemon -X” made obvious which one was the issue that he couldn’t find in logs. So i saw somewhere that icinga master is doing something that is deprecated ( i don’t remember the exact warning message) So I said ok let’s update Icinga.
At the point that the new version was installed one by one the services were coming back.
It could be a coincidence but does someone really know if having different versions affect the system?
Also if someone thinks my setup is still not the best practice again let me know, I just doubt the answer since everything is working well and I don’t see any performance/ strategic issues.