Hello, and thanks in advance for any input.
I've had multiple reads through the Distributed Monitoring guide, but am having enough trouble wrapping my head around it that I want to see if anyone here has some advice. From what I can tell, I'll need something along the lines of what's outlined in the "High-Availability Master with Clients" section of the guide, but could be wrong about that as I'm still struggling to understand it all.
Right now, I'm running Icinga2 on a Ubuntu 16.04 Server virtual machine, with 1CPU and 4GB RAM (Additional resources will be added as needed), monitoring 10 clients with 66 services. So far my network has been a staging area for a project which will soon grow rapidly. I recently added a remote network, which is connected with a site-to-site openVPN, and monitoring hosts on the other side of the tunnel is working perfectly.
A long-standing issue I've had is that if anything happens that takes the monitoring server offline, notifications can't happen. I believe having this second site available puts me in a good position to solve this problem, but I need to figure out the best way to configure Icinga2 for this functionality. If I understand right, I'll need one "Master" node at each site, since Satellites and Clients aren't used to send notifications. The High-Availability Master with Clients seems like it may be applicable, but I don't fully understand it, and my main concern there is having a master node running on a database that's on the other side of a potentially slow VPN tunnel.
Would love to hear ideas about this, even if it's completely different from what I've figured out so far.