Sync rule memory exhausted when trying to add 32k+ hosts.

This forum was archived to /woltlab and is now in read-only mode.

    I'm trying to use a sync rule to import 32k+ hosts into icinga2 director and I keep getting:

    Code
    1. Fatal error: Allowed memory size of 805306368 bytes exhausted (tried to allocate 80 bytes) in /usr/share/icingaweb2/library/vendor/Zend/Db/Statement/Pdo.php on line 219

    I've tried setting the memory limit to 6G in php.ini and /usr/share/icingaweb2/modules/director/library/Director/IcingaConfig/IcingaConfig.php but keep getting the error.


    While watching on htop it looks like it's not using that much at all though. It generally error's out when memory usage gets to 1.40G/7.79G

    It actually ended up being alot more hosts, and it's just a large network. Now that we got them all in, director and Icinga2 seem to be able to handle it well. The initial deployment was a bit of a struggle but we reached out to one of the Icinga2 director developers and they responded quickly and helped alot.

    Were monitoring mostly networking related services, to many in my opinion but the guy who signs my check disagrees. We have a pretty large WAN with LAN's hanging off it so we need to be able to see alot of information. We rely on our NMS for quite a bit though because there's not enough of us to have a clear picture on what's on our network and the history is alot of help.

    I see! I have only around 100 hosts and even that is sometimes hard to monitor with all the moving parts. I can only imagine how hectic it might be for you.

    Just for the records, in case anyone is running into a similar problem: read_eagle manually tweaked a hard-coded limit in Sync.php. It has initially been added to slightly increase the available memory while syncing. Just, in case you allowed your PHP to use more memory (as you may need to sync A LOT of objects), this leads to an artificial restriction of 768MB - which is pointless.


    Anyone interested in the final fix for this can follow the related issue we created today. For those facing similar issues please also consider upgrading to PHP 7.x, as it is much faster and safes lots of memory. The environment mentioned above still uses 5.x, which also works fine - but it costs more. It's always great to hear that Icinga behaves fine in large environments. And that's not the ending, next year we want to get even faster - and easier to scale.


    Btw, just to throw in some more numbers: today I worked with an environment with 2,5k hosts and 72k services. All hosts and 10k services are synced via Director one by one from various sources (PuppetDB, AWS, AD). The rest (62k services, 121k notification objects) is generated by apply rules. Director renders the config in slightly more than 12 seconds, also still on PHP 5. 2,2k of those hosts are running the Icinga 2 Agent, everything with just two masters, no satellite involved. Satellites would definitively reduce the load, but hey, it's running fine.


    Great to hear that even syncing 37k hosts at once is not an issue. At least not once we toggle artificial limitations ;-)

    The post was edited 1 time, last by TomGelf ().