IDO not storing data

This forum was archived to /woltlab and is now in read-only mode.
  • I've been working on getting my configuration from icinga1 ported over to icinga2 version 2.4.10-1, running on OpenBSD 6 - an extremely slow and painful process, but that's beside the point. The old install was running on OpenBSD 5.5 and icinga 1. At this point I have all my hosts moved over (447 of them), and was working on moving my services across. Everything was going fine until I got up to around 150 services or so, at which point the web interface stopped showing any hosts/services after a icinga restart, and my icinga log started showing things like:

    icinga2.log wrote:

    [2017-02-28 14:01:30 -0900] information/IdoPgsqlConnection: Query queue items: 2286, query rate: 0.366667/s (22/min 22/5min 22/15min); empty in infinite time, your database isn't able to keep up

    I found that there was an open connection to the IDO database (Postgresql 5.5 running on the localhost to rule out network issues) that was listed as "idle in transaction" state, and looking at the process list showed that this connection in fact wasn't doing anything (total time on-processor was only 1 second, CPU utilization a solid 0%). In looking around, I find that apparently there is some startup process that does a bunch of database updates that has to complete before the "normal" process can start doing its thing, so it would appear once I get over a few hundred services, this process just hangs.

    I did find some references to adding certain indexes on the database to speed things up, which I have done, but it hasn't appeared to help any.

    So, two questions:

    1) Is there a way I can fix this, and if so, what?

    2) Is there any way to use icinga2 WITHOUT ido utils? It would appear that the icinga-web2 interface relies on IDO utils, but if there was some way I could use the (vastly superior, at least from an interface standpoint) CGI interface from icinga1 instead, that would be great, and allow me to ditch IDO utils completely.

    For what it's worth, icinga2 itself appears to be fine with the number of hosts/services, it's just the IDO utils that appears to have issues.

  • I'd update to 2.6.2 and continue your analysis over there. 2.4.x is fairly old. If you say PostgreSQL 5.5 .. such a version does not exist.

    The IDO database is required for Icinga Web 2.

  • I'd update to 2.6.2 and continue your analysis over there. 2.4.x is fairly old. If you say PostgreSQL 5.5 .. such a version does not exist.

    The IDO database is required for Icinga Web 2.

    Sorry, I meant 9.5. Typo there. I'm not sure I'll be able to update to 2.6.2 - 2.4.10 is the packed version available for OpenBSD, and often getting anything else to compile is a pain. Even you guys recommend using the packaged version for your OS.

    Thanks for the confirmation that the IDO database is required for Icinga Web 2. So what about the second part of the question - can I use the much superior (in interface design at least) icinga 1 CGI's? I did notice there was an option to log in legacy format (or something like that) - can that, or similar settings, be used to make things compatible, such that IDO is not needed?

  • I don't have that much OpenBSD knowledge, but I thought that adding the ports tree would provide you with the latest and greatest software package.,-main

    Stuart keeps that in shape and pushes main releases quite frequently.

    You can use the Classic UI if you want, but I wouldn't bet on it for new features or active development. There are certain limitations which include the old-grown host/service -> contact being used as REMOTE_USER auth. Workarounds exists, but well. That's software which has reached its end of life somehow.

    I'd suggest to ensure that Icinga Web 2 keeps running. Integrations and of course the Icinga Director will certainly make sure that monitoring is a breeze.

  • I don't have that much OpenBSD knowledge, but I thought that adding the ports tree would provide you with the latest and greatest software package.,-main

    Stuart keeps that in shape and pushes main releases quite frequently

    Yeah, if I update to HEAD on the source tree. That's considered bad practice unless you want to live dangerously - it's generally preferable to stick with the STABLE branch, which is just back ports of bug fixes. In this case, however, it's probably worth a shot, so I'll try it and see what happens. Maybe it will solve all my issues.

    Assuming the update gets everything in working conditions, are there any skins or something to make the icinga web 2 interface look/act more like the classic CGI scripts? My main concern is with the amount of data displayed. For example, on the "old" interface, the tactical overview showed a nice breakdown of Network outages, hosts, services, service checks, hosts checks, and monitoring features, each with appropriate sub-categories (UP/Down/pending/unreachable, etc) in a nice, compact display. With icinga web 2, I get two HUGE colored squares for JUST the hosts, with a couple of smaller squares under each one for the services (although that's not at all clear, since they aren't labeled) - much less data displayed in much more space. Looking at the host/services overview list it's similar. The old interface displayed each host/service on one line with most of the relevant information displayed on that line across the screen. Now it's a block taking up about as much vertical space as two and a half or three lines of the classic, and using barely any of the horizontal space.

    I want to keep up with development, and I do understand that development of the old systems is, effectively, dead. I also, however, need to have a monitoring system that can present me with the information I need easily - and the classic UI does a much better job of that than the "new" UI does, even while the new UI does more things than the classic. So any suggestions for alternate UI's/skins that could facilitate actually using icinga and which would still be under development would be appreciated.

  • Upgrading to 2.6.2 worked to fix the IDO issues. So now I just have to learn to live with the web interface. One other thing I just found lacking: In the CGI's, I was able to view the actual command executed, which was a HUGE plus for debugging why checks were failing. With icinga web 2 I've apparently lost that ability, unless someone knows how to do that. At least it is working properly now. Thanks!

  • 0) Thanks for the explanation with stable and head, wasn't aware of that. We as developers release too fast for upstream anyways. But we're aware of certain bugs in 2.4.x (and also 2.5.x) and recommend to go for 2.6.2+ which improves stability and takes care of bug fixes (not so many features this time, but hey).

    1) In terms of Icinga Web 2 and its interface: I'd suggest to open feature requests, design proposals or even look into the code and send in a pull request including your patches on GitHub. That's the way open source projects benefit from their community.

    2) I know, the command expander thing always pops up. There are things you'll need to learn about that:

    In Icinga 1.x the command object is just a command line string which contains placeholders (runtime macros, $ARGx$ and so on). Icinga 2 extends the command objects to make use of its own DSL. That way you can have just one CheckCommand object for optional, required, conditional command arguments. You can even calculate specific arguments at runtime (i.e. by using lambda functions or conditionals).

    Given those possibilities, it is impossible to just parse a string containing placeholders. It does not exist, and the full command line is influenced by runtime parameters. So you cannot just parse such a string (as the Classic CGIs do) and print something. Instead you'll depend on the core to execute the command and return the full command line.

    You can of course fetch that from the REST API where you'll get the full object attributes. It is also mentioned in the troubleshooting docs, since you won't need it by default. Merely to test whether certain arguments work or not.…g#checks-executed-command

    There's a feature request for adding the command line queried from the last_check_result attribute available from the core's REST API.

  • Thanks for the info. Yeah, releases that focus on bug fixes are quite important, for sure. I think I've just been frustrated by how difficult this migration has been for me in general, from having to almost completely re-write the configs (I did find a migration tool that did some of the work, but the config it generated caused segfaults in icinga, though admittedly that was 2.4.10) to getting used to the new web interface. Once the migration's complete and I have my head better wrapped around the icinga2 way of working, I'm sure I'll have a much more rosy outlook on everything. Hopefully I'll be able to find some time to dig into the web interface a bit, although to be honest, I tend to avoid PHP as much as possible (I'm a Python/C++ guy), so I'm not sure how far I'll get there. I do need to come up with something that will work well for a status display in our server room though, so that could be a good motivation :). I'll look into the REST API a bit more. Thanks again!

  • Well, I wrote two versions of such a migration script, but from my experience it does help nothing. The time you save by mangling the old key-value objects into the new DSL is horrible and there is no way to apply new best practices to it. That is also the reason why we've purged away those scripts. Not to annoy the user, but to remove something which will make you hate Icinga 2. And that is clearly not what you want from a tool which keeps you informed about your environment.

    In terms of a dashboard, you might find Dashing interesting. Still no Python, but Ruby ;)