Posts by red_eagle

This forum was archived to /woltlab and is now in read-only mode.

    Yeah, we've been trying different settings there but none seemed to make it better. Here's iostat:


    I figured that was it, our carbon server isn't on the best hardware. It has 8GB of memory but it fills up and starts using swap all day.

    Here's our carbon.conf if anyone has any ideas.


    We get log entries like this everynight and when we come in icinga2 has died. Any ideas on how to start troubleshooting or where to look?


    Code
    1. [2017-08-24 01:58:43 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3960958, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 2 days, 2 hours, 56 minutes and 13 seconds
    2. [2017-08-24 01:58:53 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3961338, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 1 day, 4 hours, 57 minutes and 27 seconds
    3. [2017-08-24 01:59:03 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3961768, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 1 day, 1 hour, 35 minutes and 29 seconds
    4. [2017-08-24 01:59:13 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3962097, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 1 day, 9 hours, 27 minutes and 4 seconds
    5. [2017-08-24 01:59:23 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3962451, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 1 day, 7 hours, 5 minutes and 27 seconds
    6. [2017-08-24 01:59:33 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3962842, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 1 day, 4 hours, 9 minutes and 4 seconds
    7. [2017-08-24 01:59:43 -0700] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 0, rate: 65.5833/s (3935/min 15079/5min 68621/15min);
    8. [2017-08-24 01:59:43 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3962863, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 21 days, 20 hours, 8 minutes and 47 seconds
    9. [2017-08-24 01:59:53 -0700] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 3962864, rate: 198.483/s (11909/min 62931/5min 179998/15min); empty in 458 days, 15 hours, 59 minutes and 38 seconds



    icinga2 daemon -C

    I'm pretty terrible with jasper reports but I was able to get the query needed for the data from the reports in https://github.com/Icinga/icinga-reports

    If you have the availability mysql from icinga-reports you can use :


    The Hostgroup availability report in icinga-reports has a pie chart with the percent time ok and not ok. I've been struggling to get those values into a table for a report.

    Anyone know a query to get those values? I just need a table with three columns, Hostname, % Time up, % Time Down of the hosts in a hostgroup.

    I was wrong about the slow queries. I didn't know logging them wasn't persistent through service restarts.


    Our slow query log is just this query over and over.


    We set our sort and join buffer sizes and now we don't have any slow queries. Our CPU's are all still maxed out and things run slow. Here's the output of mysqladmin.

    Code
    1. Uptime: 2759  Threads: 48  Questions: 550184  Slow queries: 0  Opens: 26806  Flush tables: 1  Open tables: 416  Queries per second avg: 199.414
    2. Uptime: 3052 Threads: 68 Questions: 616221 Slow queries: 0 Opens: 27732 Flush tables: 1 Open tables: 416 Queries per second avg: 201.907


    It also looks like we have a ton of open files when we run lsof with 45 connections to another server hosting icingaweb2


    Code
    1. sudo netstat -anp | grep ESTABLISHED | wc -l
    2. 45


    Edit:

    Here's more on connections....


    When I lsof I've got alot of these as well.


    dnsmichi So here's a few things I've found but I'm a not super familiar with DB management. Any clues/guides on how I can troubleshoot it? Our event overview takes about thirty seconds plus to load and all the pages like overview/hosts take roughly five to ten seconds to load.

    I found the problem but don't know a solution. Adding the include_once to index.php gives us the results we want but it also adds the tracking code, with <script> to the icinga.min.js file, which is why the expand/collapse menu's fail.


    Edit: Nine minutes after posting this I think we got it. We created a file in /usr/share/icingaweb2/public called analyticstracking.php and copied and pasted the tracking html there. Then in /usr/share/icingaweb2/application/layouts/scripts/inline.phtml we put <?php include_once("analyticstracking.php") ?> on line two in between the two lines that were already there.

    Everything seems to be working fine now...

    Has anyone successfully integrated Google Analytics with IcingaWeb2? We have ~60 users and track what tools techs are using like grafana, nagvis, etc. with Google Analytics. We've added the PHP code

    Code
    1. include_once("analyticstracking.php");

    To /usr/share/icingaweb2/puiblic/index.php and our tracking code to a separate file called analyticstracking.php and were able to see statistics but it broke our dropdown menus in Icingaweb2. For example clicking on "Overview" would cause it to expand and collapse.

    The ndomy backend allows to configure a remote mysql host too. I would create a read only user for nagvis (or icingaweb2) which only has access from the remote web server and configure that inside nagvis.

    So install it on the IcingaWeb2 server and use ndo2db for the backend?

    Not sure if there's a better way but we added this code to our lib/icinga2.rb for dashing and just read the output.


    Code
    1. @no_hostgroup = []
    2. all_host_data = getHostObjects(["name", "state", "groups"], nil, nil)
    3. all_host_data.each do |host|
    4. htgroup = host["attrs"]["groups"]
    5. if (htgroup.length == 0)
    6. @no_hostgroup.push( host["attrs"]["name"])
    7. next
    8. end