Icinga2, Grafana Module, no datapoints in Grafana

This forum was archived to /woltlab and is now in read-only mode.
  • Hi all,


    I installed Grafana Module following Mikesch's HowTo on GIT.


    Unfortunately, none of the dashboards in Grafana display data, therefore icingaweb2 doesn't either.


    I can see datapoints created within Graphite itself, so I take it that graphite itself is working.




    Datasource in Grafana is configured as followed:




    What I'm wondering about is that when clicking "Dashboards" tab, I get presented with another Dashboard:




    I also do see many hosts which aren't in Icinga any longer, is there maybe need to delete something in Grafana or whereever?


    I'm really stuck here and any help would be highly appreciated.


    grafana.ini looks as followed:


    Code
    1. [grafana]
    2. host = "<hostname>:4443"
    3. protocol = "https"
    4. defaultdashboard = "graphite-icinga2-default?orgId=1"
    5. defaultorgid = "1"
    6. shadows = "1"
    7. defaultdashboardstore = "db"
    8. theme = "light"
    9. datasource = "graphite"
    10. accessmode = "iframe"

    I believe grafana module isn't the (main) issue as it's displaying graphs but without datapoint so I believe what I need to solve is that Grafana isn't having any datapoints:




    Thanks a lot for any help,



    Ben

  • Did you import the graphite dashboards from the grafana module? Those are required to get things working in Grafana in the first place. Once that's working, you should look into Icinga Web 2.

  • So, stop at the Grafana side as said, and try to find out why the datasources don't provide any metrics. The Grafana docs should be of help here, how to edit a graph on a dashboard and see which query is actually fired. This allows to test-drive specific things.

  • That's the query which is fired for dashboard graphite-icinga2-default


    Code
    1. xhrStatus:"complete"
    2. request:Object
    3. method:"POST"
    4. url:"api/datasources/proxy/5/render"
    5. data:"target=aliasByNode(icinga2.192_168_199_160_(MIB_Test_PDU).services.Service%3A_Ping.*.perfdata.*.value%2C%206)&from=-6h&until=now&format=json&maxDataPoints=1920"
    6. response:Array[0]


    In that dashboard I also see only one graph, is that expected?

    Additionally, this host doesn't exist any longer in Icinga. Any sense in removing stuff from Graphite? If yes what and how? Whisper data?

  • Addition:


    When I change $hostname to a particular host, I get the following:



    Code
    1. response:Object
    2. error:"Internal Server Error"
    3. response:"<body style="background-color: #666666; color: black;"> <center> <h2 style='font-family: "Arial"'> <p>Graphite encountered an unexpected error while handling your request.</p> <p>Please contact your site administrator if the problem persists.</p> </h2> <br/> <div style="width: 50%; text-align: center; font-family: monospace; background-color: black; font-weight: bold; color: #ff4422;"> </div> <div style="width: 70%; text-align: left; background-color: black; color: #44ff22; border: thin solid gray;"> <pre> Traceback (most recent call last): File &quot;/usr/lib/python2.7/dist-packages/django/core/handlers/base.py&quot;, line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/views.py&quot;, line 125, in renderView seriesList = evaluateTarget(requestContext, target) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 10, in evaluateTarget result = evaluateTokens(requestContext, tokens) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 21, in evaluateTokens return evaluateTokens(requestContext, tokens.expression) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 30, in evaluateTokens return func(requestContext, *args) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/functions.py&quot;, line 1150, in aliasByNode series.name = &#39;.&#39;.join(metric_pieces[n] for n in nodes) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/functions.py&quot;, line 1150, in &lt;genexpr&gt; series.name = &#39;.&#39;.join(metric_pieces[n] for n in nodes) IndexError: list index out of range </pre> </div> </center> "
    4. message:"Internal Server Error"

    Complete query in that case:


    Code
    1. xhrStatus:"complete"
    2. request:Object
    3. method:"POST"
    4. url:"api/datasources/proxy/5/render"
    5. data:"target=aliasByNode(icinga2.srvde197_rce-group_local.services.Service%3A_Ping.*.perfdata.*.value%2C%206)&from=-6h&until=now&format=json&maxDataPoints=1912"
    6. response:Object
    7. error:"Internal Server Error"
    8. response:"<body style="background-color: #666666; color: black;"> <center> <h2 style='font-family: "Arial"'> <p>Graphite encountered an unexpected error while handling your request.</p> <p>Please contact your site administrator if the problem persists.</p> </h2> <br/> <div style="width: 50%; text-align: center; font-family: monospace; background-color: black; font-weight: bold; color: #ff4422;"> </div> <div style="width: 70%; text-align: left; background-color: black; color: #44ff22; border: thin solid gray;"> <pre> Traceback (most recent call last): File &quot;/usr/lib/python2.7/dist-packages/django/core/handlers/base.py&quot;, line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/views.py&quot;, line 125, in renderView seriesList = evaluateTarget(requestContext, target) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 10, in evaluateTarget result = evaluateTokens(requestContext, tokens) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 21, in evaluateTokens return evaluateTokens(requestContext, tokens.expression) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/evaluator.py&quot;, line 30, in evaluateTokens return func(requestContext, *args) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/functions.py&quot;, line 1150, in aliasByNode series.name = &#39;.&#39;.join(metric_pieces[n] for n in nodes) File &quot;/usr/lib/python2.7/dist-packages/graphite/render/functions.py&quot;, line 1150, in &lt;genexpr&gt; series.name = &#39;.&#39;.join(metric_pieces[n] for n in nodes) IndexError: list index out of range </pre> </div> </center> "
    9. message:"Internal Server Error"
  • I don't see your desktop, so I don't know what "only one graph" means. Please be as verbose as possible.


    In terms of removing things from Graphite, kindly use google. I haven't done that myself, no best practices here.


    Edit: If the Graphite Web API throws an internal error, please consult the apache logs.

  • Sorry for not being verbose enough, forgot to attach the screenshot.


    By only one graph I mean the following:




    This is what I get when navigating to the dashboard in Grafana. But I could imagine this is expected, as it's $hostname and $service.


    So it's configured like this:





    Internal server error only occurs when changing $hostname and $service to a particular host and service.

  • Could this be down to an issue with Graphite itself?


    /etc/icingaweb2/modules/graphite/config.ini:

  • I now cleared /var/lib/graphite/whisper and it's filling up with fresh files.

    However, I take it that the naming might be wrong and that's why Grafana doesn't find something so it may also be down to graphite -> Grafana but I couldn't find any hint as of how graphite would need to be configured to properly work with Grafana.

  • Hm, you are mixing things here. The Icinga Web 2 module configuration shows the Graphite module, while you are talking about the Grafana module initially. Pick one of them later on, once Grafana is working. Or go for the Graphite Module, and leave Grafana behind. But please don't start to mix things here.


    If you're going the Icinga Web 2 Module Graphite route, please open a new thread.

  • Sorry for that, meaning grafana module. But I thought Grafana picks from Graphite or what am I mixing up now?

    So I am still looking to get Graphs from Icinga in Grafana, leaving Graphite aside :) Sorry mate for mixing things up by pasting wrong config.

  • Ok, then focus on graphite-web (and carbon-cache, whisper) in combination with Grafana. What for example happens, if you add an additional graph to the icinga2-default template?

  • Only a guess..

    Do your carbon-cache configuration(retentions), match your Icinga2 Service schedule ?


    We had Problems with that also and choose to switch to InfluxDB, because we won't change the Whisper Fle structur each time we need a custom service runtime.

  • Ok, then focus on graphite-web (and carbon-cache, whisper) in combination with Grafana. What for example happens, if you add an additional graph to the icinga2-default template?

    No luck :-(

    Only a guess..

    Do your carbon-cache configuration(retentions), match your Icinga2 Service schedule ?


    We had Problems with that also and choose to switch to InfluxDB, because we won't change the Whisper Fle structur each time we need a custom service runtime.

    How would that need to be tackled? How could one switch to Influx?

  • Seems you would need something like a Graphite/Grafana workshop or training, we're not doing much progress here. InfluxDB is a different TSDB with its own storage schema. The standalone parts are free, cluster and high performance is enterprise only.