Grafana preview not showing in Icinga 2.9.1 / Icingaweb2 2.6.1 / Grafana 5.2.2

Hi all,

It’s been a while since I posted on here. I love the new designed UI!

I manage a large Icinga2 install base with <10 masters and a lot of satellites. The instances do not have internet connectivity so I have been building an offline installer to get everything going on. Things worked quite OK till Grafana moved to V5 which was a major arch change and it broke my graphs. For the time being I just pegged the Grafana version at 4.6.3 and all worked OK.

I am now exploring getting Grafana to R5 to use some of its new features and running into a lot of issues which I think I overcame, for now I am using the database store method.

My issues is that the Icingaweb2 GUI does not display the preview graphs, it shows just the hyperlinked name of the check:

As you can see there is also a weird ‘Apply’ button given you a choice of ‘now’ and ‘now-6h’. When I click it I get an error:

However, when I click on the name of the service it I get the correct graph in the Grafana dashboard:

Furthermore, when I enable debug mode I see the direct link to the rendering engine and that shows a good graphs as well:

2018-08-20%2016_15_13-base-metrics%20(PNG%20Image%2C%20660%C2%A0%C3%97%C2%A0360%20pixels)

Another observation is that I can’t get the Grafana config page to show (but I had this issue in 4.6.3 as well where the graphs did work.

I run the following setup in a VM:

CentOs 7.5 (1804)
Icinga2 2.9.1
Icingaweb2 2.6.1
Grafana 5.2.2
Graphite2 1.3.10

I’ve disabled SELinux and the firewall. I am pulling my hair out. My hunch is that it’s a simple perm issue.

My grafana.ini:

[grafana]
host = “192.168.108.144:3000”
protocol = “http”
datasource = “graphite”
defaultdashboardstore = “database”
defaultdashboard = “base-metrics”
defaultdashboarduid = “1R-d3htiz”
defaultdashboardpanelid = “3”
defaultorgid = “1”
height = “360”
width = “720”
enableLink = “yes”
debug = “1”

Does anyone have any pointers I can look into?

Thanks!

No, I don’t use Grafana so expect except pointing you to the github project seeing a config option “Grafana v5” being activated…

1 Like

Please post the full config.ini and graph.ini. From first view i see you use base-metrics as default dashboard, if it is the dashboard from the module it will not work. Better use the provided default dashboard as default dashboard.

1 Like

Thanks you! See full response below

Nice to hear from the MASTER himself! Thanks @Carsten for helping everyone, keep up your good work! When I show people the graphs everyone is blown away!

Alas, I think I found my mistake. The ‘org name’ as setup in the GUI and as listed in the grafana.ini file did NOT match. As soon as I matched those, the graph appeared!

2018-08-21%2013_09_05-Grafana%20-%20Home

2018-08-21%2013_10_07-icinga29a

Like @Wolfgang mentioned there is a Grafana version 5 check box, denoted by the ‘version’ directive in grafana.ini. The DOC states that:

version Optional If you want to use Grafana 5 with all features, enable this option with 1 (true). Defaults to 0 (false).

That suggest that you could still use V5 without version set to 1, but that you won’t have ALL v5 functionality. I toggled it to see the impact but could not find a difference in behaviour immediately.

I had also played with ‘access mode’ and set it to ‘direct’ in that case the preview graphs did appear, however you needed to be logged into Grafana to see them.

The graphs work now, however, the weird ‘apply’ button is still there and clicking it gives an error. I do think this is a different issue altogether. I found this interesting posting:

#10174

Where the owner says it was a ‘rewrite’ (read write?) issue… hmm

I will upload my working configs in case other people run into this. If anyone reads this later, feel free to @napster me as I like to offer my help.

The dashboard I used was supplied with the install and located here:

/usr/share/icingaweb2/modules/grafana/dashboards/graphite/base-metrics.json

When you import the dashboard, you should check 2 things:

  1. the name of the dashboard is now base-metrics NOT base-metrics.json
  2. the UID is a new v5 thing, it can be auto generated for you, but if you want to use the config files attached here change it to ‘1R-d3htiz’

Here is my grafana.ini (in /etc/grafana/grafana.ini)
##################### Grafana Configuration Example #####################
#
# Everything has defaults so you only need to uncomment things you want to
# change

# possible values : production, development
;app_mode = production

# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
;instance_name = ${HOSTNAME}

#################################### Paths ####################################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
;data = /var/lib/grafana

# Directory where grafana can store logs
;logs = /var/log/grafana

# Directory where grafana will automatically scan and look for plugins
;plugins = /var/lib/grafana/plugins

# folder that contains provisioning config files that grafana will apply on startup and while running.
;provisioning = conf/provisioning

#################################### Server ####################################
[server]
# Protocol (http, https, socket)
protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
;domain = localhost

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
;root_url = http://localhost:3000

# Log web requests
;router_logging = false

# the path relative working path
;static_root_path = public

# enable gzip
;enable_gzip = false

# https certs & key file
;cert_file =
;cert_key =

# Unix socket path
;socket =

#################################### Database ####################################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as separate properties or as on string using the url properties.

# Either "mysql", "postgres" or "sqlite3", it's your choice
type = postgres
host = 127.0.0.1:5432
name = grafana
user = icinga
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = icinga

# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
;url =

# For "postgres" only, either "disable", "require" or "verify-full"
;ssl_mode = disable

# For "sqlite3" only, path relative to data_path setting
;path = grafana.db

# Max idle conn setting default is 2
;max_idle_conn = 2

# Max conn setting default is 0 (mean not set)
;max_open_conn =

# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
;conn_max_lifetime = 14400

# Set to true to log the sql calls and execution times.
log_queries =

#################################### Session ####################################
[session]
# Either "memory", "file", "redis", "mysql", "postgres", default is "file"
;provider = file

# Provider config options
# memory: not have any config yet
# file: session dir path, is relative to grafana data_path
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=grafana`
# mysql: go-sql-driver/mysql dsn config string, e.g. `user:password@tcp(127.0.0.1:3306)/database_name`
# postgres: user=a password=b host=localhost port=5432 dbname=c sslmode=disable
;provider_config = sessions

# Session cookie name
;cookie_name = grafana_sess

# If you use session in https only, default is false
;cookie_secure = false

# Session life time, default is 86400
;session_life_time = 86400

#################################### Data proxy ###########################
[dataproxy]

# This enables data proxy logging, default is false
;logging = false

#################################### Analytics ####################################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
;reporting_enabled = true

# Set to false to disable all checks to https://grafana.net
# for new vesions (grafana itself and plugins), check is used
# in some UI views to notify that grafana or plugin update exists
# This option does not cause any auto updates, nor send any information
# only a GET request to http://grafana.com to get latest versions
;check_for_updates = true

# Google Analytics universal tracking code, only enabled if you specify an id here
;google_analytics_ua_id =

#################################### Security ####################################
[security]
# default admin user, created on startup
admin_user = admin

# default admin password, can be changed before first start of grafana,  or in profile settings
admin_password = admin

# used for signing
secret_key = SW2YcwTIb9zpOOhoPsMm

# Auto-login remember days
;login_remember_days = 7
;cookie_username = grafana_user
;cookie_remember_name = grafana_remember

# disable gravatar profile images
;disable_gravatar = false

# data source proxy whitelist (ip_or_domain:port separated by spaces)
;data_source_proxy_whitelist =

# disable protection against brute force login attempts
;disable_brute_force_login_protection = false

#################################### Snapshots ###########################
[snapshots]
# snapshot sharing options
;external_enabled = true
;external_snapshot_url = https://snapshots-origin.raintank.io
;external_snapshot_name = Publish to snapshot.raintank.io

# remove expired snapshot
;snapshot_remove_expired = true

#################################### Dashboards History ##################
[dashboards]
# Number dashboard versions to keep (per dashboard). Default: 20, Minimum: 1
;versions_to_keep = 20

#################################### Users ###############################
[users]
# disable user signup / registration
allow_sign_up = false

# Allow non admin users to create organizations
allow_org_create = false

# Set to true to automatically assign new users to the default organization (id 1)
;auto_assign_org = true

# Default role new users will be automatically assigned (if disabled above is set to true)
;auto_assign_org_role = Viewer

# Background text for the user field on the login page
;login_hint = email or username

# Default UI theme ("dark" or "light")
;default_theme = dark

# External user management, these options affect the organization users view
;external_manage_link_url =
;external_manage_link_name =
;external_manage_info =

# Viewers can edit/inspect dashboard settings in the browser. But not save the dashboard.
;viewers_can_edit = false

[auth]
# Set to true to disable (hide) the login form, useful if you use OAuth, defaults to false
disable_login_form = false

# Set to true to disable the signout link in the side menu. useful if you use auth.proxy, defaults to false
;disable_signout_menu = false

# URL to redirect the user to after sign out
;signout_redirect_url =

#################################### Anonymous Auth ##########################
[auth.anonymous]
# enable anonymous access
enabled = True

# specify organization name that should be used for unauthenticated users
org_name = REMOVED LLC

# specify role for unauthenticated users
org_role = Viewer

#################################### Github Auth ##########################
[auth.github]
;enabled = false
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email,read:org
;auth_url = https://github.com/login/oauth/authorize
;token_url = https://github.com/login/oauth/access_token
;api_url = https://api.github.com/user
;team_ids =
;allowed_organizations =

#################################### Google Auth ##########################
[auth.google]
;enabled = false
;allow_sign_up = true
;client_id = some_client_id
;client_secret = some_client_secret
;scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
;auth_url = https://accounts.google.com/o/oauth2/auth
;token_url = https://accounts.google.com/o/oauth2/token
;api_url = https://www.googleapis.com/oauth2/v1/userinfo
;allowed_domains =

#################################### Generic OAuth ##########################
[auth.generic_oauth]
;enabled = false
;name = OAuth
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email,read:org
;auth_url = https://foo.bar/login/oauth/authorize
;token_url = https://foo.bar/login/oauth/access_token
;api_url = https://foo.bar/user
;team_ids =
;allowed_organizations =

#################################### Grafana.com Auth ####################
[auth.grafana_com]
;enabled = false
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email
;allowed_organizations =

#################################### Auth Proxy ##########################
[auth.proxy]
;enabled = false
;header_name = X-WEBAUTH-USER
;header_property = username
;auto_sign_up = true
;ldap_sync_ttl = 60
;whitelist = 192.168.1.1, 192.168.2.1

#################################### Basic Auth ##########################
[auth.basic]
;enabled = true

#################################### Auth LDAP ##########################
[auth.ldap]
;enabled = false
;config_file = /etc/grafana/ldap.toml
;allow_sign_up = true

#################################### SMTP / Emailing ##########################
[smtp]
;enabled = false
;host = localhost:25
;user =
# If the password contains # or ; you have to wrap it with trippel quotes. Ex """#password;"""
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost
;from_name = Grafana
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com

[emails]
;welcome_email_on_sign_up = false

#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
;level = info

# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
;filters =

# For "console" mode only
[log.console]
;level =

# log line format, valid options are text, console and json
;format = console

# For "file" mode only
[log.file]
;level =

# log line format, valid options are text, console and json
;format = text

# This enables automated log rotate(switch of following options), default is true
;log_rotate = true

# Max line number of single file, default is 1000000
;max_lines = 1000000

# Max size shift of single file, default is 28 means 1 << 28, 256MB
;max_size_shift = 28

# Segment log daily, default is true
;daily_rotate = true

# Expired days of log file(delete after max days), default is 7
;max_days = 7

[log.syslog]
;level =

# log line format, valid options are text, console and json
;format = text

# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
;network =
;address =

# Syslog facility. user, daemon and local0 through local7 are valid.
;facility =

# Syslog tag. By default, the process' argv[0] is used.
;tag =

#################################### Alerting ############################
[alerting]
# Disable alerting engine & UI features
;enabled = true
# Makes it possible to turn off alert rule execution but alerting UI is visible
;execute_alerts = true

#################################### Explore #############################
[explore]
# Enable the Explore section
;enabled = false

#################################### Internal Grafana Metrics ##########################
# Metrics available at HTTP API Url /metrics
[metrics]
# Disable / Enable internal metrics
;enabled           = true

# Publish interval
;interval_seconds  = 10

# Send internal metrics to Graphite
[metrics.graphite]
# Enable by setting the address setting (ex localhost:2003)
;address =
;prefix = prod.grafana.%(instance_name)s.

#################################### Distributed tracing ############
[tracing.jaeger]
# Enable by setting the address sending traces to jaeger (ex localhost:6831)
;address = localhost:6831
# Tag that will always be included in when creating new spans. ex (tag1:value1,tag2:value2)
;always_included_tag = tag1:value1
# Type specifies the type of the sampler: const, probabilistic, rateLimiting, or remote
;sampler_type = const
# jaeger samplerconfig param
# for "const" sampler, 0 or 1 for always false/true respectively
# for "probabilistic" sampler, a probability between 0 and 1
# for "rateLimiting" sampler, the number of spans per second
# for "remote" sampler, param is the same as for "probabilistic"
# and indicates the initial sampling rate before the actual one
# is received from the mothership
;sampler_param = 1

#################################### Grafana.com integration  ##########################
# Url used to to import dashboards directly from Grafana.com
[grafana_com]
;url = https://grafana.com

#################################### External image storage ##########################
[external_image_storage]
# Used for uploading images to public servers so they can be included in slack/email messages.
# you can choose between (s3, webdav, gcs, azure_blob, local)
;provider =

[external_image_storage.s3]
;bucket =
;region =
;path =
;access_key =
;secret_key =

[external_image_storage.webdav]
;url =
;public_url =
;username =
;password =

[external_image_storage.gcs]
;key_file =
;bucket =
;path =

[external_image_storage.azure_blob]
;account_name =
;account_key =
;container_name =

[external_image_storage.local]
# does not require any configuration

Here is config.ini in /etc/icingaweb2/modules/grafana/config.ini
[grafana]
version = “1”
host = “192.168.108.187:3000”
protocol = “http”
accessmode = “proxy”
datasource = “graphite”
defaultdashboardstore = “db”
defaultdashboard = “base-metrics”
defaultdashboarduid = “1R-d3htiz”
defaultdashboardpanelid = “3”
defaultorgid = “1”
authentication = “anon”
height = “360”
width = “720”
enableLink = “yes”
debug = “1”

Here is graphs.ini in /etc/icingaweb2/modules/grafana/graphs.ini

[load]
dashboard = "base-metrics"
panelId = "5"
orgId = "1"
dashboarduid = "1R-d3htiz"
repeatable = "no"

[disk]
dashboard = "base-metrics"
panelId = "4"
orgId = "1"
dashboarduid = "1R-d3htiz"
repeatable = "no"

[swap]
dashboard = "base-metrics"
panelId = "6"
orgId = "1"
dashboarduid = "1R-d3htiz"
repeatable = "no"

[memory]
dashboard = "base-metrics"
panelId = "9"
orgId = "1"
dashboarduid = "1R-d3htiz"
repeatable = "no"

[hostalive]
dashboard = "base-metrics"
panelId = "3"
orgId = "1"
dashboarduid = "1R-d3htiz"
repeatable = "no"

Hello @napster

dont use base-metrics dashboard as default dashboard. Everytime the module doesnt find any configured graph in the graphs.ini it will be used. So better take the provided dashboards\graphite\icinga2-default.json as default dashboard.

If you use Version 5 of Grafana you have to enable the Version features of the module, if you dont enable it you cant use dashboards with non unique names and maybe in the future Grafana will remove the redirects from version 4 URLs. I will modify the docs to make it more clear.

The timerange apply was a PR, i didnt test it well and jsut found there is something not working as it should be. Fix will come soon.

Regards,
Carsten

@Carsten thank you kind sir!

  1. ‘Apply’ issue: Glad it was not just me, looking fwd to the fix

  2. "version = “1"” thanks for clarifying. I left it in my config, just noted that things seem to work with it set to 0

  3. ‘base-metrics’ as default… So it means that if I have a check ‘bogus’ for which there is no graph defined in graphs.ini. It will pull up a preview of one of the graphs of base-metric? I do in fact see this e.g. ‘http’ now shows a ‘hostalive’ graph. Is that what you mean? Changing it out for icinga2-default.json would result in it showing a graph from that dashboard? Maybe I misunderstood.

Vielen dank

Hello Napster,

  1. Base-Metrics will mostly not work as default, thats the reason you should use icinga2-default dasboard.

I fixed the manual timerange in the master of the module, please test if it is working now for you.

Hi @Carsten,

I downloaded a the new master and indeed no longer get the error, also if I click navigate to Navigate -> Modules -> Configuration I no longer get the error and I see all the settings! 2 birds with 1 stone!

I’ve done some tests with timerange in config.ini and in graph.ini.

The following setting in config.ini has a global impact, all graphs now show 2 options: Now and Now-3h

timerange = “3h”

I can indeed overwrite the global value for one of the checks by setting a different value for a check in graphs.ini

Clicking apply does no longer result in an error, but also doesn’t seem to do anything. I tried with now and now-3h selecting. The GUI thinks for a second and just displays the same graph…

Is it supposed to work like that? I am OK with how it now all works and looks, my users will no longer get the error message.

I will create some additional dashboards and start converting my R4 dashboards.Thanks again for all your help!