Grafana shows gaps when zooming in

(Michael Baumgartner) #1

I’m trying to get a decent grafana graph for ages now. Theres something I’m doing very wrong i guess and I can’t figure it out.

My Grafana graphs always have a gap at the beginning, depending on the selected timeframe it is big, small or even the whole graph looks empty:



Those are 6h, 3h and 1h timeframes. 6h does show a decent amount of information. 3h shows less and 1h is empty. Everything below 1h is just empty.

I tried to play around with Storage Aggregations and I read on github somewhere, that setting storage-aggregation average to xFileFactor 0 did solve this issue with some people.
It didn’t for me, in fact, I think i broke it^^ (therefore the gap at the end…)
Probably have to delete the whisper files again and rebuild it from scratch.

Heres my storage-schemas.conf:

[carbon]
pattern = ^carbon\.
retentions = 60s:90d

[icinga_internals]
pattern = ^icinga2\..*\.(max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
retentions = 1m:7d

[icinga_default]
pattern = ^icinga2\.
retentions = 60s:28d,10m:180d,30m:4y

[default_1min_for_7days]
pattern = .*
retentions = 60s:7d,5m:30d,30m:360d

heres my current storage-aggregation.conf:

[min]
pattern = \.min$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = \.max$
xFilesFactor = 0.1
aggregationMethod = max

[sum]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

[default_average]
pattern = .*
xFilesFactor = 0.0
aggregationMethod = average

Almost all my checks which send perfdata are triggered every 60s. So as I understand my storage-schema.conf should be ok.
I am out of ideas. How do I get a consistent graph? How do I get rid of those gaps?

If you need any more info let me know.

Thx!

(Michael Baumgartner) #2

For anyone having this issue in the future: The config I posted above was working, I just had to delete all whisper files and let graphite rebuild them. After that the gaps went away and I can zoom in and out as much as I want.

The solution was indeed what I wrote:

I tried to play around with Storage Aggregations and I read on github somewhere, that setting storage-aggregation average to xFileFactor 0 did solve this issue with some people.

#3

It might not be necessary in every case to delete your data.

Graphite offers a so called whisper-resize.py Python script.

Using it somehow like this:

sudo -u _graphite find [pathToWhisper]/icinga2/[Hostname]/services/[Servicename]/ -type f -name '*.wsp' -exec python /etc/graphite/whisper-resize.py {} [Retentiontime, e.g 1m:1d 1h:2d] \;

Will enable you to iterate through your DB.

For further information you also wanna read this:
https://github.com/graphite-project/whisper
and this:
https://gist.github.com/kirbysayshi/1389254