Graphite not showing older data

graphite

(Lorenzo Graziosi) #1

Hi. I know there are a lot of threads like this, but I can’t understand why is this happening.
Basically, we’re using Graphite connected to Icinga2, and data older than two weeks is not showed.

This is our storage-schemas.conf:

[icinga2_internals]
pattern = ^icinga2\..*\.(max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
retentions = 1m:2w,5m:8w,30m:1y,360m:8y

[icinga2_default]
# intervals like PNP4Nagios uses them per default
pattern = ^icinga2\.
retentions = 1m:2w,5m:8w,30m:1y,360m:8y

[default_1min_for_1day]
pattern = .*
retentions = 1m:2w,5m:8w,30m:1y,360m:8y

This is a whisper-info example:

maxRetention: 252288000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 785920

Archive 0
retention: 1209600
secondsPerPoint: 60
points: 20160
size: 241920
offset: 64

Archive 1
retention: 4838400
secondsPerPoint: 300
points: 16128
size: 193536
offset: 241984

Archive 2
retention: 31536000
secondsPerPoint: 1800
points: 17520
size: 210240
offset: 435520

Archive 3
retention: 252288000
secondsPerPoint: 21600
points: 11680
size: 140160
offset: 645760

So everything looks fine, right?
Then why if I go to the Icinga graph, or to graphite directly, if I select like “three weeks” nothing shows up?

After two weeks it seems tha data is deleted. What am I missing?
Thank you for your help.


(Carsten Köbke) #2

(Lorenzo Graziosi) #3

Hi, thanks for the reply.
Every Icinga service has the same behaviour. Everyone has the same values, whisper-info, and graphs.
I also tried something else. If I force a value like ‘1m:4w,5m:16w,30m:2y,360m:8y’ instead of ‘1m:2w,5m:8w,30m:1y,360m:8y’, I will see the last four weeks instead of two.
It’s like only the first value is considered.
Am i missing something?


#4

Could be that i am wrong here, but have you resized your whisper files after setting the new retention times? If i am not mistaken they are created to a fixed size with the retention set at the time of creation and only change if you actively force them too or delete the dataset and create it new.

Something like this here. One of the first things i found this quick but that might be where you are stuck (or i might be wrong, but i had something like that in my Test setup a good while ago).


(Lorenzo Graziosi) #5

Hi, thanks for the reply.
Let me see if I understood correctly: if I use a whisper-resize command, they still won’t change correctly?
I used it some times, but it looked like the files were changing their size every time.
The whisper-info confirms it, am I wrong?


(Lorenzo Graziosi) #6

Okay, I think I solved the problem, but I’m still not sure why its behaviour is like this.
Basically, the xFilesFactor was the problem. After setting it to 0, and resizing every .wsp again, the graphs are correct.

If anyone has the same problem, this is my solution:

systemctl stop graphite_carbon_cache.service
cd path/to/whisperfolder
find ./ -type f -name 'value.wsp' -exec whisper-set-aggregation-method.py {} average 0 \;
find ./ -type f -name 'warn.wsp' -exec whisper-set-aggregation-method.py {} average 0 \;
find ./ -type f -name 'crit.wsp' -exec whisper-set-aggregation-method.py {} average 0 \;
find ./ -type f -name 'max.wsp' -exec whisper-set-aggregation-method.py {} max 0 \;
find ./ -type f -name '*.wsp' -exec whisper-resize.py --nobackup {} 1m:2w 5m:8w 30m:1y 360m:8y \;
systemctl start graphite_carbon_cache.service

And then changing the default values in the storage-aggregation configuration file.


#7

When the xFilesFactor fixed it you had more than 50% of your values beeing NULL (default xFilesFactor is 0,5) so when the retention run it threw away all values as too many where NULL. If this is correct (so many values beeing NULL) or where this comes from is a totaly different topic. Strange all in all, i do have some NULL’s in between my values, but over 50% should be very vissible to begin with.

Glad you figured it out!