14

I have a counter at stats.message.foo and want to move it to stats.messages.foo.

I've updated my code to populate the new counter, however the old one still exists.

I've read all I need to do to remove a stat from graphite is to delete the appropriate whisper file on disk, however it seems that within a few seconds of removing the wsp it gets regenerated (without data).

This is an annoyance if I want to rename the key that the data is stored under as I need to remember which key is the correct one.

Does anyone know how to permanently remove the old counter?

Community
  • 1
hafichuk
  • 762
  • 2
  • 5
  • 18

5 Answers5

9

The reason so many people are struggling with this issue (me included) is because statsd (depending on its settings) continues to write data to old stats (whisper files) even after you delete the whisper file (.wsp).

Check your statsd settings at /etc/statsd/config.js (on Ubuntu). Example settings at https://github.com/etsy/statsd/blob/master/exampleConfig.js. I would recommend setting deleteIdleStats to true so that statsd does not write stats when they are not received.

The downfall of deleteIdleStats is that Graphite may now have null values for certain stats. This can be handled on the visualization layer (I recommend Grafana).

You have two options to stop regeneration of deleted whisper files:

  • simply restart statsd with service statsd restart or
  • update statsd settings to not send stats to backend if the stats are not received AND then service statsd restart

To delete wsp files that you don't need anymore:

  1. List the files to be deleted withfind pathToUnwantedStatistic -print
  2. if that is printing only the files you want deleted, run the following line to delete the unwanted files find pathToUnwantedStatistic -delete
IvanD
  • 191
  • 1
  • 3
1

I had a similar problem where entries still existed in Graphite although no data was available. I am using statsd and it didn't have metrics in there either. It turns out I had to delete these folders in addition to the wsp files.

My exact case is, I used the machine name as prefix for statsd. Carbon created folder for each of the machines in e.g /opt/graphite/storage/whisper/stats/ip-0A215A08, where it stored the wsp for my metrics. Because of that, once I deleted the wsp files, I still had a bunch of "folders" in Graphite w/o any data that still showed up in the UI.

Checks on Carbon/Graphite

  • .wsp files
  • directories for the stats you got rid of

On statsd (if you are using it), verify there are no more gauges, timers or counters for the metrics you want to get rid of. For this, you may also telnet on your instance (default port 8126) and check for metrics with proper commands of the Admin interface instead of killing the process.

mrlabbe
  • 111
  • 1
0

I had the same issue. I was able to remove the directory by first killing a StatsD server that was running. Perhaps this is the same issue you were having.

pkill statsd

Gerry
  • 338
  • 2
  • 3
  • 12
0

None of the answers above helped in my case. I found this one in stackoverflow that helped me: https://stackoverflow.com/questions/15501677/deleted-empty-graphite-whisper-files-automatically-re-generating#answer-20011928

Community
  • 1
knocte
  • 347
  • 1
  • 6
  • 18
0

I agree with others that something must still be sending the old metric.

Ideally, you would address the issue on your client to resolve this, but you can also reject that metric server-side by using whisper's blacklist functionality.

To do this:

  1. ensure USE_WHITELIST=True in /opt/carbon/conf/carbon.conf
  2. add annoying metric to /opt/carbon/conf/blacklist.conf
7yl4r
  • 403
  • 5
  • 9