Munin mysql plugins results inconsistent, with IPC::Sharelite store() error: Identifier removed

Question

I've recently setup a munin-node on a CentOS server. All was working fine until I tried to add the apache plugin (which works fine).

For some odd reason, the mysql plugins for munin that used to work ceased to work... I'm now getting a weird error whenever I'm running the plugin with munin-run. For instance

munin-run mysql_files_tables

returns me

IPC::ShareLite store() error: Identifier removed at /usr/lib/perl5/vendor_perl/5.8.8/Cache/SharedMemoryBackend.pm line 156

but sometimes it will also return

table_open_cache.value 64

Open_files.value 58

Open_tables.value 64

Opened_tables.value 19341

but after a while it will revert to the previous error.

I do not have any knowledge about the IPC or the ShareLite library so I don't really know were to start looking. Since it is a module related to shared memory, I tried tracking down shared memory segments with ipcs without much success.

I haven't yet rebooted the machine as it is used for many projects (I'd obviously like to be able to diagnose the problem without requiring a restart if it was possible).

Has anyone faced this problem? (a quick search on google didn't present any relevant help)

Thanks for the help!

score 3 · Accepted Answer · answered Oct 03 '13 at 02:49

3

I've solved my problem, but I haven't found the exact cause/source yet, so I'm still open to hear from others.

Basically, the mysql plugins of munin are written in perl, use IPC::ShareLite.

Using strace I was able to determine the problem is that the application gets stuck on a semop on a particular semaphore id. Using ipcs and ipcrm, I was able to remove the semaphore and get the plugin running correctly again.

strace gives me the semid as the first argument passed to semop. With ipcrm -s semid, I remove the semaphore from the system, which fixes the issue.

Thus, I'm led to believe something went wrong with the semaphore (contention maybe, but I don't see from what...).

answered Oct 03 '13 at 02:49

tomzx

163
9

For a more detailed "explanation", you can check the post I wrote on my blog http://www.tomrochette.com/munin-mysql-and-semaphore-how-to-deal-with-the-identifier-removed-error/. – tomzx Oct 03 '13 at 02:50
I've the same problem on a debian/jessie (and fyi your javascript in the linked post is killing both my chromium and firefox) – Alex Dec 06 '16 at 21:57
@Alex, thanks for pointing that out. The problem should be resolved now. If not, let me know! – tomzx Dec 16 '16 at 17:35
yep, both working now – Alex Dec 16 '16 at 19:23

score 0 · Answer 2 · edited Dec 20 '15 at 20:22

Came across this problem few days ago. My solution was to replace Cache::SharedMemoryCache with Cache::FileCache in mysql_ plugin as shown below:

*** mysql_      2015-12-20 15:13:12.000000000 +0100
--- mysqlnew_   2015-12-20 15:12:43.000000000 +0100
*************** use Munin::Plugin;
*** 143,149 ****
  my $has_cache;

  BEGIN {
!     eval 'require Cache::SharedMemoryCache';
      $has_cache = $@ ? 0 : 1;
  }

--- 143,149 ----
  my $has_cache;

  BEGIN {
!     eval 'require Cache::FileCache';
      $has_cache = $@ ? 0 : 1;
  }

*************** my %cache_options = (
*** 172,178 ****
  my $shared_memory_cache ;
  if ($has_cache)
  {
!   $shared_memory_cache = Cache::SharedMemoryCache->new(\%cache_options)
      or die("Couldn't instantiate SharedMemoryCache");
  }

--- 172,178 ----
  my $shared_memory_cache ;
  if ($has_cache)
  {
!   $shared_memory_cache = Cache::FileCache->new(\%cache_options)
      or die("Couldn't instantiate SharedMemoryCache");
  }

Munin mysql plugins results inconsistent, with IPC::Sharelite store() error: Identifier removed

2 Answers2