Using MCollective (2.2.3) with RabbitMQ (3.1.5) on RedHat 6.5, MCollective stops responding after a first successful ping.
When I restart RabbitMQ, it works again.
root@rabbitmqserver# /etc/init.d/rabbitmq-server restart
Restarting rabbitmq-server: SUCCESS
rabbitmq-server.
root@rabbitmqserver# rabbitmqctl list_consumers -p /mcollective | grep directed
amq.gen-1amX9Z9w6jBKNP7IQrMfjg <rabbit@srv98.3.549.0> Q_/exchange/mcollective_directed/srv114 false
amq.gen-dxFvTuxvQZOU0d1e7XuJJg <rabbit@srv98.3.870.0> Q_/exchange/mcollective_directed/srv98 false
amq.gen-lwttnRCCTON9hgPg7X2zZw <rabbit@srv98.3.1047.0> Q_/exchange/mcollective_directed/srv99 false
amq.gen-pG4o5csOGcam9mxvIQvQ7g <rabbit@srv98.3.340.0> Q_/exchange/mcollective_directed/srv57 false
amq.gen-xtRFm6aAoeV2mDO9dTAyyA <rabbit@srv98.3.545.0> Q_/exchange/mcollective_directed/srv115 false
user@mco-client$ mco ping
srv114 time=66.49 ms
srv57 time=69.34 ms
---- ping statistics ----
2 replies max: 69.34 min: 66.49 avg: 67.91
user@mco-client$ mco ping
---- ping statistics ----
No responses received
root@rabbitmqserver# rabbitmqctl list_consumers -p /mcollective | grep directed
amq.gen-1amX9Z9w6jBKNP7IQrMfjg <rabbit@srv98.3.549.0> Q_/exchange/mcollective_directed/srv114 false
amq.gen-dxFvTuxvQZOU0d1e7XuJJg <rabbit@srv98.3.870.0> Q_/exchange/mcollective_directed/srv98 false
amq.gen-lwttnRCCTON9hgPg7X2zZw <rabbit@srv98.3.1047.0> Q_/exchange/mcollective_directed/srv99 false
amq.gen-pG4o5csOGcam9mxvIQvQ7g <rabbit@srv98.3.340.0> Q_/exchange/mcollective_directed/srv57 false
amq.gen-xtRFm6aAoeV2mDO9dTAyyA <rabbit@srv98.3.545.0> Q_/exchange/mcollective_directed/srv115 false
If I try using another command (such as package), it fails right after the discovery (after restarting RabbitMQ that is):
user@mco-client$ mco package status bash
Do you really want to operate on packages unfiltered? (y/n): y
| [ > ] 0 / 2
[... stalled ...]
After a while, I start seeing this in my MCollective server logs:
E, [2014-03-18T11:51:01.332969 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T11:55:35.759653 #24456] ERROR -- : agents.rb:138:in `dispatch' Execution of discovery failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T11:55:35.760040 #24456] ERROR -- : agents.rb:139:in `dispatch' /usr/lib/ruby/gems/1.8/gems/stomp-1.2.2/lib/stomp/connection.rb:331:in `publish'
/usr/libexec/mcollective/mcollective/connector/rabbitmq.rb:184:in `publish'
/usr/lib/ruby/site_ruby/1.8/mcollective/message.rb:230:in `publish'
/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
/usr/lib/ruby/site_ruby/1.8/mcollective/message.rb:218:in `publish'
/usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:129:in `reply'
/usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:83:in `agentmsg'
/usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:132:in `dispatch'
/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
/usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:125:in `dispatch'
/usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `initialize'
/usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `new'
/usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `dispatch'
/usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:82:in `agentmsg'
/usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:55:in `run'
/usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:50:in `loop'
/usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:50:in `run'
/usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:30:in `daemonize_runner'
/usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:13:in `daemonize'
/usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `fork'
/usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `daemonize'
/usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:20:in `daemonize_runner'
/usr/sbin/mcollectived:43
E, [2014-03-18T11:56:01.336954 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:01:01.343415 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:06:01.347218 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:11:01.354069 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
even though the machine is still listed as a consumer on the RabbitMQ server.
What could be going wrong here?
