0

I'm trying to start rabbitmq, yet it seems to crash upon start and producing errors:

handshake_error,opening,amqp_error,internal_error and refused for user:

rabbitmq    | 2019-08-14 15:10:16.053 [info] <0.244.0> 
rabbitmq    |  Starting RabbitMQ 3.7.16 on Erlang 22.0.7
rabbitmq    |  Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbitmq    |  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbitmq    | 
rabbitmq    |   ##  ##
rabbitmq    |   ##  ##      RabbitMQ 3.7.16. Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbitmq    |   ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbitmq    |   ######  ##
rabbitmq    |   ##########  Logs: <stdout>
rabbitmq    | 
rabbitmq    |               Starting broker...
rabbitmq    | 2019-08-14 15:10:16.054 [info] <0.244.0> 
rabbitmq    |  node           : rabbit@rabbitmq
rabbitmq    |  home dir       : /var/lib/rabbitmq
rabbitmq    |  config file(s) : /etc/rabbitmq/rabbitmq.conf
rabbitmq    |  cookie hash    : AIlteC+QMYJQCC1CZZToPg==
rabbitmq    |  log(s)         : <stdout>
rabbitmq    |  database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq
rabbitmq    | 2019-08-14 15:10:16.065 [info] <0.244.0> Running boot step pre_boot defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.065 [info] <0.244.0> Running boot step rabbit_core_metrics defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.066 [info] <0.244.0> Running boot step rabbit_alarm defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.105 [info] <0.252.0> Memory high watermark set to 12065 MiB (12651329945 bytes) of 30163 MiB (31628324864 bytes) total
rabbitmq    | 2019-08-14 15:10:16.194 [info] <0.262.0> Enabling free disk space monitoring
rabbitmq    | 2019-08-14 15:10:16.194 [info] <0.262.0> Disk free limit set to 50MB
rabbitmq    | 2019-08-14 15:10:16.198 [info] <0.244.0> Running boot step code_server_cache defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.198 [info] <0.244.0> Running boot step file_handle_cache defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.199 [info] <0.267.0> Limiting to approx 1048476 file handles (943626 sockets)
rabbitmq    | 2019-08-14 15:10:16.199 [info] <0.268.0> FHC read buffering:  OFF
rabbitmq    | 2019-08-14 15:10:16.199 [info] <0.268.0> FHC write buffering: ON
rabbitmq    | 2019-08-14 15:10:16.201 [info] <0.244.0> Running boot step worker_pool defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.201 [info] <0.245.0> Will use 8 processes for default worker pool
rabbitmq    | 2019-08-14 15:10:16.201 [info] <0.245.0> Starting worker pool 'worker_pool' with 8 processes in it
rabbitmq    | 2019-08-14 15:10:16.202 [info] <0.244.0> Running boot step database defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.203 [info] <0.244.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step database_sync defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step codec_correctness_check defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step external_infrastructure defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_registry defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_queue_location_random defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_event defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_priority_queue defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Priority queues enabled, real BQ is rabbit_variable_queue
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step kernel_ready defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_sysmon_minder defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_epmd_monitor defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.281 [info] <0.244.0> Running boot step guid_generator defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.286 [info] <0.244.0> Running boot step rabbit_node_monitor defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.286 [info] <0.324.0> Starting rabbit_node_monitor
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step delegate_sup defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step rabbit_memory_monitor defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step core_initialized defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step upgrade_queues defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_connection_tracking defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_exchange_parameters defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_policies defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_policy defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_queue_location_validator defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_vhost_limit defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.309 [info] <0.244.0> Running boot step recovery defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.346 [info] <0.578.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
rabbitmq    | 2019-08-14 15:10:16.348 [error] <0.579.0> CRASH REPORT Process <0.579.0> with 0 neighbours crashed with reason: no match of right hand value {error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}} in rabbit_recovery_terms:open_table/1 line 197
rabbitmq    | 2019-08-14 15:10:16.349 [error] <0.578.0> Unable to recover vhost <<"/">> data. Reason {badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{child,undefined,rabbit_recovery_terms,{rabbit_recovery_terms,start_link,[<<"/">>]},transient,30000,worker,[rabbit_recovery_terms]}}}}
rabbitmq    |  Stacktrace [{rabbit_recovery_terms,start,1,[{file,"src/rabbit_recovery_terms.erl"},{line,53}]},{rabbit_queue_index,start,2,[{file,"src/rabbit_queue_index.erl"},{line,502}]},{rabbit_variable_queue,start,2,[{file,"src/rabbit_variable_queue.erl"},{line,483}]},{rabbit_priority_queue,start,2,[{file,"src/rabbit_priority_queue.erl"},{line,92}]},{rabbit_amqqueue,recover,1,[{file,"src/rabbit_amqqueue.erl"},{line,238}]},{rabbit_vhost,recover,1,[{file,"src/rabbit_vhost.erl"},{line,72}]},{rabbit_vhost_process,init,1,[{file,"src/rabbit_vhost_process.erl"},{line,56}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,554}]}]
rabbitmq    | 2019-08-14 15:10:16.350 [error] <0.578.0> CRASH REPORT Process <0.578.0> with 0 neighbours exited with reason: no match of right hand value {error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,...}]}]},...}} in gen_server2:init_it/6 line 589
rabbitmq    | 2019-08-14 15:10:16.350 [error] <0.576.0> Supervisor {<0.576.0>,rabbit_vhost_sup_wrapper} had child rabbit_vhost_process started with rabbit_vhost_process:start_link(<<"/">>) at undefined exit with reason {badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib..."},...]}]},...}}} in context start_error
rabbitmq    | 2019-08-14 15:10:16.350 [warning] <0.244.0> Unable to initialize vhost data store for vhost '/'. The vhost will be stopped for this node.  Reason: {shutdown,{failed_to_start_child,rabbit_vhost_process,{badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{child,undefined,rabbit_recovery_terms,{rabbit_recovery_terms,start_link,[<<"/">>]},transient,30000,worker,[rabbit_recovery_terms]}}}}}}
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step empty_db_check defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step rabbit_looking_glass defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step background_gc defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step connection_tracking defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@rabbitmq
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@rabbitmq
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step routing_ready defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step pre_flight defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step notify_cluster defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step networking defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.354 [warning] <0.589.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
rabbitmq    | 2019-08-14 15:10:16.355 [info] <0.603.0> started TCP listener on [::]:5672
rabbitmq    | 2019-08-14 15:10:16.355 [info] <0.244.0> Running boot step direct_client defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.449 [info] <0.8.0> Server startup complete; 0 plugins started.
rabbitmq    |  completed with 0 plugins.
rabbitmq    | 2019-08-14 15:10:18.598 [info] <0.607.0> accepting AMQP connection <0.607.0> (10.0.0.69:44814 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.599 [info] <0.610.0> accepting AMQP connection <0.610.0> (10.0.0.69:44812 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.600 [info] <0.614.0> accepting AMQP connection <0.614.0> (10.0.0.69:44820 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.600 [info] <0.617.0> accepting AMQP connection <0.617.0> (10.0.0.69:44818 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.656 [error] <0.607.0> Error on AMQP connection <0.607.0> (10.0.0.69:44814 -> 10.0.0.138:5672, vhost: 'none', user: 'guest', state: opening), channel 0:
rabbitmq    |  {handshake_error,opening,
rabbitmq    |                  {amqp_error,internal_error,
rabbitmq    |                              "access to vhost '/' refused for user 'guest': vhost '/' is down",
rabbitmq    |                              'connection.open'}}

Please advise.

alexus
  • 12,342
  • 27
  • 115
  • 173
  • 2
    Something corrupted your vhost `/` files in `/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L`. Repair or restore your installation or delete the corrupted files and restart, that's for you to decide. – Lenniey Aug 14 '19 at 15:30
  • @Lenniey thanks for your comment, how does one repair? my "installation" runs from Docker (nothing to restore). if I delete that directory, is it like I clean up all data from rabbitmq? – alexus Aug 14 '19 at 15:39
  • I'd start by moving (not deleting!) the `recovery.dets` file and restart. This won't work, probably. Then move the hashed folder `628WB79CIFDYO9LJI6DKMI09L` away and restart. But it all depends if you have critical data in your broker/ vhost or not. If not: I'd probably just recreate the vhost. Be sure to always try to stop the broker gracefully, not just kill the container/ process. – Lenniey Aug 14 '19 at 15:48

0 Answers0