1

I am trying to set up a Ceph cluster. I have 4 nodes - 1 admin-node, 1 monitor and 2 object storage devices. The installation guide I am using is given at the following location: http://ceph.com/docs/master/start/quick-ceph-deploy/.

When I am trying to add the initial monitor (step 5 in the guide), I am getting the following error:

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cloud-user/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.21): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts worker-1-full
[ceph_deploy.mon][DEBUG ] detecting platform for host worker-1-full ...
[worker-1-full][DEBUG ] connection detected need for sudo
[worker-1-full][DEBUG ] connected to host: worker-1-full 
[worker-1-full][DEBUG ] detect platform information from remote host
[worker-1-full][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: Ubuntu 14.04 trusty
[worker-1-full][DEBUG ] determining if provided host has same hostname in remote
[worker-1-full][DEBUG ] get remote short hostname
[worker-1-full][DEBUG ] deploying mon to worker-1-full
[worker-1-full][DEBUG ] get remote short hostname
[worker-1-full][DEBUG ] remote hostname: worker-1-full
[worker-1-full][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[worker-1-full][DEBUG ] create the mon path if it does not exist
[worker-1-full][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-worker-1-full/done
[worker-1-full][DEBUG ] create a done file to avoid re-doing the mon deployment
[worker-1-full][DEBUG ] create the init path if it does not exist
[worker-1-full][DEBUG ] locating the `service` executable...
[worker-1-full][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=worker-1-full
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[worker-1-full][WARNIN] monitor: mon.worker-1-full, might not be running yet
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[worker-1-full][WARNIN] monitor worker-1-full does not exist in monmap
[worker-1-full][WARNIN] neither `public_addr` nor `public_network` keys are defined for monitors
[worker-1-full][WARNIN] monitors may not be able to form quorum
[ceph_deploy.mon][INFO  ] processing monitor mon.worker-1-full
[worker-1-full][DEBUG ] connection detected need for sudo
[worker-1-full][DEBUG ] connected to host: worker-1-full 
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 5
[ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 4
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 3
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 2
[ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying
[worker-1-full][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.worker-1-full.asok mon_status
[worker-1-full][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph_deploy.mon][WARNIN] mon.worker-1-full monitor is not yet in quorum, tries left: 1
[ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying
[ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum:
[ceph_deploy.mon][ERROR ] worker-1-full

"worker-1-full" is the node I am trying to set up as my monitor. The command I used is:

"ceph-deploy mon create-initial". Please help. Thanks in advance!

user3033194
  • 111
  • 1
  • 1
  • 3
  • What about the missing file: /var/run/ceph/ceph-mon.worker-1-full.asok? – Deer Hunter Jan 14 '15 at 05:06
  • That's what - I don't know how to generate it. I am doing it exactly as it is given in the link. – user3033194 Jan 14 '15 at 10:31
  • I had this problem and I got that my / (roo) is full – Mansour Apr 12 '16 at 12:33
  • I think that it's about directory permission. Would you check out your directory (/var/lib/ceph/mon and /var/run/ceph) permision as ceph:ceph user ? especially, check out file&dir. in /var/lib/ceph/mon directory. If you execute ceph-deploy command as root user, may be created /var/lib/ceph/mon directory as root user. I got it that "chown -R ceph:ceph /var/lib/ceph/mon". – neotusca Jun 10 '16 at 06:18

6 Answers6

7

I faced the same errors was able to resolve the issue by adding my other ceph node's hostname & IpAdrress and by adding "public_network ="

The sections which I tweaked in ceph.conf are:

mon_initial_members =
mon_host =
public_network =

cat /etc/ceph/ceph.conf 

[global]
fsid = 33cb5c76-a685-469e-8cdd-fee7c98c3f4d
mon_initial_members = ceph1,ceph2
mon_host = 192.168.61.39,192.168.61.40
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
public_network = 192.168.61.0/24

And the running the command:

$ ceph-deploy --overwrite-conf mon create <ceph-node>
karthik v
  • 191
  • 2
  • 9
4

It seems that names of your nodes differ from their real hostnames, so that file /var/run/ceph/ceph-mon.worker-1-full.asok has the incorrect name. I had changed name of hosts and that worked for me.

HBruijn
  • 72,524
  • 21
  • 127
  • 192
Богдан Тимофеев
  • 143
  • 5
0

Does the /var/run/ceph directory exist? If not, then Ceph probably won't be able to create the .asok (admin socket) file.

sleinen
  • 241
  • 1
  • 2
0

You could check the MON log at worker-1-full:/var/log/ceph/ceph-mon.worker-1-full.log. If the MON doesn't come up or cannot create the socket, something should be logged over there.

hennejg
  • 31
  • 2
0

In my case it was caused by the previous command failing, which was not detected by the script:

# ceph-deploy mon create-initial
...
[ceph-server-1][DEBUG ] create the init path if it does not exist
[ceph-server-1][INFO  ] Running command: systemctl enable ceph.target
[ceph-server-1][INFO  ] Running command: systemctl enable ceph-mon@ceph-server-1
[ceph-server-1][INFO  ] Running command: systemctl start ceph-mon@ceph-server-1
[ceph-server-1][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-server-1.asok mon_status
[ceph-server-1][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

With journalctl -xe I found:

Jul 05 12:12:29 ceph-server-1.novalocal ceph-mon[27403]: 2019-07-05 12:12:29.389 7f7b5aed7180 -1 rocksdb: IO error: while open a file for lock: /var/lib/ceph/mon/ceph-ceph-server-1/store.db/LOCK: Permission denied

To solve:

# chown -R ceph:ceph /var/lib/ceph/mon/ceph-ceph-server-1/store.db/
# chown    ceph:ceph /var/lib/ceph/mon/ceph-ceph-server-1/keyring

Now I have a happy Ceph.

Alberto Chiusole
  • 103
  • 4
-1

you can flush iptables with "iptables -F" command. Using this command will be disabled your iptable rules. This solved for me.

Best regards

Ali Abaszadeh

ali abaszadeh
  • 1
  • 1
    Thank you for answering and sharing your experience. Your suggestion is dangerous from a security perspective. Flushing iptables erases all the rules and leaves the service enabled. Generally flushing iptables rules is followed by reloading the rule set with updates applied. From a cyberscurity perspective, It would be irresponsible to follow such your suggestion. IF the root of the issue is that iptables is blocking the quorum, THEN the solution is to ADD iptables rules(s) to allow the traffic. – Jeter-work Mar 18 '20 at 15:21