I'm trying to setup a NFS client on a new CentOS 6.6 server to an existing NFS server and am having problems that I cannot figure out. When starting rpcbind
/nfs
I get:
# /etc/init.d/rpcbind start
# /etc/init.d/nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: Cannot register service:
RPC: Unable to receive; errno = Connection refused
rpc.rquotad: unable to register (RQUOTAPROG, RQUOTAVERS, udp).
[FAILED]
Starting NFS mountd: [ OK ]
Starting NFS daemon: [ OK ]
rpcinfo -p
works fine
# rpcinfo -p
program vers proto port service
100000 4 tcp 111 portmapper
.. snip
100021 4 tcp 47627 nlockmgr
but rpcinfo -p 127.0.0.1
(or localhost) returns an error:
# rpcinfo -p 127.0.0.1
rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused
Checking the ports with (netstat -an) it seems the portmap/rpcbind is not listening on port 111. Nothing is listening on port 111 and rpcbind is listening on port 3421 (edit: not random like I originally thought).
Things I've checked or tried include:
- Disabling selinux (was not enabled)
- Clearing
/etc/hosts.deny
- Clearing
iptables
(it had no rules to start) - Check logs (found nothing relevant)
- Restarted server (no effect)
- Checked on CentOS 5 servers (portmap is listening there on port 111)
- Checked Google: A lot of similar error messages but nothing specifically about rpcbind not listening on port 111 (and none the solutions worked anyways).
- 'telnet' to port 111 (timed out which confirms that nothing is listening there)
- Checked 127.0.0.1/localhost settings (they work fine)
At this point I don't really have any options or things left to check and need some ideas. I'll pretty much try anything at this point as the server is relatively useless without NFS.
Update 1:
Running rpcinfo -s localhost
on the client works but rpcinfo -s server
does not work. I'm wondering if there is an issue with having the client be CentOS 6 and the server be CentOS 5.
Update 2:
Output from showmount
is identical on the new client and the NFS server:
# showmount -e 10.2.212.10
Export list for 10.2.212.10:
/shared/uesp/maps 67.205.112.104
/shared/uesp/skins 67.205.112.104
/shared/phpsessions 10.2.212.30,[...snip...],10.2.212.11
/shared/uesp/filecache 10.2.212.30,[...snip...],10.2.212.11
/shared/uesp/wikiimages 10.2.212.30,[...snip...],10.2.212.11
/shared/eqwiki/filecache 10.2.212.30,[...snip...],10.2.212.11
/shared/eqwiki/wikiimages 10.2.212.30,[...snip...],10.2.212.11
/shared/davewiki/filecache 10.2.212.30,[...snip...],10.2.212.11
/shared/davewiki/wikiimages 10.2.212.30,[...snip...],10.2.212.11
I've snipped the long list of IPs, but 10.2.212.30 is the new client. Trying to mount a NFS share on the new client results in this:
# mount -v 10.2.212.10:/shared/phpsessions /mnt/phpsessions
mount: no type was given - I'll assume nfs because of the colon
mount.nfs: timeout set for Thu Jan 15 18:53:43 2015
mount.nfs: trying text-based options 'vers=4,addr=10.2.212.10,clientaddr=10.2.212.30'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=10.2.212.10'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Unable to receive
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Unable to receive - Connection refused
mount.nfs: trying text-based options 'vers=4,addr=10.2.212.10,clientaddr=10.2.212.30'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=10.2.212.10'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Unable to receive
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Unable to receive - Connection refused
...
The last few lines repeat until it times out.
Update 3:
So if I use iptables
to redirect port 111 to the port rpcbind
is using on the client then I can get rpcinfo -p localhost
to work on the client. However, this doesn't do anything with the mount issues. I've tried forcing the NFS to use v2 like:
# mount -v -t nfs -o nfsvers=3 10.2.212.10:/shared/phpsessions /mnt/phpsessions
but that results in the same error message as above. The fact that when I run on the client I get:
# rpcinfo -s server
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused
which is the same error message from mount. I'm guessing that mount on the CentOS 6 client is trying to use the "new" rpcbind on the CentOS 5 server which is running the "old" portmap so it is failing. I would assume it should be possible to mix CentOS versions like this somehow.
Update 4:
I created a test NFS share on the client and was able to successfully mount it on the server. This seems to match the theory that it is something to do with a portmap/rpcbind mismatch. The client can act as a NFS server as rpcbind (rpc v3/4) is backwards compatible with portmap (rpc v2) but portmap is not forwards compatible with rpcbind.
Update 5 (Possible Solution)
Looking at the client more carefully I noticed rpcbind
seems to always be on port 3421 (I thought it was random initially). On the server I used iptables
to forward port 3421 to port 111. After this I was able to successfully mount the a NFS share from the server on the client. This may be a valid solution since I'll only have a mix of CentOS 5/6 servers for a few weeks as I upgrade all servers to CentOS 6.
Unless a better answer comes up I'll answer my own question in a few days.