mv fails but cp succeeds on nfs mount

Question

I am trying to move a file from a local directory to an nfs mounted directory on a CentOS 7 server. The export is provided by FreeNAS.

Both directories are owned by the current user with at least 755 (nfs shows as 777 but I'm not sure I believe it).

My fstab looks like this

host:/path/to/export /mnt/nfs         nfs     defaults        0 0

I cannot move the file

mv /local/file /mnt/nfs/file
mv: cannot create regular file 'file': Operation not permitted

However I can copy and remove it

cp /local/file /mnt/nfs/file
rm /local/file

Output of mount

host:/path/to/export on /mnt/nfs type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=host,mountvers=3,mountport=908,mountproto=udp,local_lock=none,addr=host)

Directory permissions on the client

ls -ld /local /mnt/nfs
drwxrwxrwx. 15 user user 17 Nov 28 08:32 /mnt/nfs/
drwxrwxrwx. 2  root root 17 Nov 29 12:20 /local

After further investigation this seems to be regarding changing permissions. mv takes the permissions of the file with it, however cp creates a new file which inherits permissions from the parent directory. It appears the nfs mount does not allow me to chown files even if I am root or the owner of the files. So now my question is how do I allow the changing of permissions, or ignore the permissions given in the mv.

Uhm, nothing interesting in it :( ... Did you have access to the server-side logs and/or to the `/etc/exports` file? Another thing: does it change anything to `mv` a file as a normal user (**not** root) ? — shodanshok, Nov 24 '16 at 22:14
sounds like you are missing **no_root_squash** export option — kofemann, Nov 25 '16 at 10:50
I am not root when doing this. I am the user who own the directory (UID 1001). The mount is being served by my FreeNAS box. — Jacob Tomlinson, Nov 28 '16 at 08:47
Can you provide the status of `getenforce` if its enforcing retry the work and provide the output of `ausearch -ts recent -m avc`. — Matthew Ife, Nov 28 '16 at 15:58
@MatthewIfe Yes it is enforcing, however when running your `ausearch` command after getting an `Operation not permitted` it returns ``. — Jacob Tomlinson, Nov 29 '16 at 10:36
@MatthewIfe also if I disable SELinux I still get the same errors. — Jacob Tomlinson, Nov 29 '16 at 10:37
Can you supply the `ls -ld` output of `/local` and `/mnt/nfs` — Matthew Ife, Nov 29 '16 at 11:13
@MatthewIfe `drwxrwxrwx. 15 user user 17 Nov 28 08:32 /mnt/nfs/` `drwxrwxrwx. 2 root root 17 Nov 29 12:20 /local` — Jacob Tomlinson, Nov 29 '16 at 12:21
If it's a permissions thing, do you get the same error with `cp -p` instead of `cp`? Can you provide `ls -l /local/file` please (I'm interested in the permissions set) — roaima, Nov 30 '16 at 23:24
Can you try this on the NFS server? `all_squash,anongid=0,anonuid=0` — mzhaase, Dec 01 '16 at 15:49
@roaima `cp -p` succeeds but the permissions are ignored. The file I'm copying is `-rw-rw-r--. 1 jacob jacob 6 Nov 28 08:25 /tmp/test` and after the `cp -p` it is written as `-rwxrwxrwx. 1 user user 6 Nov 28 08:25 test`. — Jacob Tomlinson, Dec 02 '16 at 08:10
@mzhaase In FreeNAS the 'Map All User' and 'Map All Group' settings are already set to uid 1001 (the owner of the files). Which I think achieves the same thing. — Jacob Tomlinson, Dec 02 '16 at 08:28
@JacobTomlinson this is really strange. Across filesystems `mv a b` is equivalent to `cp -p a b && rm a` — roaima, Dec 02 '16 at 12:34
@roaima that's exactly what I though. There must be some subtle difference. — Jacob Tomlinson, Dec 05 '16 at 09:01

score 3 · Answer 1 · answered Dec 01 '16 at 07:23

Are your password / group databases in sync between client and server?
Do you see any files on nfs mounted directory on nfs client as owned by

nobody nobody
Can you post output of nfsidmap -d from server and client? Hint - when using NFSV4 outputs should match.

Most likely you are running into discrepancy between UID / GID on the NFS server and NFS client. I will show how this works based on a simple example.

Let's say you are sharing on your NFS client /nfs_share like this. Notice that nfs_share is writeable by anyone(777).

[root@nfsserver nfs_share]# cat /etc/exports
/nfs_share      192.168.0.52(rw,no_root_squash) 

[root@nfsserver nfs_share]# ls -ld /nfs_share
drwxrwxrwx. 2 root root 4096 Nov 30 23:55 /nfs_share

And mounting on your NFS Client like this

mount 192.168.0.51:/nfs_share /mnt

Now you have on nfs server user called dmitry

[root@nfsserver nfs_share]# getent passwd|grep dmitry
dmitry:x:500:500::/home/dmitry:/bin/bash
[root@nfsserver nfs_share]# getent group|grep dmitry
dmitry:x:500:

And on your nfs client you have user helen

[root@nfsclient ~]# getent passwd|grep helen
helen:x:500:500::/home/helen:/bin/bash
[root@nfsclient ~]# getent group|grep helen
helen:x:500:

Notice that despite those are different users - they have same UID and GID. So what happens if I touch as user helen file on nfs share?

[helen@nfsclient mnt]$ touch helen_client
[helen@nfsclient mnt]$ ls -lrt
[helen@nfsclient mnt]$ ls -lrt
total 0
-rw-rw-r--. 1 nobody nobody 0 Nov 30 23:58 helen_client

On NFS client this new file will show up as owned by nobody nobody. This is because nfsidmap can't map client_user_name@domain to server_user_name @domain. And now moment of truce. Let's check what's the file owner on the nfs server.

[root@nfsserver nfs_share]# ls -rlt
total 0
-rw-rw-r--. 1 dmitry dmitry 0 Nov 30 23:58 helen_client

Surprised yet?
Yet there is nothing strange actually. This works as expected.
NFS server can't map user helen, but what it received is UID and GID. So it created file (since folder is world writeable) with UID 500 and GID 500, which is mapped to dmitry:dmitry on NFS server.

Now let's say we have other user who's UID / GID and names match between server and client

[root@nfsserver mnt]# getent passwd|grep angelina
angelina:x:501:501::/home/angelina:/bin/bash
[root@nfsserver mnt]# getent group|grep angelina
angelina:x:501:

[angelina@nfsclient mnt]$ getent passwd|grep angelina
angelina:x:501:501::/home/angelina:/bin/bash
[angelina@nfsclient mnt]$ getent group|grep angelina
angelina:x:501:

And if I touch file on nfs client as user angelina - I will see correct user name / group on both Server and Client

[angelina@nfsclient mnt]$ pwd
/mnt
[angelina@nfsclient mnt]$ touch angelina_1
[angelina@nfsclient mnt]$ ls -l
total 0
-rw-rw-r--. 1 angelina angelina 0 Dec  1  2016 angelina_1
-rw-rw-r--. 1 nobody   nobody   0 Dec  1 00:16 helen_1

[root@nfsserver nfs_share]# pwd
/nfs_share
[root@nfsserver nfs_share]# ls -l
total 0
-rw-rw-r--. 1 angelina angelina 0 Dec  1 00:27 angelina_1
-rw-rw-r--. 1 dmitry   dmitry   0 Dec  1 00:16 helen_1

Bottom line is for NFSV4 to work correctly you need to have

Server and client password / group database in sync. Preferably use ldap.
client and server should agree on the common domain name nfsidmap -d

The `mount` flags show the OP is using NFSv3 so some of this answer is not really relevant here — roaima, Dec 01 '16 at 08:35
`nfsidmap` is not relevant, but on nfsv3 same id mapping is performed by `rpcidmapd`. Actually sorry, that's Linux flavor dependent. Still nfsv4 applicable. So yes, some of the answers are not applicable to v3, however what I described about importance of syncronized password dbs is till valid. — Dmitry Zayats, Dec 01 '16 at 14:58
Both systems have a user with uid 1001 and gid 1001. However the username is different. All files are owned by uid 1001 on the host system. — Jacob Tomlinson, Dec 02 '16 at 08:06
Sadly this answer didn't solve my problem. But it's a well written and thought out answer and the bounty period is over, so I'm going to award it to you. — Jacob Tomlinson, Dec 06 '16 at 09:35
What release of FreeNas do you have? Is it stable 9.10 or beta? — Dmitry Zayats, Dec 07 '16 at 07:51

mv fails but cp succeeds on nfs mount

1 Answers1