I recently started experiencing problems when running mv
on my ReadyNAS 312 system. It appears the call blocks immediately, i.e. when I run strace
on it it appears to get stuck before doing much:
open("[...]", O_RDONLY|O_NOFOLLOW) = 3
fstat(3, {st_mode=S_IFREG|0666, st_size=287873154, ...}) = 0
open("[...]", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
ioctl(4, BTRFS_IOC_CLONE^C^C^Z
[1] + 7789 suspended strace mv .
The full strace
output is at [0].
Conversely a cp
works without problems (the output is at [1]).
While mv
hangs it does use 100% CPU according to top
and according to ps r -A
it's running:
% ps r -A
PID TTY STAT TIME COMMAND
2490 ? R 0:45 [btrfs-transacti]
20257 pts/0 R+ 1:15 mv [...]
21097 pts/1 R+ 0:00 ps r -A
However as can see above after a few seconds the process [btrfs-transacti] joins it (using 100% CPU as well).
The output of cat /proc/20257/status
is
% cat /proc/20257/status
Name: mv
State: R (running)
Tgid: 20257
Ngid: 0
Pid: 20257
PPid: 20254
TracerPid: 20254
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 64
Groups: 0
NStgid: 20257
NSpid: 20257
NSpgid: 20254
NSsid: 4613
VmPeak: 24084 kB
VmSize: 24084 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 1616 kB
VmRSS: 1616 kB
VmData: 220 kB
VmStk: 136 kB
VmExe: 124 kB
VmLib: 2516 kB
VmPTE: 64 kB
VmPMD: 12 kB
VmSwap: 0 kB
Threads: 1
SigQ: 0/7915
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
Cpus_allowed: f
Cpus_allowed_list: 0-3
Mems_allowed: 1
Mems_allowed_list: 0
voluntary_ctxt_switches: 308
nonvoluntary_ctxt_switches: 467
With the number of nonvoluntary context switches rising slowly, while the voluntary context switches stay constant.
The disk system is what Netgear calls X-RAID, however it's basically RAID-1. The kernel is
% uname -a
Linux readynas312 4.1.30.x86_64.1 #1 SMP Thu Sep 22 16:30:32 PDT 2016 x86_64 GNU/Linux
Please let me know if there's any more information I should provide.
[0] https://gist.github.com/dinyar/6e32fae19d38fd66f942dd70ee6cda8c
[1] https://gist.github.com/dinyar/4c40ff7bc0fca986a1ea875287810928