Strange tgtd behavior: Multiple instances?

Question

I'm trying to add a LUN to a running iSCSI target under CentOS 6.5. The iSCSI target is tgtd from the scsi-target-utils package. I'm seeing some pretty weird behavior and I’m a bit of a loss as to how to proceed.

First of all, it seems as if there are multiple instances of tgtd running.

[eric@<host>]~% sudo service tgtd status
tgtd (pid 8713 8712) is running...
[eric@<host>]~% sudo service tgtd reload
Updating SCSI target daemon configuration: not running     [FAILED]

However, I have several virtual machines's that are currently running off of iSCSI LUNs from this system, so I know that tgtd is running and functioning at least somewhat normally.

I grabbed the exit status of the failed reload with echo $? and it returned 7.

I then put set -x at the top of the init script and ran it again to see where it was returning 7.

[eric@<host>]~% sudo service tgtd reload   
+-zsh:19> sudo service tgtd reload
+ case "$1" in
++ id -u
+ '[' 0 = 0 ']'
+ rh_status_q
+ rh_status
+ reload
+ '[' -f /etc/sysconfig/tgtd ']'
+ . /etc/sysconfig/tgtd
+ echo -n 'Updating SCSI target daemon configuration: '
Updating SCSI target daemon configuration: + tgt-admin --update ALL -c /etc/tgt/targets.conf
+ retval=107
+ '[' 107 -eq 107 ']'
+ echo -n 'not running'
not running+ failure
+ local rc=0
+ '[' color '!=' verbose -a -z '' ']'
+ echo_failure
+ '[' color = color ']'
+ echo -en '\033[60G'
                                                           + echo -n '['
[+ '[' color = color ']'
+ echo -en '\033[0;31m'
+ echo -n FAILED
FAILED+ '[' color = color ']'
+ echo -en '\033[0;39m'
+ echo -n ']'
]+ echo -ne '\r'
+ return 1
+ '[' -x /bin/plymouth ']'
+ /bin/plymouth --details
+ return 0
+ echo

+ return 7
+ exit 7

It looks like its failing in the reload function. Who would have thought? Specifically, this bit of code.

TGTD_CONFIG=/etc/tgt/targets.conf
...
tgt-admin --update ALL -c $TGTD_CONFIG >/dev/null 2>&1
retval=$?
if [ "$retval" -eq 107 ] ; then
        echo -n $"not running"
        failure
        echo
        return 7

I then ran that command:

[eric@<host>]~% sudo tgt-admin --update ALL -c /etc/tgt/targets.conf
tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
Command:
        tgtadm -C 0 --lld iscsi --op new --mode target --tid 1 -T <redacted iqn>
exited with code: 107.

This is where I'm at a loss. I Googled that error without much success -- it looks like I'm going to have to kill the process and restart it. Does anyone know what this could be indicative of, and why tgtadm can’t find a process that is clearly running? And better yet, does anyone know if I can fix this without killing the process to avoid interrupting service?

Here’s some other relevant info:

[eric@<host>]~% sudo ps waux | grep tgt
root      8712  0.0  0.0 1163084 2648 ?        Ssl  Oct08   0:44 tgtd
root      8713  0.0  0.0  14888   500 ?        S    Oct08   0:03 tgtd
eric     20075  0.0  0.0 103244   900 pts/4    S+   14:09   0:00 grep tgt
[eric@<host>]~% uname -a
Linux <host> 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

Try to update RPC on your system, then restart tgtd. In my case this helped:

yum update libtirpc