7

(Running on an Ubuntu 10.04 64-bit server)

Yesterday, I made the mistake of starting a process (which I didn't realize was going to take several days to run) over SSH without using screen. I've spent all day today trying to figure out some way that I can pry the process's output from SSH's iron grasp so that I can reboot my client machine, but am not getting anywhere.

I tried using gdb and following the instructions at the bottom of this page, but when I run the first gdb command to create a file, I get an error message stating No symbol table is loaded. Use the "file" command. From what I gathered, this means I have to recompile the program whose output I'm trying to redirect, which of course is absolutely no help to me now that it's already running.

I also thought I might be able to use retty to redirect output to another terminal, but evidently it does not compile on 64-bit platforms.

How can I redirect this process's output to another terminal or to a file?

nonoitall
  • 173
  • 1
  • 3

3 Answers3

6

It looks like those instructions given for gdb are incorrect and incomplete in several ways.

First, you need to use 

gdb [executablefile] [pid]

so that GDB knows what the program it's connecting to actually is. I just tested this, and it can use a stripped executable. Note that while gdb is attached to the process the process will be suspended. If this process is communicating over a network, type quickly or the network connection will probably timeout.

Second, the commands given don't explain what they're doing, and the instruction that you should cd "to a directory that you want your program to write files to" is wrong, since gdb is asking the original program to execute the creat() function. The example given will create myprog.stderr and myprog.stdout files in the Current Working Directory of the running program, not in the directory you ran gdb. Use absolute pathnames here if you don't know what that program's CWD is (or look at ls -l /proc/[pid]/cwd).

Third, running with the lack of explanation, it's important to know that the first parameter to dup2() is the file descriptor number returned by the previous creat() so if this running program had several files open you might end up with an exchange like

(gdb) call creat("/home/me/myprog.stdout",0600)
$1 = 7
(gdb) call dup2(7,1)
$2 = 1
(gdb) call creat("/home/me/myprog.stderr",0600)
$3 = 8
(gdb) call dup2(8,2)
$4 = 2

When you quit gdb, it'll ask you if you want to "quit anyway (and detach it)" the answer is yes.

Finally, bg and disown are bash builtins. If you weren't using bash, then you're on your own from here. bg moves a suspended job to the background as if it were started there using somecommand &, and disown removes the program from bash's list of active programs to SIGHUP when bash exits.

DerfK
  • 19,313
  • 2
  • 35
  • 51
  • I followed your instructions, altering the commands accordingly and the commands succeeded this time! However, nothing is being written to the files that were created. Currently, ls -l /proc/[pid]/fd reads as follows (minus the irrelevant bits): 0 -> /dev/pts/0 (deleted); 1 -> /dev/pts/0 (deleted); 2 -> /dev/pts/0 (deleted); 3 -> /dev/sdb; 4 -> /root/myprog.stdout; 5 -> /root/myprog.stdout; 6 -> /root/myprog.stderr; 7 -> /root/myprog.stderr (And in case you're wondering, the process in question *is* running as root, so it should be able to write to those files.) – nonoitall Dec 16 '10 at 06:04
  • Something went wrong with `call dup2()` then, it should have duplicated the fd for "myprog.stdout" over fd 1 rather than making a new fd 5. You should still be able to fix this if you redo gdb and `call dup2(4,1)` and `call dup2(6,2)`. Hopefully the extra copy of the file being open won't interfere, but don't try `call close(5)` on the spare copy, I just tested that out and the program immedeately segfaulted. – DerfK Dec 16 '10 at 06:25
  • Ah, I see what went wrong. I was in a hurry to enter in the gdb commands and I left out the '2' in 'dup2'. All seems to be right now though as the output is getting written to the stderr file I created. There are still a couple extra descriptors open, but it doesn't seem like they're causing any harm. Thanks! – nonoitall Dec 16 '10 at 06:47
0

You cannot do anything remotely. Log into the other server using ssh and the same account that was used to start the process. You will be able to control the process from the new session.

David Harris
  • 239
  • 1
  • 4
0

You can use reredirect (https://github.com/jerome-pouiller/reredirect/).

Type

reredirect -m FILE PID

and outputs (standard and error) will be written in FILE.

reredirect README also explains how to restore original state of process, how to redirect to another command or to redirect only stdout or stderr.

Jérôme Pouiller
  • 131
  • 3