Is there a way to measure/report the overall latency in a tunneled SSH session?
My particular setup is:
I'm interested in seeing the latency between the console on my local machine and the final machine on which I have the session open.
See the sshping utility: https://github.com/spook/sshping
Example:
# sshping 172.16.47.143
--- Login: 1725 msec
--- Minimum Latency: 4046 nsec
--- Median Latency: 11026 nsec +/- 0 std dev
--- Average Latency: 178105 nsec
--- Maximum Latency: 8584886 nsec
--- Echo count: 1000 Bytes
--- Transfer Speed: 11694919 Bytes/second
# sshping --help
Usage: sshping [options] [user@]addr[:port]
SSH-based ping that measures interactive character echo latency
and file transfer throughput. Pronounced "shipping".
Options:
-c --count NCHARS Number of characters to echo, default 1000
-e --echocmd CMD Use CMD for echo command; default: cat > /dev/null
-h --help Print usage and exit
-i --identity FILE Identity file, ie ssh private keyfile
-p --password PWD Use password PWD (can be seen, use with care)
-r --runtime SECS Run for SECS seconds, instead of count limit
-t --tests e|s Run tests e=echo s=speed; default es=both
-v --verbose Show more output, use twice for more: -vv
I skipped some steps suggested by @nicht-verstehen:
python -m timeit --setup 'import subprocess; p = subprocess.Popen(["ssh", "user@host", "cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0)' 'p.stdin.write(b"z"); assert p.stdout.read(1) == b"z"'
Where
python -m timeit executes the timeit Python module.
The -s/--setup option tells timeit which statement(s) to execute before each repeat.
subprocess.Popen(["ssh", "user@host", "cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0) launches ssh - executing cat on your host - as a child/subprocess, redirecting its IO streams to Python file-like objects. bufsize=0 makes sure no IO is buffered, which may cause IO waits.
And for each loop:
p.stdin.write(b"z") writes a single byte to the child (in turn through ssh to cat).
p.stdout.read(1) reads a single byte from the child. The assertion around it tests whether that byte is the same as the one you wrote to it.
Boils down to the same thing, but skips creating the named pipes (mkfifo). I noticed that the more loops you run, the faster each loop is. Control it using -n/--number: python -m timeit --number 50 ...
Was trying to do this myself and came up with this. Probably there is a simpler way, but this is what I came up with.
First, prepare pipes which will be used to make the benchmarking program communicate through the SSH connection.
$ mkfifo /tmp/up /tmp/down
Then establish a connection in ControlMaster mode without executing any remote command. This allows us to authenticate with the host interactively. After the connection is established, SSH will just "hang" here in foreground.
$ ssh $HOST -N -M -S /tmp/control
In a parallel terminal, execute remote cat in background. It will be our echo server whose latency we will measure. Inputs and outputs are connected to FIFOs:
$ ssh $HOST -S /tmp/control cat </tmp/up >/tmp/down &
And then benchmark a small program (send a byte to up FIFO, receive a byte from down FIFO):
$ python -m timeit -s 'import os' \
'os.write(3, "z"); z=os.read(4, 1); assert z=="z", "got %s" % z' \
3>/tmp/up 4</tmp/down
10 loops, best of 3: 24.6 msec per loop
The measure obviously shows the round-trip latency. If you need to repeat the experiment, run the last two commands (ssh and python) again.
If something seems to go wrong, use SSH -v flag to get more debugging output.
My idea was to use terminal query sequences for this; the advantage is that this can simply be run on the server, the disadvantage is that it measures terminal latency, not only the latency of the connection (but i guess, usually, your terminal's response time will be negligible compared to network delays)—maybe this is even what you mean with overall latency
#!/usr/bin/env python3
# Measure terminal latency (round-trip time) using "Query device code" command
from sys import stdin, stdout
import tty, termios, time
oldtty = termios.tcgetattr(stdin)
try:
tty.setcbreak(stdout)
runs = 10
results = []
for _ in range(runs):
stdout.write("\x1b[c")
stdout.flush()
t1 = time.time()
ch = stdin.read(1)
assert(ch == '\x1b')
t2 = time.time()
while stdin.read(1) != 'c': # swallow rest of report
continue
latency = (t2 - t1) * 1000
print('%.1fms' % (latency))
results.append(latency)
print()
print('avg: %.1fms min: %.1fms max: %.1fms' % (
sum(results) / runs,
min(results),
max(results)))
finally:
termios.tcsetattr(stdin, termios.TCSADRAIN, oldtty)
(This uses "Device Code Query", all terminals that I've tried respond to this: xterm, alacritty, gnome-terminal. I cannot myself try this on MacOS. So YMMV, if this one doesn't, another of the requests that interrogate some information about the terminal might work, see http://web.archive.org/web/20190624214929/http://www.termsys.demon.co.uk/vtansi.htm)
I wrote a Bash script that repeatedly calculates ping-pong RTT/latency. The basic concepts are similar to what others have suggested. The main part does something along these lines:
fifo="fifo"; mkfifo "$fifo"; sleep=1; now=(date +%s%N)
( echo 'o' >"$fifo"; ) & # to trigger the following loop
cat 0<> "$fifo" | ssh $sshargs $ssh cat \
| while read -r R ; do
echo $(( $(date +%s%N)-now ))
sleep "$sleep"
echo 'o' >"$fifo"
now=$(date +%s%N)
done
I have to admit it works better than I expected :]
Example output for my sshpingpong:
Logfile is spp_example_net.log
2022-07-24 14:41:00 seq=0 13.349
2022-07-24 14:41:05 seq=1 14.640
2022-07-24 14:41:10 seq=2 13.990
2022-07-24 14:41:15 seq=3 12.802
2022-07-24 14:41:20 seq=4 12.070
2022-07-24 14:41:25 seq=5 11.869
...
Example output of compact mode using sshpingpong -L -c user@example.net where "_" means the RTT was <100ms and e.g. "2" means the RTT was >=200 and < 300ms:
0 _ 100 1 200 2 300 3 400 4 500 5 600 6 700 7 800 8 900 9 999 # ...
__________________________1_________________________1__________
______________________________________1____________#_2_________
____1____________________________________#______1______________
____1_____________________________________________1_____1___1__
_____________________1____1____________________________________
__________1__________________1__________________1______________
___2___________________1_______________________________________
_____________________22____1____11____59####4444454523444543444
12222#21112______#________________...
The full script is available as Github gist.
