How I solved the SSH 100-packet-per-keystroke problem with LLM
SSH is a protocol for securely connecting to a remote server through a terminal emulator, and one might think that it is far simpler than communication using a browser, etc. However, one programmer noticed that 100 packets were being sent with each keystroke while using SSH, and he investigated the cause and published the details of the solution he came up with on his blog.
Why does SSH send 100 packets per keystroke? · eieio.games
Game programmer Nolen Royalty was developing a game that ran over SSH. The game ran on an 80x60 terminal emulator, updating the screen 10 times per second. He expected at least 2,000 simultaneous connections, meaning the server needed to update nearly 100 million cells per second. Since performance was a priority, Royalty connected hundreds of bots via SSH and constantly monitored their status. However, he encountered a strange phenomenon when he made a breaking change to the test harness . A 'breaking change' is a change that causes existing functionality or behavior to stop working.
Normally, the server periodically sends game data to the bots. However, due to the breaking change, the server started sending the bots a single 'Screen too small' error message. As a result, the game's CPU and bandwidth usage dropped from 100% to 50%. Royalty was initially pleased that performance had improved, but once he realized what was happening, he wondered: 'If the bots weren't receiving game data, CPU usage should have been 0%, so why was it only 50%?'
First, to debug the test harness, we recorded game data traffic with tcpdump and checked whether there was any difference before and after making breaking changes to the test harness.
[code]
# Record traffic on port 22 after making breaking changes
timeout 30s tcpdump -i eth0 'port 22' -w with-breaking-change.pcap
# Revert the breaking change and log traffic on port 22
timeout 30s tcpdump -i eth0 'port 22' -w without-breaking-change.pcap
[/code]
We discovered that after the breaking change, it was no longer possible to render the game via SSH. This means that the post-change recording, 'with-breaking-change.pcap,' does not contain any game rendering data, but only packets indicating connection overhead. To clarify the situation, we asked Claude Code , who was using the debugging tool, to summarize the contents of the recorded traffic data.
[code]
Wanna take a look yourself? I put with-breaking-change.pcap in this directory
--
Wow! Here's what I found:
Packet Size Distribution (413,703 total packets):
274,907 packets (66%): Exactly 36 bytes
138,778 packets (34%): 0 bytes (TCP ACKs)
18 packets (<0.1%): 72 bytes
[/code]
By shortening the recording time and further analyzing the output from Claude Code, they discovered that the 'mysterious packets' were arriving approximately every 20 milliseconds. Neither Royalty nor Claude Code knew what was going on, so they came up with several hypotheses about the packets' identity.
SSH flow control messages
- Polling the size of the terminal emulator or some other status check
Something strange is happening with bubbletea or wish
One possible clue was that 'packet transmission and reception was initiated by the client side, not the server side,' so Royalty decided to try recording a similar session in a normal SSH session.
[code]
# Run on the terminal emulator screen of the client PC
sudo tcpdump -ien0 'port 22'
# Run in another terminal emulator window on the client PC
ssh $some_vm_of_mine
[/code]
After waiting for the initial packet transmission to stop, I sent a single keystroke to the remote server and checked the tcpdump output, and found the exact same issue occurring, which means that this issue is not caused by the game, but is an inherent characteristic of SSH.
Running ' ssh -vvv ' gave me a pretty clear picture of what was going on.
[code]
debug3: obfuscate_keystroke_timing: starting: interval ~20ms
debug3: obfuscate_keystroke_timing: stopping: chaff time expired (49 chaff packets sent)
debug3: obfuscate_keystroke_timing: starting: interval ~20ms
debug3: obfuscate_keystroke_timing: stopping: chaff time expired (101 chaff packets sent)
[/code]
With the '20 millisecond interval' as definitive evidence, we were able to determine that the situation perfectly matched the 'mystery packets' we had seen earlier. Other parts of the message also contained significant information, revealing that the first keystroke contained 49 'chaff' packets, and the second keystroke contained 101 'chaff' packets. 'chaff' packets were added to SSH in 2023 to obfuscate keystroke timing, preventing attacks from exploiting the difference in typing speed between characters.
The presence of 'chaff' packets is very important in a normal SSH session where privacy is important, because the timing of packets transmitting keystrokes can be hidden by sending a large number of 'chaff' packets. However, from the perspective of an online game where low latency is required, the presence of 'chaff' packets is nothing more than overhead.
We updated the test harness to disable 'keystroke obfuscation' on the SSH client side, and as expected, CPU usage dropped dramatically and the bot received valid data. However, in reality, it's not practical to ask players to launch SSH with the mysterious option every time. When we asked Claude Code if it was possible to disable keystroke obfuscation on the server side, he initially replied, 'Probably not.'
Fortunately, Royalty was able to find
[code]
Total CPU 29.90% -> 11.64%
Syscalls 3.10s -> 0.66s
Crypto 1.6s -> 0.11s
Bandwidth ~6.5 Mbit/sec -> ~3 Mbit/sec
[/code]
Claude was also very excited when he saw the results.
'I was worried that LLM would take the fun out of the problem-solving process, but I had a blast debugging this problem with Claude Code,' Royalty said.
Related Posts:
