TTYs: Never gets boring
June 16, 2013

Just a short rant: I’m working on an interactive console used for debugging a computer cluster. It connects to all nodes in the cluster and provides you with a single place to run queries. It uses the new (not yet officially-released) zero-deploy feature of RPyC, which sets up a secure, single-use RPyC server on a machine, requiring only SSH access. Once the client connection closes, the zero-deployed server will shut down and delete itself from the file system.

It’s a cool feature on its own (and I’ll blog about it soon), but there’s a reason I’m getting you through all of the details here. You see, the debugging console fires up SSH subprocesses in the background, over which RPyC connections are tunneled… and then the strangest thing happened. I was running a query which was taking too long and hit Ctrl+C to kill it and return to the interpreter. The query indeed stopped, but all of my RPyC connections have died with it. Huh?

Here’s a really short way to reproduce this scenario:

>>> from subprocess import Popen, PIPE
>>> p=Popen(["sleep", "60"], stdin=PIPE, stdout=PIPE, stderr=PIPE)
>>>
>>> p.poll()      # poll() returns None as the process is still running in the background
>>>
>>>               # now hit Ctrl+C in the interactive prompt
KeyboardInterrupt
>>>
>>> p.poll()      # and voila, `sleep` was killed by SIGINT
-2

It’s terribly confusing at first, but it happens because child processes inherit their paren’t session ID. Terminal events, such as SIGINT and SIGHUP, are dispatched to all processes belonging to the terminal’s process group, so it’s not just the Python interpreter to receive the signal – every child process it spawned will also suffer. In my case, it killed all of the SSH tunnels I had set up.

The solution is to setsid before execing the child:

>>> import os
>>> p=Popen(["sleep", "60"], stdin=PIPE, stdout=PIPE, stderr=PIPE, preexec_fn=os.setsid)
>>> p.poll()
>>>
KeyboardInterrupt
>>> p.poll()
>>>

So I had to add this feature to plumbum, and while I was at it, I also added daemonization support. In other words, I’ll have to release 1.3 soon – even though I released 1.2 not two weeks ago. Life’s a bitch and TTYs are the mother of all monsters :)