Monday, January 05, 2009

Silencing the output of Python's subprocess.Popen

I'm learning Python these days while writing an script to automate the testing of ATF under multiple virtual machines. I had this code in a shell script, but it is so ugly and clumsy that I don't even dare to add it to the repository. Hopefully, the new version in Python will be more robust and versatile enough to be published.

One of the things I've been impressed by is the subprocess module and, in special, its Popen class. By using this class, it is trivial to spawn subprocesses and perform some IPC with them. Unfortunately, Popen does not provide any way to silence the output of the children. As I see it, it'd be nice if you'd pass an IGNORE flag as the stdout/stderr behavior, much like you can currently set those to PIPE or set stderr to STDOUT.

The following trivial module implements this idea. It extends Popen so that the callers can pass the IGNORE value to the stdout/stderr arguments. (Yes, it is trivial but it is also one of the first Python code I write so... it may contain obviously non-Pythonic, ugly things.) The idea is that this exposes the same interface so that it can be used as a drop-in replacement. OK, OK, it lacks some methods and the constructor does not match the original signature, but this is enough for my current use cases!

import subprocess


IGNORE = -3
STDOUT = subprocess.STDOUT
assert IGNORE != STDOUT, "IGNORE constant is invalid"


class Popen(subprocess.Popen):
"""Extension of subprocess.Popen with built-in support
for silencing the output channels of a child process"""

__null = None

def __init__(self, args, stdout = None, stderr = None):
subprocess.Popen.__init__(self, args = args,
stdout = self._channel(stdout),
stderr = self._channel(stderr))

def __del__(self):
self._close_null()

def wait(self):
r = subprocess.Popen.wait(self)
self._close_null()
return r

def _null_instance(self):
if self.__null == None:
self.__null = open("/dev/null", "w")
return self.__null

def _close_null(self):
if self.__null != None:
self.__null.close()
self.__null = None
assert self.__null == None, \
"Inconsistent internal state"

def _channel(self, behavior):
if behavior == IGNORE:
return self._null_instance()
else:
return behavior
By the way, somebody else suggested this same thing a while ago. Don't know why it hasn't been implemented in the mainstream subprocess module.

4 comments:

Anonymous said...

Why not just use buildbot?

Julio M. Merino Vidal said...

I've been looking at buildbot but I don't think it suits my needs. What I basically have to automate is the start a vm, build inside it, shutdown the vm cycle. buildbot doesn't seem to support this case.

Anonymous said...

You can tell subprocess.Popen to store the output for you! Just tell Popen that you want direct stdout into a pipe.

--pehr

The following demonstrates how to do this.

# import subprocess
# f = subprocess.Popen(["ls","-l"], stdout=subprocess.PIPE)
# f
subprocess.Popen object at 0xb7c353ac
# dir(f)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', '_close_fds', '_execute_child', '_get_handles', '_handle_exitstatus', '_set_cloexec_flag', '_translate_newlines', 'communicate', 'pid', 'poll', 'returncode', 'stderr', 'stdin', 'stdout', 'universal_newlines', 'wait']
# f.stdout.read()
'total 72\n-rw-r--r-- 1 root root 232 ...'

Linus said...

Don't just send it to a PIPE, you can get problem with buffering. I had problem with this while using ffmpeg and oggenc as subprocesses. Basicly they generated so much stderr output that .communicate() couldn't handle it.

My problem at StackOverflow