[IronPython] new subprocess.py module. Need eyes.

Danil Dotsenko dotsa at hotmail.com
Sat Sep 11 08:48:56 PDT 2010

(Hm.. first this mailing list really did not like my first, HTML-formatted message, it seems.)
I extended Jeff Hardy's 2008 version of subprocess.py (http://bitbucket.org/jdhardy/code/src/tip/subprocess.py) module to support (among other things):- Redirect to stderr and stdout to file-likes and file descriptors (.Net threads, one per output / input type) On large output sizes, speed is on par with cPython's version.- Binary subprocess output capture- Implemented .communicate()- Added a non-spec .communicateIO() that behaves like .communicate(), but:  - persists captured data to disk (tempfile.TemporaryFile()) after a custom-defined max on MemoryStream size is reached.  - outputs out, and err as file-likes, not strings
.communicateIO() method was added to support to support binary, big outputs from the subprocess. I have a patch / module extension for cPython's version subprocess.py and contemplating submitting it for inclusion along 3.3 timeline, of 3.2 if there is time, help and support for that.
Code is here: http://github.com/dvdotsenko/IronPythonParts   Target functionality is cPython 2.6.x subprocess.py. Working on adding check_output() method from 2.7 subprocess.py
1. I would appreciate some input on the efficiency of the redirect I am using in _StreamRedirector and _StreamPersistor classes. Because the output must be consumable by "regular" python code, expecting cPython file-like methods on the object, I had a choice of (a) milking data from process.StandardOutput.BaseStream to .Net stream and, then, converting it to file-like or (b) milking the output directly into python file-like, while reboxing Array[Byte] data. I chose the latter.   (A) is faster than comparable functionality in cPython, but: the resulting file-like does not autodelete itself after f.close() and it would work only in case of .communicate(), as redirect to file-like already gives me a python-style file-like.  (B) is slightly slower than comparable functionality in cPython, but allows this module to support any type or subclass of any Python file-like, as there is only one requirement - .write() method that takes bytes- or buffer-like input.For a second i thought that any give python file-like could in actuality be some sort of .Net stream and was contemplating obtaining a pointer to that .Net "base stream" and writing to it in .Net style. However, realized that some mmaped python file-likes would be implemented on top of byte arrays that are not "streams" so gave up on that idea. The only thing i see as possible to improve is finding the fastest way to blip .Net Array[byte] into python file-like that is not necessarily a file. <- Tips on this appreciated.
2. I am unittesting against cPython 2.6.x's version of test_subprocess.py. They have a bit of "let's see if we are losing pointers, file descriptors" checks in there. Was wondering if IronPython would have similar but different gunk that needs to be checked. I am hoping that I don't need to del MemoryStream objects after .Close() etc. If a good eye could take a brief look at the code with intention of finding leaks, I would sincerely appreciate that.

More information about the Users mailing list