[IronPython] parallel importing

Michael Foord fuzzyman at voidspace.org.uk
Mon Oct 13 10:16:49 PDT 2008


Curt Hagenlocher wrote:
> The" .NET people" might say they already have an answer which involves 
> precompiling the IL with NGEN.  And in fact, using DLR precompilation 
> and NGEN with IronPython results in some pretty nice performance 
> improvements.
>
> But using IronPython directly against .py files incurs several 
> expenses that aren't part of normal .NET startup.  We have to parse 
> the .py file, build DLR-level expression trees from the language tree 
> and build IL from the expression tree.  Only then are we at the point 
> where a C# starts, and can JIT the IL.
>
> It's worth noting that CPython has to do similar work when it first 
> loads a .py file.  It, too, has to parse into a tree structure and 
> generate byte codes for the tree.  And even though the CPython byte 
> code is at a higher level than the IL generated by the DLR, CPython 
> still has a performance issue -- which is addressed by storing the 
> byte codes as a .pyc file.

The difference is still orders of magnitude smaller though...

We're looking forward to moving to precompiled binaries with IronPython 
2 at Resolver Systems - but there we will still have situations where we 
need to import from py files.

Michael

>
> I'm hoping that over the next year, we'll come up with a convention 
> (like .pyc) that allows subsequent runs of a program to benefit from 
> the work done in previous runs -- without having to go through the 
> manual step of DLR precompilation.
>
> On Mon, Oct 13, 2008 at 9:57 AM, Vernon Cole <vernondcole at gmail.com 
> <mailto:vernondcole at gmail.com>> wrote:
>
>     Help me out here.  I have seen many references to schemes to
>     optimize import speeds on Iron Python. Isn't this an attempt to
>     make up for the snail-like startup speeds suffered by *all* .net
>     applications? It seems to me that when Iron Python finally starts
>     running, it imports and performs very well. The slowdown is in
>     getting the .net engine started. Try comparing startup times for
>     "Hello World" in C Python and Iron.
>        It is my experience that I can identify .net programs in
>     general by the fact that I have double-clicked its desktop icon
>     three times (thinking I had missed it somehow) before the first
>     instance displays its splash screen, and then the next two display
>     an "already running" dialog.
>     Any chance of getting the .net people to help out with this?
>     --
>     Vernon Cole
>
>
>     On Mon, Oct 13, 2008 at 10:00 AM, Dino Viehland
>     <dinov at microsoft.com <mailto:dinov at microsoft.com>> wrote:
>
>         This is all still on 1.x, right?  It looks like #1 is fixed in
>         2.0 (we are locking but on the wrong object in 1.x).
>
>         #2 is still broken in 2.x though as well.
>
>         -----Original Message-----
>         From: users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>
>         [mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>] On Behalf Of
>         Kamil Dworakowski
>         Sent: Monday, October 13, 2008 4:45 AM
>         To: users at lists.ironpython.com <mailto:users at lists.ironpython.com>
>         Subject: [IronPython] parallel importing
>
>         Importing time is such a pain that I wanted to do it in
>         parallel on
>         many threads. I figured that if the modules where imported in
>         such a
>         way that:
>
>         no two threads import the same module at the same time
>
>         everything would be fine. To ensure that condition, it is
>         enough to
>         build a dependency graph and import based on that.
>
>         I did it: Resolver One start up time improved by 20% on a two-core
>         machine. But it crashes, surprisingly on single core machines
>         it is
>         more often (6 crashes on 200 starts).
>
>         So far I have identified two causes for crashes:
>
>         1. One thread imports a module with class B inside while
>         another is
>         importing a module with class C inside. If B and C are
>         subclasses of
>         A, it can result in IndexOutOfRangeException being raised,
>         when, under
>         the hood, IronPython.Runtime.Types.DynamicType.AddSubclass is
>         being
>         executed.
>
>         2. Attributes on .NET modules are loaded lazily, so importing
>         namespaces only is not enough. Attribute getting from reflected
>         packages is not thread safe. Looks like I would have to import
>         every
>         class explicitly (would that be enough?).
>
>         Second cause would be pretty easy to address, but I'm not so sure
>         about the first one. Are there any more potential points of
>         problems?
>         I am beginning to think I was to optimistic about all of this
>         importing on multiple cores, but if these are the only ones it
>         could
>         probably be still fixed.
>
>         If anyone is interested the code for it is on github:
>         http://github.com/luntain/ipy-parallel-import.
>
>         --
>         Kamil Dworakowski
>         Resolver Systems Ltd.
>         _______________________________________________
>         Users mailing list
>         Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
>         http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>         _______________________________________________
>         Users mailing list
>         Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
>         http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
>
>     _______________________________________________
>     Users mailing list
>     Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
>     http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.resolverhacks.net/
http://wwww.theotherdelia.co.uk/



More information about the Users mailing list