[IronPython] Slow Performance of CPython libs?

Dan Shechter dans at houmus.org
Sat Feb 23 08:25:25 PST 2008


Well it does look interesting, but I think there is something flawed 
with my own suggestion as a general rule.

As someone mentioned here, the CompileToAssembly() does create a new 
assembly, which gets loaded it into the AppDomain (eventually) and it 
doesn't get reclaimed for eternity (forgive my melodrama).
So I would definitely not "suggest" it as a general solution for all 
Regex woes in the world.

I would try, just for the fun of it to patch IP to use 
CompileToAssembly() just to see how much time in reality does it  knock 
off the profiling / general runtime.

Also note that your profiler might be causing you faux-visages of 
performance. Have you tried to just time the two different versions 
without profiling? perhaps performance is already "Good Enough"?

Birsch wrote:
> Update:
> 
> I patched re.cs with a compile-check flag (same as Dino), and while it 
> knocked off 25%, it still remained very slow in comparison to CPython 
> (x5-x10).
> So I ran the profiler again on the patched code:
> 
> Namespace 	Method name 	Time (sec.) 	Time with children (sec.) 	Hit 
> count 	Source file
> IronPython.Runtime.Operations 	Ops.CallWithContext(ICallerContext 
> context, object func, object arg0, object arg1) 	5.7779 	35.5915 	75882 	
> System.Text.RegularExpressions 	CompiledRegexRunner.Go() 	5.3627 
> 19.9116 	13684621 	
> IronPython.Runtime.Calls 	Method.Call(ICallerContext context, object 
> arg0) 	5.0284 	36.0412 	50886 	
> IronPython.Runtime.Calls 	PythonFunction.CallInstance(ICallerContext 
> context, object arg0, object arg1) 	5.0055 	36.0168 	50886 	
> IronPython.Runtime.Calls 	Function2.Call(ICallerContext context, object 
> arg0, object arg1) 	4.7494 	35.7496 	47485 	
> IronPython.Runtime.Operations 	Ops.CallWithContext(ICallerContext 
> context, object func, object arg0) 	4.3452 	35.3645 	146642 	
> System.Text.RegularExpressions 	RegexCharClass.CharInClassRecursive(char 
> ch, string set, int start) 	4.0303 	6.0294 	31498121 	
> System.Text.RegularExpressions 	RegexCharClass.CharInClass(char ch, 
> string set) 	3.7132 	9.7426 	31498121 	
> System.Text.RegularExpressions 	RegexRunner.CharInClass(char ch, string 
> charClass) 	3.1865 	12.9291 	31498114 	
> IronPython.Runtime.Calls 	Method.Call(ICallerContext context, object 
> arg0, object arg1) 	3.1467 	8.2760 	15008 	
> IronPython.Runtime.Calls 	PythonFunction.CallInstance(ICallerContext 
> context, object arg0, object arg1, object arg2) 	3.1440 	8.2725 	15008 	
> System.Text.RegularExpressions 	RegexCharClass.CharInClassInternal(char 
> ch, string set, int start, int mySetLength, int myCategoryLength) 
> 1.9806 	1.9991 	31498121 	
> System.Text.RegularExpressions 	RegexRunner.EnsureStorage() 	1.6674 
> 1.6674 	37765633 	
> System.Text.RegularExpressions 	RegexRunner.Scan(Regex regex, string 
> text, int textbeg, int textend, int textstart, int prevlen, bool 
> quick) 	1.5465 	22.9673 	39137 	
> IronPython.Runtime.Operations 	Ops.GetAttr(ICallerContext context, 
> object o, SymbolId name) 	1.5139 	5.5342 	546878 	
> System.Text.RegularExpressions 	CompiledRegexRunner.FindFirstChar() 
> 1.3493 	1.4119 	13689019 	
> IronPython.Runtime.Types 	OldClass.TryLookupSlot(SymbolId name, out 
> object ret) 	1.2987 	2.4052 	350822 	
> System.Globalization 	CompareInfo.IsPrefix(string source, string prefix, 
> CompareOptions options) 	0.3656 	0.3669 	3421 	
> IronPython.Runtime 	SymbolId.op_Equality(SymbolId a, SymbolId b) 
> 0.3635 	0.3635 	7974446 	
> IronPython.Runtime 	CustomSymbolDict.TryGetValue(SymbolId key, out 
> object value) 	0.2286 	1.3466 	1165376 	
> System 	String.InternalSubString(int startIndex, int length, bool 
> fAlwaysCopy) 	0.2253 	0.3669 	52536 	
> IronPython.Runtime.Calls 
> FunctionEnvironmentNDictionary.TryGetExtraValue(SymbolId key, out object 
> value) 	0.2228 	0.4089 	152594 	
> IronPython.Runtime.Types 	OldInstance.TryGetAttr(ICallerContext context, 
> SymbolId name, out object value) 	0.2156 	3.0540 	396617 	
> IronPython.Runtime.Calls 	Function3.Call(ICallerContext context, object 
> arg0, object arg1, object arg2) 	0.2109 	5.3373 	11221 	
> System 	String.wstrcpy(char* dmem, char* smem, int charCount) 	0.1831 
> 0.1831 	621212 	
> System.Collections.Generic 	Dictionary<TKey, TValue>.FindEntry(TKey 
> key) 	0.1460 	0.3733 	558255 	
> IronPython.Runtime.Calls 
> FunctionEnvironment32Dictionary.TryGetExtraValue(SymbolId key, out 
> object value) 	0.1457 	0.2680 	103477 	
> IronPython.Runtime.Types 	DynamicType.TryGetSlot(ICallerContext context, 
> SymbolId name, out object ret) 	0.1367 	0.3618 	406706 	
> IronPython.Runtime.Operations 	Ops.CallWithContext(ICallerContext 
> context, object func, object[] args) 	0.1326 	1.7429 	10267 	
> 
> 	sgmllib.finish_starttag$f318(FunctionEnvironment32Dictionary $env, 
> object self, object tag, object attrs) 	0.1213 	5.0059 	3787 	
> 
> 
> If I'm reading this correctly, it now runs the compiled regex code, and 
> the performance diff boils down to regex performance itself.
> 
> Dan Shechter mentioned earlier an option of generating 
> regex-pattern-specific runtime code. With almost an order of magnitude 
> to knock off here - do you think that will do the trick?
> 
> -Birsch
> 
> On Fri, Feb 22, 2008 at 10:24 PM, Birsch <birsch at gmail.com 
> <mailto:birsch at gmail.com>> wrote:
> 
>     Great, you've got my vote :)
> 
>     Thanks a lot for your help.
> 
> 
>     On Fri, Feb 22, 2008 at 9:03 PM, Dino Viehland
>     <dinov at exchange.microsoft.com <mailto:dinov at exchange.microsoft.com>>
>     wrote:
> 
>         Oh, and I've gone ahead and created a bug for this one for 2.0:
>         http://www.codeplex.com/IronPython/WorkItem/View.aspx?WorkItemId=15289
> 
>          
> 
>         *From:* users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>
>         [mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>] *On Behalf Of *Birsch
>         *Sent:* Friday, February 22, 2008 9:40 AM
> 
>         *To:* Discussion of IronPython
>         *Subject:* Re: [IronPython] Slow Performance of CPython libs?
> 
>          
> 
>         Dino - do you want me to register a workitem? Also, if I wanted
>         to patch v1.1.1, where should I start?
> 
>         On Thu, Feb 21, 2008 at 8:48 PM, Dino Viehland
>         <dinov at exchange.microsoft.com
>         <mailto:dinov at exchange.microsoft.com>> wrote:
> 
>         After Birsh's last comment that they're calling compile I was
>         thinking we might be doing something really stupid.  Turns out
>         that we are.  For our supposedly compiled regex's we're not
>         passing the Compiled option.  So they're all getting
>         interpretted and of course that's going to suck big time.
> 
>         So the fix for this is probably to just pass
>         RegexOptions.Compiled to the Regex object being contructed into
>         RE_Pattern.  Internally the Regex class will compile to dynamic
>         methods (much better than compiling to an assembly because it'll
>         be collectible).
> 
>         It's actually a little more complex than that because we create
>         RE_Pattern objects for the non-compiled so we need to flow the
>         option in, but it's still a simple fix.
> 
> 
>         -----Original Message-----
>         From: users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>
>         [mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>] On Behalf Of Dan
>         Shechter
>         Sent: Thursday, February 21, 2008 10:19 AM
>         To: Discussion of IronPython
>         Subject: Re: [IronPython] Slow Performance of CPython libs?
> 
>         Just out of curiosity...
>         I'm guessing that there are two levels of caching that could be
>         used.
>         One would be to hold a global dictionary of some sort of pattern ->
>         Regex that will save the lengthy parsing of the re.
> 
>         The other would be to actually call Regex.CompileToAssembly to get a
>         more efficient reperesentation (in runtime that is) of the re.
> 
>         Would you (the IP team) favor the former or the latter?
> 
>                Shechter.
> 
> 
>         Birsch wrote:
>          > I checked both sgmllib.py and BeautifulSoup.py - and it seems
>         both are
>          > reusing the same regexps (searched for re.compile).
>          >
>          > I think your suggestion is very relevant in this case. It
>         makes sense to
>          > replicate the "compile once use many" behavior that is
>         commonly used
>          > with regexp.
>          >
>          > -Birsch
>          >
>          > On Thu, Feb 21, 2008 at 7:30 PM, Dino Viehland
>          > <dinov at exchange.microsoft.com
>         <mailto:dinov at exchange.microsoft.com>
>         <mailto:dinov at exchange.microsoft.com
>         <mailto:dinov at exchange.microsoft.com>>> wrote:
>          >
>          >     Do you know if the same reg ex is being used repeatedly?
>          If so
>          >     maybe we can cache & compile the regex instead of
>         interpretting it
>          >     all the time.
>          >
>          >
>          >
>          >     *From:* users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>
>          >     <mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>>
>          >     [mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>
>          >     <mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>>] *On Behalf Of *Birsch
>          >     *Sent:* Thursday, February 21, 2008 8:30 AM
>          >
>          >     *To:* Discussion of IronPython
>          >     *Subject:* Re: [IronPython] Slow Performance of CPython libs?
>          >
>          >
>          >
>          >     I took on Cooper's advice and profiled with ANTS. Here
>         are the top
>          >     methods:
>          >
>          >     *Namespace*
>          >
>          >
>          >
>          >     *Method name*
>          >
>          >
>          >
>          >     *Time (sec.)*
>          >
>          >
>          >
>          >     *Time with children (sec.)*
>          >
>          >
>          >
>          >     *Hit count*
>          >
>          >
>          >
>          >     *Source file*
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Go()
>          >
>          >
>          >
>          >     37.0189
>          >
>          >
>          >
>          >     94.4676
>          >
>          >
>          >
>          >     13689612
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Operator()
>          >
>          >
>          >
>          >     6.2411
>          >
>          >
>          >
>          >     6.2411
>          >
>          >
>          >
>          >     131146274
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Advance(int i)
>          >
>          >
>          >
>          >     5.9264
>          >
>          >
>          >
>          >     8.7202
>          >
>          >
>          >
>          >     66000263
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.SetOperator(int op)
>          >
>          >
>          >
>          >     5.5750
>          >
>          >
>          >
>          >     5.5750
>          >
>          >
>          >
>          >     131146274
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Backtrack()
>          >
>          >
>          >
>          >     5.5692
>          >
>          >
>          >
>          >     9.4895
>          >
>          >
>          >
>          >     37781343
>          >
>          >
>          >
>          >     IronPython.Runtime.Operations
>          >
>          >
>          >
>          >     Ops.CallWithContext(ICallerContext context, object func,
>         object
>          >     arg0, object arg1)
>          >
>          >
>          >
>          >     5.5572
>          >
>          >
>          >
>          >     114.5245
>          >
>          >
>          >
>          >     79754
>          >
>          >
>          >
>          >     IronPython.Runtime.Calls
>          >
>          >
>          >
>          >     Method.Call(ICallerContext context, object arg0)
>          >
>          >
>          >
>          >     4.9052
>          >
>          >
>          >
>          >     114.8251
>          >
>          >
>          >
>          >     50886
>          >
>          >
>          >
>          >     IronPython.Runtime.Calls
>          >
>          >
>          >
>          >     PythonFunction.CallInstance(ICallerContext context,
>         object arg0,
>          >     object arg1)
>          >
>          >
>          >
>          >     4.8876
>          >
>          >
>          >
>          >     114.8059
>          >
>          >
>          >
>          >     50886
>          >
>          >
>          >
>          >     IronPython.Runtime.Calls
>          >
>          >
>          >
>          >     Function2.Call(ICallerContext context, object arg0,
>         object arg1)
>          >
>          >
>          >
>          >     4.6400
>          >
>          >
>          >
>          >     114.5471
>          >
>          >
>          >
>          >     47486
>          >
>          >
>          >
>          >     IronPython.Runtime.Operations
>          >
>          >
>          >
>          >     Ops.CallWithContext(ICallerContext context, object func,
>         object arg0)
>          >
>          >
>          >
>          >     4.2344
>          >
>          >
>          >
>          >     114.1604
>          >
>          >
>          >
>          >     146658
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexBoyerMoore.Scan(string text, int index, int
>         beglimit, int endlimit)
>          >
>          >
>          >
>          >     3.6465
>          >
>          >
>          >
>          >     3.6465
>          >
>          >
>          >
>          >     13678131
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexCharClass.CharInClassRecursive(char ch, string set,
>         int start)
>          >
>          >
>          >
>          >     3.6288
>          >
>          >
>          >
>          >     5.7113
>          >
>          >
>          >
>          >     31508162
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Goto(int newpos)
>          >
>          >
>          >
>          >     3.2058
>          >
>          >
>          >
>          >     5.1470
>          >
>          >
>          >
>          >     27364668
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Operand(int i)
>          >
>          >
>          >
>          >     3.1923
>          >
>          >
>          >
>          >     3.1923
>          >
>          >
>          >
>          >     73230687
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexRunner.EnsureStorage()
>          >
>          >
>          >
>          >     3.0803
>          >
>          >
>          >
>          >     3.0803
>          >
>          >
>          >
>          >     51474823
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexCharClass.CharInClass(char ch, string set)
>          >
>          >
>          >
>          >     3.0713
>          >
>          >
>          >
>          >     8.7827
>          >
>          >
>          >
>          >     31508162
>          >
>          >
>          >
>          >     IronPython.Runtime.Calls
>          >
>          >
>          >
>          >     Method.Call(ICallerContext context, object arg0, object arg1)
>          >
>          >
>          >
>          >     2.9821
>          >
>          >
>          >
>          >     7.8675
>          >
>          >
>          >
>          >     15012
>          >
>          >
>          >
>          >     IronPython.Runtime.Calls
>          >
>          >
>          >
>          >     PythonFunction.CallInstance(ICallerContext context,
>         object arg0,
>          >     object arg1, object arg2)
>          >
>          >
>          >
>          >     2.9794
>          >
>          >
>          >
>          >     7.8639
>          >
>          >
>          >
>          >     15012
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Forwardcharnext()
>          >
>          >
>          >
>          >     2.8852
>          >
>          >
>          >
>          >     2.8852
>          >
>          >
>          >
>          >     62865185
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Forwardchars()
>          >
>          >
>          >
>          >     2.8279
>          >
>          >
>          >
>          >     2.8279
>          >
>          >
>          >
>          >     59436277
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexCharClass.CharInClassInternal(char ch, string set,
>         int start,
>          >     int mySetLength, int myCategoryLength)
>          >
>          >
>          >
>          >     2.0632
>          >
>          >
>          >
>          >     2.0826
>          >
>          >
>          >
>          >     31508162
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexRunner.Scan(Regex regex, string text, int textbeg,
>         int textend,
>          >     int textstart, int prevlen, bool quick)
>          >
>          >
>          >
>          >     1.8376
>          >
>          >
>          >
>          >     101.7226
>          >
>          >
>          >
>          >     43009
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.FindFirstChar()
>          >
>          >
>          >
>          >     1.6405
>          >
>          >
>          >
>          >     5.3456
>          >
>          >
>          >
>          >     13701755
>          >
>          >
>          >
>          >     IronPython.Runtime.Types
>          >
>          >
>          >
>          >     OldClass.TryLookupSlot(SymbolId name, out object ret)
>          >
>          >
>          >
>          >     1.5573
>          >
>          >
>          >
>          >     2.8124
>          >
>          >
>          >
>          >     389516
>          >
>          >
>          >
>          >     IronPython.Runtime.Operations
>          >
>          >
>          >
>          >     Ops.GetAttr(ICallerContext context, object o, SymbolId name)
>          >
>          >
>          >
>          >     1.5365
>          >
>          >
>          >
>          >     5.3456
>          >
>          >
>          >
>          >     558524
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Textpos()
>          >
>          >
>          >
>          >     1.4020
>          >
>          >
>          >
>          >     1.4020
>          >
>          >
>          >
>          >     32648926
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Advance()
>          >
>          >
>          >
>          >     1.1916
>          >
>          >
>          >
>          >     2.9526
>          >
>          >
>          >
>          >     13703950
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Textto(int newpos)
>          >
>          >
>          >
>          >     1.1218
>          >
>          >
>          >
>          >     1.1218
>          >
>          >
>          >
>          >     24120890
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.TrackPeek()
>          >
>          >
>          >
>          >     1.0579
>          >
>          >
>          >
>          >     1.0579
>          >
>          >
>          >
>          >     24120894
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     Regex.Run(bool quick, int prevlen, string input, int
>         beginning, int
>          >     length, int startat)
>          >
>          >
>          >
>          >     0.7280
>          >
>          >
>          >
>          >     102.4644
>          >
>          >
>          >
>          >     43009
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.TrackPush(int I1)
>          >
>          >
>          >
>          >     0.6834
>          >
>          >
>          >
>          >     0.6834
>          >
>          >
>          >
>          >     13745149
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.StackPush(int I1)
>          >
>          >
>          >
>          >     0.6542
>          >
>          >
>          >
>          >     0.6542
>          >
>          >
>          >
>          >     13703955
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.TrackPop()
>          >
>          >
>          >
>          >     0.6068
>          >
>          >
>          >
>          >     0.6068
>          >
>          >
>          >
>          >     13663035
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.TrackPush()
>          >
>          >
>          >
>          >     0.6049
>          >
>          >
>          >
>          >     0.6049
>          >
>          >
>          >
>          >     13708230
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.StackPop()
>          >
>          >
>          >
>          >     0.5836
>          >
>          >
>          >
>          >     0.5836
>          >
>          >
>          >
>          >     13703956
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.Bump()
>          >
>          >
>          >
>          >     0.4987
>          >
>          >
>          >
>          >     0.4987
>          >
>          >
>          >
>          >     10472790
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.TrackPush(int I1, int I2)
>          >
>          >
>          >
>          >     0.4864
>          >
>          >
>          >
>          >     0.4864
>          >
>          >
>          >
>          >     10472790
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.TrackPeek(int i)
>          >
>          >
>          >
>          >     0.4663
>          >
>          >
>          >
>          >     0.4663
>          >
>          >
>          >
>          >     10457859
>          >
>          >
>          >
>          >     System.Text.RegularExpressions
>          >
>          >
>          >
>          >     RegexInterpreter.TrackPop(int framesize)
>          >
>          >
>          >
>          >     0.4396
>          >
>          >
>          >
>          >     0.4396
>          >
>          >
>          >
>          >     10457859
>          >
>          >
>          >
>          >
>          >     Moving up the stack of regex.Go(), most calls originate from
>          >     sgmllib's parse_starttag.
>          >
>          >     HTH,
>          >     -Birsch
>          >
>          >     On Thu, Feb 21, 2008 at 2:35 PM, Birsch <birsch at gmail.com
>         <mailto:birsch at gmail.com>
>          >     <mailto:birsch at gmail.com <mailto:birsch at gmail.com>>> wrote:
>          >
>          >     Thanks Michael and Dino.
>          >
>          >     I'll prof and send update. Got a good profiler
>         recommendation for .Net?
>          >     Meanwhile I noticed the sample site below causes
>         BeautifulSoup to
>          >     generate quite a few [python] exceptions during __init__.
>         Does
>          >     IronPython handle exceptions significantly slower than
>         CPtyhon?
>          >
>          >     Repro code is simple (just build a BeautifulSoup obj with
>         mininova's
>          >     home page).
>          >     Here are the .py and .cs I used to time the diffs:
>          >
>          >     *bstest.py:*
>          >     #Bypass CPython default socket implementation with IPCE/FePy
>          >     import imp, os, sys
>          >     sys.modules['socket'] = module = imp.new_module('socket')
>          >     execfile('socket.py', module.__dict__)
>          >
>          >     from BeautifulSoup import BeautifulSoup
>          >     from urllib import urlopen
>          >     import datetime
>          >
>          >     def getContent(url):
>          >         #Download html data
>          >         startTime = datetime.datetime.now()
>          >         print "Getting url", url
>          >         html = urlopen(url).read()
>          >         print "Time taken:", datetime.datetime.now() - startTime
>          >
>          >         #Make soup
>          >         startTime = datetime.datetime.now()
>          >         print "Making soup..."
>          >         soup = BeautifulSoup(markup=html)
>          >         print "Time taken:", datetime.datetime.now() - startTime
>          >
>          >     if __name__ == "__main__":
>          >         print getContent("www.mininova.org
>         <http://www.mininova.org> <http://www.mininova.org>")
>          >
>          >
>          >     *C#:*
>          >     using System;
>          >     using System.Collections.Generic;
>          >     using System.Text;
>          >     using IronPython.Hosting;
>          >
>          >     namespace IronPythonBeautifulSoupTest
>          >     {
>          >         public class Program
>          >         {
>          >             public static void Main(string[] args)
>          >             {
>          >                 //Init
>          >                 System.Console.WriteLine("Starting...");
>          >                 DateTime start = DateTime.Now;
>          >                 PythonEngine engine = new PythonEngine();
>          >
>          >                 //Add paths:
>          >                 //BeautifulSoup.py, socket.py, bstest.py
>         located on exe dir
>          >                 engine.AddToPath(@".");
>          >                 //CPython Lib (replace with your own)
>          >                 engine.AddToPath(@"D:\Dev\Python\Lib");
>          >
>          >                 //Import and load
>          >                 TimeSpan span = DateTime.Now - start;
>          >                 System.Console.WriteLine("[1] Import: " +
>          >     span.TotalSeconds);
>          >                 DateTime d = DateTime.Now;
>          >                 engine.ExecuteFile(@"bstest.py");
>          >                 span = DateTime.Now - d;
>          >                 System.Console.WriteLine("[2] Load: " +
>         span.TotalSeconds);
>          >
>          >                 //Execute
>          >                 d = DateTime.Now;
>          >                
>         engine.Execute("getContent(\"http://www.mininova.org\
>         <http://www.mininova.org%5C>")");
>          >                 span = DateTime.Now - d;
>          >                 System.Console.WriteLine("[3] Execute: " +
>          >     span.TotalSeconds);
>          >                 span = DateTime.Now - start;
>          >                 System.Console.WriteLine("Total: " +
>         span.TotalSeconds);
>          >
>          >
>          >             }
>          >         }
>          >     }
>          >
>          >
>          >     On Wed, Feb 20, 2008 at 6:57 PM, Dino Viehland
>          >     <dinov at exchange.microsoft.com
>         <mailto:dinov at exchange.microsoft.com>
>         <mailto:dinov at exchange.microsoft.com
>         <mailto:dinov at exchange.microsoft.com>>>
>          >     wrote:
>          >
>          >     We've actually had this issue reported once before a long
>         time ago -
>          >     it's a very low CodePlex ID -
>          >    
>         http://www.codeplex.com/IronPython/WorkItem/View.aspx?WorkItemId=651
>          >
>          >     We haven't had a chance to investigate the end-to-end
>         scenario.  If
>          >     someone could come up with a smaller simpler repro that'd
>         be great.
>          >      Otherwise we haven't forgotten about it we've just had more
>          >     immediately pressing issues to work on :(.
>          >
>          >
>          >     -----Original Message-----
>          >     From: users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>
>          >     <mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>>
>          >     [mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>
>          >     <mailto:users-bounces at lists.ironpython.com
>         <mailto:users-bounces at lists.ironpython.com>>] On Behalf Of
>         Michael Foord
>          >     Sent: Wednesday, February 20, 2008 5:20 AM
>          >     To: Discussion of IronPython
>          >     Subject: Re: [IronPython] Slow Performance of CPython libs?
>          >
>          >     Birsch wrote:
>          >      > Hi - We've been using IronPython successfully to allow
>         extensibility
>          >      > of our application.
>          >      >
>          >      > Overall we are happy with the performance, with the
>         exception of
>          >      > BeautifulSoup which seems to run very slowly: x5 or
>         more time to
>          >      > execute compared to CPython.
>          >      >
>          >      > Most of the time seems to be spent during __init__()
>         of BS, where the
>          >      > markup is parsed.
>          >      >
>          >      > We suspect this has to do with the fact that our
>         CPython env is
>          >      > executing .pyc files and can precompile its libs,
>         while the
>          >     IronPython
>          >      > environment compiles each iteration. We couldn't find
>         a way to
>          >      > pre-compile the libs and then introduce them into the
>         code, but
>          >     in any
>          >      > case this will result in a large management overhead
>         since the amount
>          >      > of CPython libs we expose to our users contains 100's
>         of modules.
>          >      >
>          >      > Any ideas on how to optimize?
>          >
>          >     I think it is worth doing real profiling to find out
>         where the time is
>          >     being spent during parsing.
>          >
>          >     If it is spending most of the time in '__init__' then the
>         time is
>          >     probably not spent in importing - so compilation isn't
>         relevant and it
>          >     is a runtime performance issue. (Importing is much slower
>         with
>          >     IronPython and at Resolver Systems we do use precompiled
>         binaries - but
>          >     strangely enough it doesn't provide much of a performance
>         gain.)
>          >
>          >     Michael
>          >     http://www.manning.com/foord
>          >
>          >      >
>          >      > Thanks,
>          >      > -Birsch
>          >      >
>          >      > Note: we're using FePy/IPCE libs with regular IP
>         v1.1.1 runtime DLLs
>          >      > (this was done to overcome library incompatibilities
>         and network
>          >      > errors). However, the relevant slow .py code (mainly
>         SGMLParser and
>          >      > BeautifulSoup) is the same.
>          >      >
>          >    
>         ------------------------------------------------------------------------
>          >      >
>          >      > _______________________________________________
>          >      > Users mailing list
>          >      > Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>
>         <mailto:Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>>
>          >      >
>         http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>          >      >
>          >
>          >     _______________________________________________
>          >     Users mailing list
>          >     Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>
>         <mailto:Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>>
>          >     http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>          >     _______________________________________________
>          >     Users mailing list
>          >     Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>
>         <mailto:Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>>
>          >     http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>          >
>          >
>          >
>          >
>          >
>          >
>          >     _______________________________________________
>          >     Users mailing list
>          >     Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>
>         <mailto:Users at lists.ironpython.com
>         <mailto:Users at lists.ironpython.com>>
>          >     http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>          >
>          >
>          >
>          >
>         ------------------------------------------------------------------------
>          >
>          > _______________________________________________
>          > Users mailing list
>          > Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
>          > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>         _______________________________________________
>         Users mailing list
>         Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
>         http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>         _______________________________________________
>         Users mailing list
>         Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
>         http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> 
>          
> 
> 
>         _______________________________________________
>         Users mailing list
>         Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
>         http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com


More information about the Users mailing list