[IronPython] pybench results for CPython and IronPython

Jim Hugunin Jim.Hugunin at microsoft.com
Sun Apr 22 00:00:48 PDT 2007

I'd like to point out one fact about these benchmarks that seems to have been missed.  Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation.  The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.

I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now.  I compared with CPython-2.5.1 (the latest version).

Running on .NET, I find that IronPython is faster on some tests and CPython is faster on others.  We know that we still have performance work to do.  After all, IronPython just reached its 1.1 release and almost all of our recent work has been about compatibility and completeness.  However, the story is already quite interesting.  Out of the 51 tests in pybench, CPython is more than 2x faster on 10 of the tests and IronPython is more than 2x faster on 9 of the tests.  Depending on what your code does, either implementation could run faster.

The most interesting cases to me are the 5 tests where CPython is more than 3x faster than IronPython and the other 5 tests where IronPython is more than 3x faster than CPython.  CPython's strongest performance is in dictionaries with integer and string keys, list slicing, small tuples and code that actually throws and catches exceptions.  IronPython's strongest performance is in calling builtin functions, if/then/else blocks, calling python functions, deep recursion, and try/except blocks that don't actually catch an exception.

The places where IronPython is performing strongest are where it can use the .NET code generation and JIT optimization most effectively.  The places where it is slowest are mainly in runtime library implementation of core datatypes.  CPython's list and dictionary datatypes are written in C and have been hand-tuned for Python-style workloads over the past 10 years.  IronPython's datatypes are much newer and clearly need more tuning as the implementation matures.

The only two tests that really stand out as showing a deep performance issue are the two exception handling ones.  These are the only tests where either implementation is more than 4x faster than the other.  IronPython is 10x faster on the try/catch without an exception and CPython is 30x faster when an exception is actually raised.  This is a deliberate design decision within .NET to make code that doesn't throw exceptions run faster - even if that means slowing down code that does throw exceptions.  I'm fairly confident this was the right decision - and even remember the day long ago when Guido was discussing Python's exception system and explained that he'd accept almost any slow-down to the exceptional case in return for removing a single instruction from the non-exceptional path.

Thanks - Jim

My results
* using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.clock

Benchmark: ipy11.pybench

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    cli-32bit

       Implementation: Python
       Executable:     c:\ironpython-1.1\ipy.exe
       Version:        1.1.0
       Compiler:       X
       Bits:           32bit
       Build:          0 0 (#0)
       Unicode:        UCS2

Comparing with: py25.pybench

    Rounds: 10
    Warp:   10
    Timer:  time.clock

    Machine Details:
       Platform ID:    Microsoft-Windows-32bit-WindowsPE

       Implementation: Python
       Executable:     c:\python25\python.exe
       Version:        2.5.1
       Compiler:       MSC v.1310 32 bit (Intel)
       Bits:           32bit
       Build:          Apr 18 2007 08:51:08 (#r251:54863)
       Unicode:        UCS2

Test                             minimum run-time        average  run-time
                                 this    other   diff    this    other   diff
          BuiltinFunctionCalls:    44ms   181ms  -75.7%    48ms   182ms  -73.7%
           BuiltinMethodLookup:   335ms   160ms +109.1%   344ms   162ms +112.0%
                 CompareFloats:    99ms   111ms  -11.3%   105ms   112ms   -6.0%
         CompareFloatsIntegers:    63ms   125ms  -49.7%    65ms   126ms  -48.7%
               CompareIntegers:   102ms   114ms  -10.4%   105ms   115ms   -8.9%
        CompareInternedStrings:   175ms   126ms  +39.0%   180ms   127ms  +41.6%
                  CompareLongs:   111ms   105ms   +5.3%   114ms   106ms   +7.2%
                CompareStrings:   167ms   126ms  +32.7%   172ms   127ms  +35.1%
                CompareUnicode:   121ms   125ms   -3.4%   123ms   126ms   -2.0%
                 ConcatStrings:   482ms   272ms  +77.2%   533ms   275ms  +93.4%
                 ConcatUnicode:   305ms   216ms  +41.0%   350ms   217ms  +61.5%
               CreateInstances:   102ms   148ms  -31.1%   106ms   149ms  -28.9%
            CreateNewInstances:   326ms   131ms +148.0%   335ms   133ms +151.9%
       CreateStringsWithConcat:   228ms   147ms  +55.9%   238ms   148ms  +61.0%
       CreateUnicodeWithConcat:    91ms   154ms  -41.2%    94ms   157ms  -40.4%
                  DictCreation:   137ms   105ms  +30.0%   143ms   106ms  +33.9%
             DictWithFloatKeys:   301ms   230ms  +30.8%   306ms   232ms  +32.0%
           DictWithIntegerKeys:   343ms   106ms +223.0%   351ms   108ms +224.7%
            DictWithStringKeys:   388ms   102ms +282.1%   395ms   102ms +285.1%
                      ForLoops:    39ms    95ms  -59.0%    40ms    97ms  -58.5%
                    IfThenElse:    35ms   111ms  -68.9%    37ms   112ms  -66.8%
                   ListSlicing:   468ms   155ms +201.3%   477ms   157ms +204.4%
                NestedForLoops:    55ms   120ms  -54.1%    57ms   122ms  -53.0%
          NormalClassAttribute:   276ms   127ms +117.7%   282ms   129ms +119.7%
       NormalInstanceAttribute:   106ms   119ms  -11.0%   109ms   120ms   -9.7%
           PythonFunctionCalls:    37ms   130ms  -71.7%    40ms   131ms  -69.5%
             PythonMethodCalls:   127ms   158ms  -19.6%   132ms   159ms  -16.8%
                     Recursion:    49ms   177ms  -72.5%    50ms   179ms  -71.8%
                  SecondImport:   189ms   126ms  +50.4%   195ms   127ms  +53.6%
           SecondPackageImport:   196ms   134ms  +45.8%   202ms   136ms  +48.5%
         SecondSubmoduleImport:   266ms   177ms  +50.6%   272ms   180ms  +51.3%
       SimpleComplexArithmetic:    97ms   148ms  -34.4%   101ms   149ms  -32.0%
        SimpleDictManipulation:   311ms   118ms +164.4%   318ms   118ms +168.0%
         SimpleFloatArithmetic:    64ms   119ms  -46.6%    69ms   121ms  -43.2%
      SimpleIntFloatArithmetic:    41ms   100ms  -59.4%    43ms   102ms  -58.2%
       SimpleIntegerArithmetic:    43ms   100ms  -57.2%    44ms   101ms  -56.4%
        SimpleListManipulation:   121ms   104ms  +16.6%   124ms   105ms  +18.3%
          SimpleLongArithmetic:   126ms   113ms  +11.2%   131ms   115ms  +14.1%
                    SmallLists:   223ms   154ms  +45.0%   229ms   156ms  +47.0%
                   SmallTuples:   508ms   144ms +252.2%   523ms   145ms +259.7%
         SpecialClassAttribute:   275ms   124ms +121.2%   283ms   126ms +124.5%
      SpecialInstanceAttribute:   105ms   210ms  -50.1%   109ms   212ms  -48.5%
                StringMappings:   342ms   633ms  -46.0%   351ms   637ms  -44.9%
              StringPredicates:   222ms   217ms   +2.3%   228ms   219ms   +4.4%
                 StringSlicing:   245ms   162ms  +51.4%   260ms   163ms  +59.4%
                     TryExcept:     7ms   100ms  -93.4%    11ms   102ms  -89.4%
                TryRaiseExcept:  3413ms   114ms +2896.2%  3453ms   115ms +2910.6%
                  TupleSlicing:   227ms   150ms  +51.6%   235ms   151ms  +54.9%
               UnicodeMappings:   227ms   126ms  +79.6%   233ms   129ms  +81.4%
             UnicodePredicates:   218ms   129ms  +68.8%   224ms   132ms  +70.4%
                UnicodeSlicing:   211ms   182ms  +15.8%   223ms   184ms  +21.3%
Totals:                         12781ms  7659ms  +66.9% 13191ms  7741ms  +70.4%

(this=ipy11.pybench, other=py25.pybench)

More information about the users mailing list