Performance Python
As I needed some heavy optimization of some inner loop in an O(n^2) problem, I turned to C extensions for Python this weekend. Well, writing native C extensions for Python (2.x) turned out to be a real mess. The Python C API isn’t something you want to use, which is understandable since Python is a heavily object oriented language where C is only procedural.
But, there are some cool extensions which make calling C code from Python easy. Of all those extensions, I tried out Pyrex.
Pyrex is basically a language that is very similar to Python. It’s so similar, that you can just cut-and-paste most of your Python code into a Pyrex module file (.pyx) (there are some limitations, e.g. no *-comprehension, but these can be fixed easily, most of the time). The next step is to compile the Pyrex Code into C code with the Pyrex compiler and then compile this into shared library with gcc. This shared library can then be imported into Python as a module and the methods can be called.
The fun starts when you a) start calling other native C libraries and b) start writing optimized Pyrex code. Calling native C libraries is pretty easy, most of the time. The Pyrex compiler builds the required code to convert the basic python types into C types. E.g. if you have a C function that want’s a char *p as an argument, you can just assign it with a python str-Object, same goes for int and float.
The second use-case is writing really fast Pyrex code. If you just cut-and-paste Python code, your variables will be complex Python-objects and the C output of the Pyrex compiler contains some really nasty Python-API calls on these. E.g. adding two python variables is anything else then trivial and results in something like this:
add.pyx:
a = 1
b = 2
c = a + b
turns into:
[…snip…]
PyObject *__pyx_1 = 0;
PyObject *__pyx_2 = 0;
PyObject *__pyx_3 = 0;
[..snip…]
/* “/home/kai/add.pyx”:1 */
[…snip (3 nasty lines for value assignment to Python-object)…]
/* “/home/kai/add.pyx”:2 */
[…snip (same here) …]
/* “/home/kai/add.pyx”:3 */
__pyx_1 = __Pyx_GetName(__pyx_m, __pyx_n_a); if (!__pyx_1) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 3; goto __pyx_L1;}
__pyx_2 = __Pyx_GetName(__pyx_m, __pyx_n_b); if (!__pyx_2) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 3; goto __pyx_L1;}
__pyx_3 = PyNumber_Add(__pyx_1, __pyx_2); if (!__pyx_3) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 3; goto __pyx_L1;}
Py_DECREF(__pyx_1); __pyx_1 = 0;
Py_DECREF(__pyx_2); __pyx_2 = 0;
if (PyObject_SetAttr(__pyx_m, __pyx_n_c, __pyx_3) < 0) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 3; goto __pyx_L1;}
Py_DECREF(__pyx_3); __pyx_3 = 0;
[…snip…]
This is really nasty and anything else then fast, isn’t it?
BUT! If we tell Pyrex to treat our variables as C integers, it can generate clean C code:
cdef int a
cdef int b
cdef int c
a = 1
b = 2
c = a + b
what we get is:
[…snip…]
static int __pyx_v_3add_a;
static int __pyx_v_3add_b;
static int __pyx_v_3add_c;
[…snip…]
/* “/home/kai/add.pyx”:4 */
__pyx_v_3add_a = 1;
/* “/home/kai/add.pyx”:5 */
__pyx_v_3add_b = 2;
/* “/home/kai/add.pyx”:6 */
__pyx_v_3add_c = (__pyx_v_3add_a + __pyx_v_3add_b);
[…snip…]
And this is just clean C code. The same goes for loops. If you just use your python code, alot of Python-object operations have to be used. But if you use C variables, then Pyrex can create really fast C code and can even do pointer arithmetics. Pyrex allows you to speed up your code where you really need it without all the hassle of using the Python C API.
And my code turned out about five times faster after doing just some minor C adjustments :)
As I mentioned above, there are some other methods of improving Python performance, there is a good overview from the people at SciPy. And before you turn to Python+C, check the Python-performance tips first.