Python is fast enough for the vast majority of programs. And in those cases where it isn’t, we can often achieve sufficient speedups by using concurrency, as we saw in the previous chapter. Sometimes, though, we really do need to do faster processing. There are three key ways we can make our Python programs run faster: we can use PyPy (pypy.org), which has a built-in JIT (Just in Time compiler); we can use C or C++ code for time-critical processing; or we can compile our Python (or Cython) code into C using Cython.★
Once PyPy is installed, we can execute our Python programs using the PyPy interpreter rather than the standard CPython interpreter. This will give us a significant speedup for long-running programs, since the cost of the JIT compiling will be outweighed by the reduced runtime, but might produce slower execution for programs with very short runtimes.
To use C or C++, whether our own code or third-party libraries, we must make the code available to our Python program so that it can benefit from the C or C++ code’s fast execution. For those who want to write their own C or C++ code, a sensible approach is to make direct use of the Python C in- terface (docs.python.org/3/extending). For those who want to make use of ex- isting C or C++ code, there are several possible approaches. One option is to use a wrapper that will take the C or C++ and produce a Python interface for it. Two popular tools for this are SWIG (www.swig.org) and SIP (www.riverbank- computing.co.uk/software/sip). Another option for C++ is to useboost::python (www.boost.org/libs/python/doc/). A newer entry into this field is CFFI (C For- eign Function Interface for Python), which despite its newness is being used by the well-established PyPy (bitbucket.org/cffi/cffi).
★New Python compilers are becoming available; for example, Numba (numba.pydata.org) and Nuitka (nuitka.net).
179
ptg11539634
Extending Python on OS X and Windows !
Although the examples in this chapter have been tested only on Linux, they should all work fine on both OS X and Windows. (For manyctypesand Cython programmers, these are their primary development platforms.) How- ever, some platform-specific tweaks may be necessary. This is because where- as most Linux systems use a packaged GCC compiler and system-wide li- braries with the appropriate word size for the machine they are running on, the situation for OS X and Windows systems is usually more complicated, or at least a bit different.
On OS X and Windows, it is generally necessary to match the compiler and word size (32- or 64-bit) used to build Python with that used for any external shared libraries (.dylibor.DLLfiles) or to build Cython code. On OS X, the compiler might be GCC but nowadays is most likely to be Clang; on Windows it could be some form of GCC or a commercial compiler such as those sold by Microsoft. Furthermore, OS X and Windows often have shared libraries in application directories rather than system wide, and header files may need to be obtained separately. So, rather than giving lots of platform- and compiler-specific configuration information (which might quickly become outdated with new compiler and operating system versions), we focus instead on how tousectypes and Cython, leaving readers on non-Linux systems to determine their own system’s particular requirements when they are ready to use these technologies.
All of the possibilities described so far are worth exploring, but in this chapter we will focus on two other technologies: thectypespackage that comes as part of Python’s standard library (docs.python.org/3/library/ctypes.html) and Cython (cython.org). Both of these can be used to provide Python interfaces for our own or for third-party C and C++ code, and Cython can also be used to compile both Python and Cython code into C to improve its performance—sometimes with dramatic results.
5.1. Accessing C Libraries with ctypes
The standard library’sctypespackage provides access to our own or third-party functionality written in C or C++ (or indeed any compiled language that uses the C calling convention) and that has been compiled into a stand-alone shared library (.soon Linux,.dylibon OS X, or.DLLon Windows).
For this section, and for the following section’s first subsection (§5.2.1,➤188), we will create a module that provides Python access to some C functions in a shared library. The library we will use is libhyphen.so, or, on some systems, libhyphen.uno.so. (See the “Extending Python on OS X and Windows” sidebar.) This library usually comes with OpenOffice.org or LibreOffice and provides a
ptg11539634
5.1. Accessing C Libraries with ctypes 181
function that, when given a word, produces a copy of the word with hyphens inserted wherever they are valid. Although the function does what sounds like a simple task, the function’s signature is quite complicated (which makes it ideal as actypesexample). And, in fact, there are three functions that we will need to use: one for loading in a hyphenation dictionary, one for doing the hyphenation, and one for freeing up resources when we have finished.
A typical pattern of use for ctypes is to load the library into memory, take references to the functions we want to use, then call the functions as required.
TheHyphenate1.pymodule follows this pattern. First, let’s see how the module is used. Here is an interactive session done at a Python prompt (e.g., in IDLE):
>>> import os
>>> import Hyphenate1 as Hyphenate
>>>
>>> # Locate your hyph*.dic files
>>> path = "/usr/share/hyph_dic"
>>> if not os.path.exists(path): path = os.path.dirname(__file__)
>>> usHyphDic = os.path.join(path, "hyph_en_US.dic")
>>> deHyphDic = os.path.join(path, "hyph_de_DE.dic")
>>>
>>> # Create wrappers so you don't have to keep specifying the dictionary
>>> hyphenate = lambda word: Hyphenate.hyphenate(word, usHyphDic)
>>> hyphenate_de = lambda word: Hyphenate.hyphenate(word, deHyphDic)
>>>
>>> # Use your wrappers
>>> print(hyphenate("extraordinary")) ex-traor-di-nary
>>> print(hyphenate_de("auòergewửhnlich")) auòerge-wửhn-lich
The only function we use outside the module isHyphenate1.hyphenate(), which uses the library’s hyphenation function. Inside the module there are a couple of private functions that access another couple of functions from the library. Inci- dentally, the hyphenation dictionaries are in the format used by the open-source TEX typesetting system.
All the code is in theHyphenate1.pymodule. The three functions we need from the library are:
HyphenDict *hnj_hyphen_load(const char *filename);
void hnj_hyphen_free(HyphenDict *hdict);
int hnj_hyphen_hyphenate2(HyphenDict *hdict, const char *word, int word_size, char *hyphens, char *hyphenated_word, char ***rep, int **pos, int **cut);
ptg11539634 These signatures are taken from thehyphen.h header file. A * in C and C++
signifies apointer. A pointer holds the memory address of a block of memory;
that is, of a contiguous block of bytes. The block may be as small as a single byte but could be of any size; for example, 8 bytes for a 64-bit integer. Strings typically take between 1 and 4 bytes per character (depending on the in-memory encoding) plus some fixed overhead.
The first function,hnj_hyphen_load(), takes a filename passed as a pointer to a block ofchars (bytes). This file must be a hyphenation dictionary in TEX format.
The hnj_hyphen_load() function returns a pointer to a HyphenDict struct—a complex aggregate object (rather like a Python class instance). Fortunately, we don’t need to know anything about the internals of aHyphenDict, since we only ever need to pass around pointers to them.
In C, functions that acceptC-strings—that is, pointers to blocks of characters or bytes—normally take one of two approaches: either they require just a pointer, in which case they expect the last byte to be0x00('\0') (that is, for the C-string to be null-terminated), or they take a pointer and a byte count. The hnj_hyphen_load()function takes only a pointer, so the given C-string must be null terminated. As we will see, if thectypes.create_string_buffer()function is passed astr, it returns an equivalent null-terminated C-string.
For every hyphenation dictionary that we load, we must eventually free it. (If we don’t do this, the hyphenation library will stay in memory needlessly.) The second function, hnj_hyphen_free(), takes a HyphenDict pointer and frees the resources associated with it. The function has no return value. Once freed, such a pointer must never be reused, just as we would never use a variable after it has been deleted withdelin Python.
The third function,hnj_hyphen_hyphenate2(), is the one that performs the hy- phenation service. Thehdict argument is a pointer to a HyphenDict that has been returned by thehnj_hyphen_load()function (and that has not yet been freed with thehnj_hyphen_free()function). Thewordis the word we want to hyphenate provided as a pointer to a block of UTF-8-encoded bytes. Theword_sizeis the number of bytes in the block. Thehyphensis a pointer to a block of bytes that we don’t want to use, but we must still pass a valid pointer for it for the function to work correctly. Thehyphenated_wordis a pointer to a block of bytes long enough to hold the original UTF-8-encoded word with hyphens inserted. (The library actually inserts= characters as hyphens.) Initially, this block should hold all 0x00bytes. Therepis a pointer to a pointer to a pointer to a block of bytes; we don’t need this, but we must still pass a valid pointer for it. Similarly,posand cutare pointers to pointers toints that we aren’t interested in, but we must still pass valid pointers for them. The function’s return value is a Boolean flag, with 1 signifying failure and 0 signifying success.
ptg11539634
5.1. Accessing C Libraries with ctypes 183
Now that we know what we want to wrap, we will review theHyphenate1.py module’s code (as usual, omitting the imports), starting with finding and loading the hyphenation shared library.
class Error(Exception): pass
_libraryName = ctypes.util.find_library("hyphen") if _libraryName is None:
_libraryName = ctypes.util.find_library("hyphen.uno") if _libraryName is None:
raise Error("cannot find hyphenation library") _LibHyphen = ctypes.CDLL(_libraryName)
We begin by creating an exception class,Hyphenate1.Error, so that users of our module can distinguish between module-specific exceptions and more general ones likeValueError. Thectypes.util.find_library()function looks for a shared library. On Linux it will prefix the given name withliband add an extension of .so, so the first call will look forlibhyphen.soin various standard locations. On OS X, it will look forhyphen.dylib, and on Windows, forhyphen.dll. This library is sometimes calledlibhyphen.uno.so, so we search for this if it wasn’t found under the original name. And if we can’t find it, we give up by raising an exception.
If the library is found, we load it into memory using thectypes.CDLL()func- tion and set the private_LibHyphen variable to refer to it. For those wanting to write Windows-only programs that access Windows-specific interfaces, the ctypes.OleDLL()andctypes.WinDLL()functions can be used to load Windows API libraries.
Once the library is loaded, we can create Python wrappers for the library func- tions we are interested in. A common pattern for this is to assign a library function to a Python variable, and then specify the types of the arguments (as a list ofctypestypes) and the return type (as a singlectypestype) that the func- tion uses.
If we specify the wrong number or types of arguments, or the wrong return type, our program will crash! The CFFI package (bitbucket.org/cffi/cffi) is more robust in this respect and also works much better with the PyPy interpreter (pypy.org) thanctypes.
_load = _LibHyphen.hnj_hyphen_load
_load.argtypes = [ctypes.c_char_p] # const char *filename _load.restype = ctypes.c_void_p # HyphenDict *
Here, we have created a private module function,_load(), that when called will actually call the underlying hyphenation library’shnj_hyphen_load() function.
Once we have a reference to the library function, we must specify its argument
ptg11539634 and return types. Here, there is just one argument (of C typeconst char *),
which we can represent directly withctypes.c_char_p (“C character pointer”).
The function returns a pointer to aHyphenDict struct. One way to handle this would be to create a class that inheritsctypes.Structureto represent the type.
However, since we only ever have to pass this pointer around and never access what it points to ourselves, we can simply declare that the function returns a ctypes.c_void_p(“C void pointer”), which can point to any type at all.
These three lines (in addition to finding and loading the library in the first place) are all we need to provide a_load()method that will load a hyphenation dictionary.
_unload = _LibHyphen.hnj_hyphen_free
_unload.argtypes = [ctypes.c_void_p] # HyphenDict *hdict _unload.restype = None
The code here follows the same pattern as before. The hnj_hyphen_free() function takes a single argument, a pointer to aHyphenDict struct, but since we only ever pass such pointers, we can safely specify avoidpointer—providing we always actually pass in aHyphenDict structpointer. This function has no return value; this is signified by setting itsrestypetoNone. (If we don’t specify arestype, it is assumed that the function returns anint.)
_int_p = ctypes.POINTER(ctypes.c_int) _char_p_p = ctypes.POINTER(ctypes.c_char_p) _hyphenate = _LibHyphen.hnj_hyphen_hyphenate2 _hyphenate.argtypes = [
ctypes.c_void_p, # HyphenDict *hdict ctypes.c_char_p, # const char *word ctypes.c_int, # int word_size
ctypes.c_char_p, # char *hyphens [not needed]
ctypes.c_char_p, # char *hyphenated_word _char_p_p, # char ***rep [not needed]
_int_p, # int **pos [not needed]
_int_p] # int **cut [not needed]
_hyphenate.restype = ctypes.c_int # int
This is the most complicated function we need to wrap. Thehdict argument is a pointer to aHyphenDict struct, which we specify as a Cvoid pointer. Then we have thewordto be hyphenated, passed as a pointer to a block of bytes for which we use a C character pointer. This is followed by theword_size, a count of the bytes that we specify as an integer (ctypes.c_int). Next, we have the hyphensbuffer that we don’t need, then thehyphenated_word, again specified as a C character pointer. There is no built-inctypestype for a pointer to a pointer to a character (byte), so we have created our own type,_char_p_p, specifying it
ptg11539634
5.1. Accessing C Libraries with ctypes 185
as a pointer to a C character pointer. We have done a similar thing for the two pointers to pointer to integers.
Strictly speaking, we don’t have to specify arestype, since the function’s return type is an integer, but we prefer to be explicit.
We have created private wrapper functions for the hyphenation library’s func- tions, since we want to insulate users of our module from the low-level details.
To this end, we will provide a single public function,hyphenate(), which will ac- cept a word to be hyphenated, a hyphenation dictionary to use, and the hyphen- ation character to use. For efficiency, we will only ever load any particular hy- phenation dictionary once. And, of course, we will make sure that all hyphen- ation dictionaries that have been loaded are freed at program termination.
def hyphenate(word, filename, hyphen="-"):
originalWord = word
hdict = _get_hdict(filename) word = word.encode("utf-8")
word_size = ctypes.c_int(len(word)) word = ctypes.create_string_buffer(word)
hyphens = ctypes.create_string_buffer(len(word) + 5)
hyphenated_word = ctypes.create_string_buffer(len(word) * 2) rep = _char_p_p(ctypes.c_char_p(None))
pos = _int_p(ctypes.c_int(0)) cut = _int_p(ctypes.c_int(0))
if _hyphenate(hdict, word, word_size, hyphens, hyphenated_word, rep, pos, cut):
raise Error("hyphenation failed for '{}'".format(originalWord)) return hyphenated_word.value.decode("utf-8").replace("=", hyphen)
The function begins by storing a reference to the word passed in to be hyphenat- ed so that we can use it in an error message, if necessary. Then, we get the hy- phenation dictionary: the private_get_hdict()function returns a pointer to the HyphenDict structthat corresponds to the given filename. If the dictionary has already been loaded, the pointer created at that time is returned; otherwise, the dictionary is loaded for the first and only time, its pointer stored for later use, and returned.
The word must be passed to the hyphenation function as a block of UTF-8- encoded bytes, which is easily achieved using the str.encode() method. We also need to pass the number of bytes the word occupies: we compute this and convert the Pythonintinto a Cint. We can’t pass a raw Pythonbytesobject to a C function, so we create a string buffer (really a block of Cchars) that contains the word’s bytes. Thectypes.create_string_buffer()creates a block of Cchars based on abytesobject or of the given size. Although we don’t want to use the hyphensargument, we must still properly prepare it, and the documentation says
ptg11539634 that it must be a pointer to a block of Cchars whose length is five more than
the length of the word (in bytes). So, we create a suitable block of chars. The hyphenated word will be put into a block of Cchars that is passed to the function, so we must make a block of sufficient size. The documentation recommends a size twice that of the word’s size.
We don’t want to use therep,pos, orcutarguments, but we must pass correct values for them or the function won’t work. Therepis a pointer to a pointer to a pointer to a block of Cchars, so we have created a pointer to an empty block (a null pointer in C, i.e., a pointer that points to nothing) and then assigned a pointer to a pointer to this pointer to therep variable. For thepos andcut arguments, we have created pointers to pointers to integers of value 0.
Once all the arguments have been set up, we call the private _hyphenate() function (under the hood, we are really calling the hyphenation library’s hnj_hyphen_hyphenate2()function) and raise an error if the function returns a nonzero (i.e., failure) result. Otherwise, we extract the rawbytesfrom the hy- phenated word using thevalueproperty (which returns a null-terminatedbytes, i.e., one whose last byte is0x00). Then we decode the bytes using the UTF-8 en- coding into astrand replace the hyphenation library’s=hyphens with the user’s preferred hyphen (which defaults to-). This string is then returned as thehy- phenate()function’s result.
Note that for C functions that usechar *and sizes rather than null-terminated strings, we can access the bytes using theraw property rather than thevalue property.
_hdictForFilename = {}
def _get_hdict(filename):
if filename not in _hdictForFilename:
hdict = _load(ctypes.create_string_buffer(
filename.encode("utf-8"))) if hdict is None:
raise Error("failed to load '{}'".format(filename)) _hdictForFilename[filename] = hdict
hdict = _hdictForFilename.get(filename) if hdict is None:
raise Error("failed to load '{}'".format(filename)) return hdict
This private helper function returns a pointer to aHyphenDict struct, reusing pointers to dictionaries that have already been loaded.
If the filename is not in the _hdictForFilename dict, it is a new hyphenation dictionary and must be loaded. Because the filename is passed as a Cconst char *(i.e., is immutable), we can create and pass it as a ctypesstring buffer