In python official document there are two chapters with the topic of integrating python and C/C++
- Extending and Embedding – tutorial for C/C++ programmer
- Python/C API
In addition, there are several open-source utilities that make programmers’ lives easier, the most popular ones are
- SWIG
- Boost.Python/Pyl11
- Cython
- CFFI
This series posts explain the practical experience of the integration of python and C/C++.
Create a simple python module with C
The Python document gives a very simple example (spam), with full source code and detail explanation. What is missing is how you run on a real computer and what the output looks like.
Source Code
Here is the full source code with some concise comment
/* module method
the name does not matter, and should be static, because the method
is exposed by its implementation name, it is by method-define-array
(see below)
as practical convention, make your function name as module_method_name
*/
static PyObject*
spam_system(PyObject *self, PyObject *args)
{
const char* command;
int rc;
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
rc = system(command);
return Py_BuildValue("i", rc);
}
/*
An array of PyMethodDef is passed to module initializer.
This array defines all methods, each item is
{ name, function_pointer, argument_type, description }
This array should also be static (no need to be exposed)
*/
static PyMethodDef SpamMethods[] = {
{ "system", spam_system, METH_VARARGS, "Execute a shell command." },
// other methods
// end of list
{NULL, NULL, 0, NULL }
};
/*
Each should have one (and only one) module initializer, that introduces
module objects into python namespace.
Its name MUST BE init.
For example if the name is changed to init_spam, it still can be
compiled but python can't import it:
python test.py echo hello
Traceback (most recent call last):
File "test.py", line 4, in
import spam
ImportError: dynamic module does not define init function (initspam)
*/
PyMODINIT_FUNC
initspam(void)
{
PyObject *m;
m = Py_InitModule("spam", SpamMethods);
if (m == NULL)
return;
}
Manually build (on linux)
The document doesn’t mention the build process, instead it recommends to use distutils module. It is true, however, for curiosity I still decide to try to build manually. Here is the make file
$ cat manual.mk
# use pkg-config to find python information
#
# pkg-config --list-all
# pkg-config --cflags --libs python2
#
# -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -lpython2.7
CFLAGS = -g -fPIC -I/usr/include/python2.7
LDFLAGS = -g -shared -fPIC -L/usr/lib/python2.7
LIBS = -lpython2.7
all: spam.so
spam.o: spam.c
$(CC) -c $(CFLAGS) -o $@ $<
spam.so: spam.o
$(LD) -o $@ $< $(LDFLAGS) $(LIBS)
clean:
rm spam.so spam.o
The build process
$ make -f manual.mk
cc -c -g -fPIC -I/usr/include/python2.7 -o spam.o spam.c
ld -o spam.so spam.o -g -shared -fPIC -L/usr/lib/python2.7 -lpython2.7
The python script that tests our new module spam
$ cat test.py
#! /usr/bin/env python
import sys,os
import spam
cmd = ' '.join(sys.argv[1:])
print 'cmd=[{}]'.format(cmd)
rc = spam.system(cmd)
print 'rc={:x}'.format(rc)
And the result of running.
$ python test.py hello
cmd=[hello]
sh: 1: hello: not found
rc=7f00
$ python test.py echo hello
cmd=[echo hello]
hello
rc=0
Proper method of building (on linux)
As python document recommends, it is much easier (and more standard and flexible) to build with distutils module
$ cat setup.py
#from distutils.core import setup, Extension
# distutils.core is obsolete, use setuptools instead
from setuptools import setup, Extension
# a package consists of one or more modules, each module consists of
# one or more source files
# this is module spam, built by source: spam.c
module1 = Extension('spam', sources = ['spam.c'])
# this is a package spam, consist of module1
setup(name='spam',
version='1.0',
description='This is a demo package',
ext_modules=[module1])
The process that run with distutils
$ python setup.py build
running build
running build_ext
building 'spam' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c spam.c -o build/temp.linux-x86_64-2.7/spam.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/spam.o -o build/lib.linux-x86_64-2.7/spam.so
jasonz@jzdebian $ find build
build
build/lib.linux-x86_64-2.7
build/lib.linux-x86_64-2.7/spam.so
build/temp.linux-x86_64-2.7
build/temp.linux-x86_64-2.7/spam.o
Compare the output of the two build method
$ ls -l spam.o build/lib.linux-x86_64-2.7/spam.so
-rwxr-xr-x 1 jasonz jasonz 17304 Nov 30 13:46 build/lib.linux-x86_64-2.7/spam.so
-rw-r--r-- 1 jasonz jasonz 16752 Nov 30 13:49 spam.o
$ file spam.o build/lib.linux-x86_64-2.7/spam.so
spam.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
build/lib.linux-x86_64-2.7/spam.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=9cdc9b1ebf2df10d243fbd931ecea38f3bd63dfc, not stripped
Build with distutils on Windows
Python 3.5 and later support VS2015 and later. For Python 2.7, Microsoft offers a special version of VC++, which is actually an engine of VC++2008.
Note:
- Python 2.7 for Windows is built by VC2008, this can be checked by launching python interpreter and check its build info, which should by MSC v.1500
- Without VC2008, setup.py auto detect and complains error “Unable to find vcvarsall.bat“.
- some old version of setup.py uses module distutils.core, which lacks the auto-detect ability, replace it with setuptools
- Microsoft requests to update setuptools to version 6.0 or later for letting the special version VC++ works
- download setuptools 6.0.1 ez_setup.py
- run command python ez_setup.py
Here is the process of build module on Windows.
D:\codex\python\ext>python setup.py build
running build
running build_ext
building 'spam' extension
creating build
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
C:\Users\jasonz\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tcspam.c /Fobuild\tem
p.win-amd64-2.7\Release\spam.obj
spam.c
creating build\lib.win-amd64-2.7
C:\Users\jasonz\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild\amd64 /LIBPA
TH:C:\Python27\PC\VS9.0\amd64 /EXPORT:initspam build\temp.win-amd64-2.7\Release\spam.obj /OUT:build\lib.win-amd64-2.7\spam.pyd /IMPLIB:build\temp.win-amd64-2.7\Release\spam.lib /MANIFESTFILE:build\tem
p.win-amd64-2.7\Release\spam.pyd.manifest
spam.obj : warning LNK4197: export 'initspam' specified multiple times; using first specification
Creating library build\temp.win-amd64-2.7\Release\spam.lib and object build\temp.win-amd64-2.7\Release\spam.exp
D:\codex\python\ext>python test.py echo hello
cmd=[echo hello]
hello
rc=0
D:\codex\python\ext>python test.py hello
cmd=[hello]
'hello' is not recognized as an internal or external command,
operable program or batch file.
rc=1
D:\codex\python\ext>
XXX 