Integrate Python and C/C++ (1)

In python official document there are two chapters with the topic of integrating python and C/C++

  • Extending and Embedding – tutorial for C/C++ programmer
  • Python/C API

In addition, there are several open-source utilities that make programmers’ lives easier, the most popular ones are

  • SWIG
  • Boost.Python/Pyl11
  • Cython
  • CFFI

This series posts explain the practical experience of the integration of python and C/C++.

Create a simple python module with C

The Python document gives a very simple example (spam), with full source code and detail explanation. What is missing is how you run on a real computer and what the output looks like.

Source Code

Here is the full source code with some concise comment

/* module method

 the name does not matter, and should be static, because the method
 is exposed by its implementation name, it is by method-define-array
 (see below)

 as practical convention, make your function name as module_method_name
*/

static PyObject*
spam_system(PyObject *self, PyObject *args)
{
 const char* command;

 int rc;

 if (!PyArg_ParseTuple(args, "s", &command))
 return NULL;

 rc = system(command);
 return Py_BuildValue("i", rc);
}

/*
 An array of PyMethodDef is passed to module initializer.
 This array defines all methods, each item is

 { name, function_pointer, argument_type, description }

 This array should also be static (no need to be exposed)
*/
static PyMethodDef SpamMethods[] = {
 { "system", spam_system, METH_VARARGS, "Execute a shell command." },

 // other methods

 // end of list
 {NULL, NULL, 0, NULL }
};


/*
 Each should have one (and only one) module initializer, that introduces
 module objects into python namespace.

 Its name MUST BE init.

 For example if the name is changed to init_spam, it still can be
 compiled but python can't import it:

python test.py echo hello
Traceback (most recent call last):
 File "test.py", line 4, in 
 import spam
ImportError: dynamic module does not define init function (initspam)

*/
PyMODINIT_FUNC
initspam(void)
{
 PyObject *m;

 m = Py_InitModule("spam", SpamMethods);
 if (m == NULL)
 return;
}

Manually build (on linux)

The document doesn’t mention the build process, instead it recommends to use distutils module. It is true, however, for curiosity I still decide to try to build manually. Here is the make file

$ cat manual.mk
# use pkg-config to find python information
#
# pkg-config --list-all
# pkg-config --cflags --libs python2
#
# -I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7 -lpython2.7
CFLAGS = -g -fPIC -I/usr/include/python2.7
LDFLAGS = -g -shared -fPIC -L/usr/lib/python2.7
LIBS = -lpython2.7

all: spam.so

spam.o: spam.c
        $(CC) -c $(CFLAGS) -o $@ $<

spam.so: spam.o
        $(LD) -o $@ $< $(LDFLAGS) $(LIBS)

clean:
        rm spam.so spam.o

The build process

$ make -f manual.mk
cc -c -g -fPIC -I/usr/include/python2.7 -o spam.o spam.c
ld -o spam.so spam.o -g -shared -fPIC -L/usr/lib/python2.7 -lpython2.7

The python script that tests our new module spam

$ cat test.py
#! /usr/bin/env python

import sys,os
import spam

cmd = ' '.join(sys.argv[1:])
print 'cmd=[{}]'.format(cmd)
rc = spam.system(cmd)
print 'rc={:x}'.format(rc)

And the result of running.

$ python test.py hello
cmd=[hello]
sh: 1: hello: not found
rc=7f00
$ python test.py echo hello
cmd=[echo hello]
hello
rc=0

Proper method of building (on linux)

As python document recommends, it is much easier (and more standard and flexible) to build with distutils module

$ cat setup.py
#from distutils.core import setup, Extension
# distutils.core is obsolete, use setuptools instead
from setuptools import setup, Extension

# a package consists of one or more modules, each module consists of
# one or more source files
# this is module spam, built by source: spam.c
module1 = Extension('spam', sources = ['spam.c'])

# this is a package spam, consist of module1
setup(name='spam',
      version='1.0',
      description='This is a demo package',
      ext_modules=[module1])

The process that run with distutils

$ python setup.py build
running build
running build_ext
building 'spam' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c spam.c -o build/temp.linux-x86_64-2.7/spam.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/spam.o -o build/lib.linux-x86_64-2.7/spam.so
jasonz@jzdebian $ find build
build
build/lib.linux-x86_64-2.7
build/lib.linux-x86_64-2.7/spam.so
build/temp.linux-x86_64-2.7
build/temp.linux-x86_64-2.7/spam.o

Compare the output of the two build method

$ ls -l spam.o build/lib.linux-x86_64-2.7/spam.so
-rwxr-xr-x 1 jasonz jasonz 17304 Nov 30 13:46 build/lib.linux-x86_64-2.7/spam.so
-rw-r--r-- 1 jasonz jasonz 16752 Nov 30 13:49 spam.o
$ file spam.o build/lib.linux-x86_64-2.7/spam.so
spam.o:                             ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
build/lib.linux-x86_64-2.7/spam.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=9cdc9b1ebf2df10d243fbd931ecea38f3bd63dfc, not stripped

Build with distutils on Windows

Python 3.5 and later support VS2015 and later. For Python 2.7, Microsoft offers a special version of VC++, which is actually an engine of VC++2008.

Note:

  1. Python 2.7 for Windows is built by VC2008, this can be checked by launching python interpreter and check its build info, which should by MSC v.1500
  2. Without VC2008, setup.py auto detect and complains error “Unable to find vcvarsall.bat“.
  3. some old version of setup.py uses module distutils.core, which lacks the auto-detect ability, replace it with setuptools
  4. Microsoft requests to update setuptools to version 6.0 or later for letting the special version VC++ works
  5. download setuptools 6.0.1 ez_setup.py
  6. run command python ez_setup.py
Here is the process of build module on Windows.
D:\codex\python\ext>python setup.py build
running build
running build_ext
building 'spam' extension
creating build
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
C:\Users\jasonz\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tcspam.c /Fobuild\tem
p.win-amd64-2.7\Release\spam.obj
spam.c
creating build\lib.win-amd64-2.7
C:\Users\jasonz\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild\amd64 /LIBPA
TH:C:\Python27\PC\VS9.0\amd64 /EXPORT:initspam build\temp.win-amd64-2.7\Release\spam.obj /OUT:build\lib.win-amd64-2.7\spam.pyd /IMPLIB:build\temp.win-amd64-2.7\Release\spam.lib /MANIFESTFILE:build\tem
p.win-amd64-2.7\Release\spam.pyd.manifest
spam.obj : warning LNK4197: export 'initspam' specified multiple times; using first specification
   Creating library build\temp.win-amd64-2.7\Release\spam.lib and object build\temp.win-amd64-2.7\Release\spam.exp

D:\codex\python\ext>python test.py echo hello
cmd=[echo hello]
hello
rc=0

D:\codex\python\ext>python test.py  hello
cmd=[hello]
'hello' is not recognized as an internal or external command,
operable program or batch file.
rc=1

D:\codex\python\ext>