Application crash when using Boost.Python in a plug-in DLL for that application

[Emmanuel Giasson <zirconia_@…>]

I am developping a DLL for a Win32 application that already host a Python interpreter (Python 2.5.x) which I extend using Boost.Python.

As far as Boost.Python is concerned, the sequence of operations (which I can't change) is :

1. App calls Py_Initialize

2. App loads my DLL

3. my DLL extend Python with a "boost module" (such as the "Hello World" example in the documentation

4. App notifies my DLL that it is about to be unloaded

5. App unloads my DLL

6. App calls Py_Finalize... and crash!

The application doesn't use Boost.Python, and doesn't even share the C run-time (i.e. different heaps). Whether I use static or dynamic linking of Boost.Python doesn't seem relevant in that case. Yes, I know, Boost.Python doc states that Py_Finalize should not be called. But in that case, it IS called after my DLL is unloaded and I can't do anything about it (and I'd like to use Boost.Python in such a scenario).

Cause :

I manage to pinpoint where it fails from Py_Finalize. At that point, the 'Boost.Python.function' python object (defined in the function_type global variable in libs/python/src/object/function.cpp) is internally referenced by Python (by base class type among other things) and Python eventually tries to access it (at least change its ref count I guess). However, since the DLL is already unloaded, the memory that held the global variable is not mapped anymore.

Resolution :

Not being a Python expert, I haven't found a way to remove references to that 'Boost.Python.function' when my DLL is notified about being unloaded. I'd be glad to know if there is. However, it was fairly simple to get Python to allocate this object in its own heap, so that it lives as long as the interpreter. Attached is the patch file for that fix. I can't tell though how "clean" or not is copying a complete 'PyTypeObject' structure instance over a somewhat-already-initialized instance from PyObject_New()... but that seems to work.

Note that to avoid the crash, I also need to "undef" all function definitions from the boost module with PyObject_DelAttr() (else Py_Finalize accesses them but they're from a heap that was deallocated from memory when the DLL was unloaded). That does not however require any modification to Boost.Python.

[Emmanuel Giasson <zirconia_@…>]

Potential way to fix that issue

[Emmanuel Giasson <zirconia_@…>]

I just realized that my fix may cause another bug should the application "reset" the Python interpreter (such as calling Py_Finalize then Py_Initialize) while the DLL with Boost.Python is still loaded. The pointer in function.cpp on the Boost.Python.function object then becomes a dangling reference. Perhaps a cleanup function registered with Py_AtExit could fix that.

[erkki.vahala@…]

Boost python has also other problems related to initialization via static variables: on Windows, loading a DLL, which defines boost python modules with BOOST_PYTHON_MODULE macros, causes RTTI data to be stored in a static registry (converter/registry.cpp). If the DLL is then unloaded, registry will contain danging pointers - subsequent attempts to load DLLs with modules/exercise the registry by other means will cause a memory access violation as the registry tree comparison operations try to access the stale RTTI data.

[Amaury Forgeot d'Arc <amauryfa@…>]

Unloading a Python extension module is not supported. Even if you manage to copy the type, all its members (even tp_name!) are invalid, and will lead to a crash.

Static types are static; this is a limitation of CPython, not a defect of Boost.Python.