boost 1.37.0 does not link against libpython

[fundawang@…]

libboost_python.so is calling python functions, but there is no direct linking to libpython, which will cause undefined references when building with LDFLAGS="-Wl,--no-undefined"

[Dave Abrahams]

I don't believe this can be considered a bug. Indeed, when Boost.Python is used for extending Python, it is important that libboost_python.so not be linked against libpython, since that will cause duplicate symbol errors. Python itself already contains all the symbols. Unless you can convince me that we should build otherwise-identical separate versions of libboost_python for extending and embedding (seems crazy to me), my answer is "don't use --no-undefined." If you have a good counterargment, please feel free to reopen this ticket.

[Jürgen Hunold]

I think that "direct linking" against libpython.so (for dynamic linking only, of course) will not embed the symbols in libboost_python.so, but enable the linker to verify that all weak references into libpython.so can be resolved at runtime. Using "--no-undefined" is a way to check that you explicitly link against _all_ required libraries. If linking "normally" gnu ld will link everything together happily (with weak symbols) and trust the runtime loader to pull everything together. In this case, your builds might fail on the windows platform for example where the msvc linker (and dllexport/-import) always enforce explicit listing of all needed libraries at link time. Qt > 4.3 uses this flag to improve cross platform consistency (in addition to gcc visibility support).

[Dave Abrahams]

I have some questions; please excuse my ignorance.

• What does "direct linking" mean, precisely?

• Will "direct linking" against libpython.so force libpython.so to be loaded when the Python executable loads an extension module that uses libboost_python.so?

• What does the windows platform have to do with this?

[Jürgen Hunold]

What does "direct linking" mean, precisely?

adding -lpython to the linker command line when linking libboost_python.so

Will direct linking" against libpython.so force libpython.so to be loaded when the Python executable loads an extension module that uses libboost_python.so?

Yes, of course. Does this cause any problems ? I've just dug out that I've patched python.jam to link extension modules against libpython.so due to a bug in the preview version of gold, the new ld replacement from Google which did not support undefined symbols at that time. Will attach the patch right now, although it seems not to be needed anymore with binutils 2.19. My (one and only experimental) module works fine with and without explicit linking to libpython.

What does the windows platform have to do with this?

Nothing directly. -no-undefined is just a tool to assist in cross-platform development. The usage is to spot unexported/undefined symbols when using gnu ld on unix platforms. Remember, gnu ld allows for symbols to be undefined at link time (this is used when linking libboost_python right now). msvc link.exe needs all symbols to be explicitly exported/imported for all libraries the current .dll depends on. So if you use gcc visibility and -no-undefined you are less likely to check out on windows and hit "missing" symbols due to forgotten "dllexport/-import" specifications." This is the way Qt goes and which I use to minimize broken builds on Windows because my main development platform is Linux.

[Jürgen Hunold]

Patch to link explicitly againt libpython

[anonymous]

Replying to jhunold:

What does "direct linking" mean, precisely?

adding -lpython to the linker command line when linking libboost_python.so

Will direct linking" against libpython.so force libpython.so to be loaded when the Python executable loads an extension module that uses libboost_python.so?

Yes, of course. Does this cause any problems ?

Yes, of course. It forces a dependency on libpython.so and increases the overall runtime image, aside from very likely causing duplicate copies of the code, and more worryingly, globals.

I've just dug out that I've patched python.jam to link extension modules against libpython.so due to a bug in the preview version of gold, the new ld replacement from Google which did not support undefined symbols at that time. Will attach the patch right now, although it seems not to be needed anymore with binutils 2.19. My (one and only experimental) module works fine with and without explicit linking to libpython.

Can you pass the Boost.Python test suite?

   cd libs/python/test
   bjam

What does the windows platform have to do with this?

Nothing directly. -no-undefined is just a tool to assist in cross-platform development. The usage is to spot unexported/undefined symbols when using gnu ld on unix platforms. Remember, gnu ld allows for symbols to be undefined at link time

Believe me, I know

(this is used when linking libboost_python right now). msvc link.exe needs all symbols to be explicitly exported/imported for all libraries the current .dll depends on. So if you use gcc visibility and -no-undefined you are less likely to check out on windows and hit "missing" symbols due to forgotten "dllexport/-import" specifications." This is the way Qt goes and which I use to minimize broken builds on Windows because my main development platform is Linux.

It's a nice technique, but it's not equivalent at all when it comes to Python because on Windows, python.exe uses the python shared library while on unix the python executable does not.

Still willing to have you re-open the ticket if you still believe I'm wrong.

[Arfrever]

Replying to anonymous:

on Windows, python.exe uses the python shared library while on unix the python executable does not.

On Unix, python executable uses python shared library if './configure --enable-shared' has been used during building Python. Majority of GNU/Linux distributions build Python with --enable-shared option.