bjam hangs when building libs/python/examples/quickstart under MacOS X

[konrad.hinsen@…]

After compilation/installation of Boost, trying to build the Boost.Python example in libs/python/examples/quickstart makes bjam hang after executing the step

darwin.link.dll bin/darwin-4.0.1/debug/extending.so

The library file extending.so is still built (apparently correctly), but bjam does not continue after this step. By running the link command line (obtained with bjam -n -a) directly in the terminal, I have verified that it is not the linker that hangs.

Tested with today's development trunk under MacOS 10.4.11 (Intel) and with Boost 1.35.0 under MacOS 10.4.11 (Intel and PowerPC). I have found reports about the same problem under MacOS 10.5 in various newsgroups.

[Marshall Clow]

Just tried it with the current trunk and gcc 4.0.1 (Xcode 3.1 install) and $ gcc --version i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5484) $ bjam --version Boost.Build V2 (Milestone 12) Boost.Jam 03.1.16

No hang - closing as "works for me" Please re-open if this still happens for you using current boost/gcc/bjam.

[konrad.hinsen@…]

I updated to XCode 2.5 (the latest for MacOS 10.4) and tried with boost 1.36.0. It still hangs, at the same place.

$ gcc --version i686-apple-darwin8-gcc-4.0.1 (GCC) 4.0.1 (Apple Computer, Inc. build 5370)

$ ./tools/jam/src/bin.macosxx86/bjam --version Boost.Build V2 (Milestone 12) Boost.Jam 03.1.16

Note: I tried to change the status to "reopen", but apparently I do not have the necessary permissions. The site just says "Warning: No permission to change ticket fields." I don't see a way to create an account either, so I am blocked. Now I try to add my comment without changing the status as this still seems better than doing nothing at all...

[jason@…]

I am having the same problem. Building the example "quickstart" for Boost.Python hangs on the line described above. Versions: OS X - 10.5.6 gcc - i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5490) Boost 1.38.0 Boost.Jam 03.1.17

I've also tried using bjam -n -a to get at the commands being run and to run them manually, as outlined here: ​http://mail.python.org/pipermail/cplusplus-sig/2008-August/013580.html

I managed to run the command that hangs under bjam, successfully. But in the next step I ran into "darwin.link.dll: command not found" and that (alias?) didn't seem to be explicitly defined in the commands I saw, so I set this aside for the moment.

[Steven Watanabe]

darwin.link.dll is a Boost.Build rule, not a command that you can run from a shell.

[Dave Abrahams]

This is happening for me, too. I'm not sure where to start in trying to fix it, though!

[Dave Abrahams]

Another strange fact: adding -d+2 doesn't cause the command to be printed out before Python is started. I even added an explicit fflush to the bjam sources to try to force it. If I do a "ps waux" I can clearly see that it has started running the Python command, though!

[anonymous]

Replying to dave:

Another strange fact: adding -d+2 doesn't cause the command to be printed out before Python is started. I even added an explicit fflush to the bjam sources to try to force it. If I do a "ps waux" I can clearly see that it has started running the Python command, though!

This was part of [38009]. It's annoying and should probably be disabled for single threaded builds.

[anonymous]

could somebody run bjam with '-l5' (undocumented) option and see if it helps?

[anonymous]

updated

[Dave Abrahams]

With -l5, (that's a lowercase "L", right?) I get:

#!/bin/sh
$ bjam -l5
...patience...
...patience...
...found 3236 targets...
...updating 2 targets...
capture-output ../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test
5 second time limit exceeded

    DYLD_LIBRARY_PATH=/System/Library/Frameworks/Python.framework/Versions/2.5/lib:/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config:/Users/dave/src/boost/bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug:/Users/dave/src/boost/bin.v2/libs/python/build/darwin-4.0.1/debug:$DYLD_LIBRARY_PATH
export DYLD_LIBRARY_PATH

    PYTHONPATH=../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug
export PYTHONPATH
 "/usr/bin/python2.5" "python_test.py"   > "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output" 2>&1
    status=$?
    echo >> "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output"
    echo EXIT STATUS: $status >> "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output"
    if test $status -eq 0 ; then
        cp "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output" "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test"
    fi
    verbose=0
    if test $status -ne 0 ; then
        verbose=1
    fi
    if test $verbose -eq 1 ; then
        echo ====== BEGIN OUTPUT ======
        cat "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output"
        echo ====== END OUTPUT ======
    fi
    exit $status

...failed capture-output ../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test...
...failed updating 1 target...
...skipped 1 target...
zreba:test dave$ 

I did this one in /libs/parameter/test because it has only one Python-related test. I don't know if you're suggesting it, but I'll try the patch next.

[Dave Abrahams]

Applying the patch makes no apparent difference to my results

[Dave Abrahams]

I'm able to run these tests with CMake, so this is not a blocker for me, but it surely is for someone else!

[ilyasokol@…]

removes o_nonblock from both ends of pipe

[ilyasokol@…]

If the patch above doesn't work then I am out of ideas. I hope we are hitting this bug: ​http://lists.apple.com/archives/Darwin-kernel/2005/Dec/msg00058.html

[Dave Abrahams]

Makes no difference for me, I'm afraid. Note, though, that the 2nd hunk of your patch applied at an offset of 4 lines, which makes me think you're working with a different version of the source.

If you need it I can hook you up with a MacOS account to try this on.

[anonymous]

Replying to dave:

If you need it I can hook you up with a MacOS account to try this on.

That would be great. Please contact me by email ilyasokol<at>gmail<dot>com

[Randolph Fritz <rfritz@…>]

Still seeing this with Boost 1.39.0 on Mac OS X 10.5.7. I will try adding the timeout described in ​http://article.gmane.org/gmane.comp.lib.boost.build/18302. To be continued...

[Randolph Fritz <rfritz@…>]

The hang actually occurs in test_extending.py. I am starting to doubt this is a bjam problem.

[Randolph Fritz <rfritz@…>]

Situation is:

• Configure dynamic libraries and PYTHONPATH from command line. Run "python test_extending.py". Test succeeds.

• Run tests from bjam, with command "bjam test_ext". Test hangs.

• Run tests from bjam, with timeout; command "bjam -l15 test_ext". Test fails by timeout.

It is probably some subtle environment-related bug, either in the Apple release of python 2.5.1 or in Boost.Python itself, but so far I am unable to determine which.

[anonymous]

I know it's been bumped down, but can this be bumped back up to showstopper? I't seem like one -- I can find no consistent workaround, and it's made boost.python unavailable to what appears to be a large segment of population for at least a year now. (I've spoken to two other people who have abandoned boost because of this bug.)

[anonymus]

apparently, somebody got it running here ​http://lists.boost.org/boost-users/2009/02/45404.php

go to later posts in this thread.

[Marshall Clow]

Additional data point: I just tried this on Mac OS X 10.6.2 / gcc 4.2.1 (Intel), and was unable to reproduce.

[anonymous]

FWIW, 'bjam toolset=darwin test' still hangs for me. However, 'extending' and 'embedding' both compile without problems, and I can import the generated extension module (after moving 'libboost_python.dylib' into a library path.) It sounds like 'test' works on Snow Leopard, but not on Leopard. Given that boost.python obviously works on Leopard, this isn't a showstopper for me, but it would have been really great if there had been a note somewhere, because I all but abandoned boost.python for over a year because I thought it simply didn't work on OS X (and others have said the same).

• OS X 10.5.8

• Developer Tools 3.1.4

• gcc version 4.0.1 (Apple Inc. build 5493)

• boost_1_43_0

• Boost.Jam Version 3.1.17. OS=MACOSX

[anonymous]

hanging at:

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

---

OS X 10.6.4

darwin-4.2.1

boost_1_44_0

Boost.Jam 03.1.18

[anonymous]

hanging at:

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

---

OS X 10.6.4

darwin-4.2.1

boost_1_44_0

Boost.Jam 03.1.18

[meowsqueak@…]

I have this problem right now with:

$ bjam toolset=darwin --verbose-test test

It hangs after the line:

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

Ctrl-c and inspecting the bin/ directory shows some .o and .so files, but no .dylib files, in bin/darwin-4.2.1/debug/ but these don't seem to be 'importable' by Python.

This command (not sure if that's the right PYTHONPATH?) fails:

$ PYTHONPATH=bin/darwin-4.2.1/debug python test_extending.py ... ImportError: dlopen(/Users/david/Downloads/boost_1_50_0/libs/python/example/quickstart/bin/darwin-4.2.1/debug/extending.so, 2): Library not loaded: libboost_python.dylib

Not sure why bjam created a .so instead of a .dylib?

My platform: *OS X 10.6.8 64bit *Python 2.7.3 (built locally with pythonbrew) *i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3) *boost_1_50_0 *Boost.Build 2011.12-svn (bjam built locally from boost_1_50_0)

[meowsqueak@…]

Replying to meowsqueak@…:

This command (not sure if that's the right PYTHONPATH?) fails:

$ PYTHONPATH=bin/darwin-4.2.1/debug python test_extending.py ... ImportError: dlopen(/Users/david/Downloads/boost_1_50_0/libs/python/example/quickstart/bin/darwin-4.2.1/debug/extending.so, 2): Library not loaded: libboost_python.dylib

Just as a follow up, after hitting Ctrl-C at the hang, I was able to successfully run 'test_extending.py' with:

 $ export DYLD_LIBRARY_PATH=../../../../bin.v2/libs/python/build/darwin-4.2.1/debug
 $ export LD_LIBRARY_PATH=../../../../bin.v2/libs/python/build/darwin-4.2.1/debug
 $ export PYTHONPATH=bin/darwin-4.2.1/debug
 $ python test_extending.py
 ...
  9 passed and 0 failed.
 Test passed.

[jamestwebber@…]

I'm having the exact problem that meowsqueak reports: bjam hanged after

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

But I can run test_extending.py fine after killing the command and export the relevant variables. No new wrinkles from what's been previously reported, but wanted to add another data point.

I'm using OS X 10.7.5 and python 2.7.3 built with homebrew, boost 1.50.0 and bjam built locally from boost.

Digging a bit into the commands, I can see as dave did that Python is running, with the command

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/bin/python" "test_extending.py"   > "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output" 2>&1

But when I run that command alone (after compiling) it runs fine. The test_ext.output looks fine except that it starts with the characters \E[?1034h which appears to be related to some weird behavior from readline (CAPTCHA isn't working so I can't post this link, but: search for "readline+1034h")? I have no idea if that might cause a problem--I'm not sure how it would.

If I kill the Python process it remains hanging, but killing the bash script stops the test (which then fails because the output is wrong).

Actually…

I tried running the testing script manually, by pasting this into the terminal:

/bin/sh -c 
    DYLD_LIBRARY_PATH="/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib:/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config:/usr/local/boost_1_51_0/bin.v2/libs/python/build/darwin-4.2.1/debug:/usr/local/boost_1_51_0/libs/python/example/quickstart/bin/darwin-4.2.1/debug:$DYLD_LIBRARY_PATH"
export DYLD_LIBRARY_PATH

    PYTHONPATH="bin/darwin-4.2.1/debug"
export PYTHONPATH
 "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/bin/python" "test_extending.py"   > "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output" 2>&1
    status=$?
    echo >> "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output"
    echo EXIT STATUS: $status >> "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output"
    if test $status -eq 0 ; then
        cp "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output" "bin/test_ext.test/darwin-4.2.1/debug/test_ext"
    fi
    verbose=1
    if test $status -ne 0 ; then
        verbose=1
    fi
    if test $verbose -eq 1 ; then
        echo ====== BEGIN OUTPUT ======
        cat "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output"
        echo ====== END OUTPUT ======
    fi
    exit $status

The result is that my terminal hangs--I have to close the window and open a new one. It does successfully run the tests, though, and outputs the results.

If I omit the last line (exit $status), it runs fine and doesn't hang. So perhaps this is happening because the shell script is exiting out of the user's bash shell, never returning control to bjam? This doesn't explain why I didn't see the test.output file before, I guess. And I would guess that the exit status is good to know. But this might be the cause of the problem.

[anonymous]

Replying to jamestwebber@…:

In hindsight, that is clearly a wrong explanation, because Python has already completed by the time the console hangs, and in the bug it is still running. So, uh, never mind...

[jamestwebber@…]

Sweet Fancy Moses, I got it to work!

(I've searched for answers to this for a long time, so I'm going to post my solution in multiple places for the sake of posterity)

I was able to get all Boost Python tests (in /test and in /example/quickstart) by uninstalling the Homebrew-installed readline. This strongly suggests to me that a bug in readline is causing these tests to fail.

While trying to figure this out, I noticed that only the tests using doctest were failing. doctest uses readline internally, and this might be causing an issue. As has been noted in other places, readline seems to have a bug in which it emits an extra character when first imported in python. At first this seemed unrelated but now I think it (or some other readline bug) is the cause of much headaches.

This is rather unfortunate because uninstalling readline breaks the Python REPL (e.g. arrow keys no longer work). So obviously this is not a long-term fix, but it does answer some questions. Weirdly, readline is not necessarily for doctests to function.

So, I don't have a patch or anything, but I think I have tracked down the bug to readline.

They didn't seem all that interested in patching this bug, but maybe this will give them a good reason to do so (or look for the real bug?).

[jamestwebber@…]

One quick update: I don't think the odd readline character is the cause of this problem (which is not terribly surprising, I don't know how it would be). But I do think something about readline is the issue.

[Dave Abrahams]

Thanks for tracking down that readline bug!