Boost C++ Libraries: Ticket #2185: bjam hangs when building libs/python/examples/quickstart under MacOS X

status, component, milestone changed; resolution set

Marshall Clow — Tue, 09 Sep 2008 19:51:07 GMT

status new → closed
resolution → worksforme
component None → Building Boost
milestone Boost 1.36.0 → Boost 1.37.0

Just tried it with the current trunk and gcc 4.0.1 (Xcode 3.1 install) and $ gcc --version i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5484) $ bjam --version Boost.Build V2 (Milestone 12) Boost.Jam 03.1.16

No hang - closing as "works for me" Please re-open if this still happens for you using current boost/gcc/bjam.

konrad.hinsen@… — Fri, 19 Sep 2008 19:41:01 GMT

I updated to XCode 2.5 (the latest for MacOS 10.4) and tried with boost 1.36.0. It still hangs, at the same place.

$ gcc --version i686-apple-darwin8-gcc-4.0.1 (GCC) 4.0.1 (Apple Computer, Inc. build 5370)

$ ./tools/jam/src/bin.macosxx86/bjam --version Boost.Build V2 (Milestone 12) Boost.Jam 03.1.16

Note: I tried to change the status to "reopen", but apparently I do not have the necessary permissions. The site just says "Warning: No permission to change ticket fields." I don't see a way to create an account either, so I am blocked. Now I try to add my comment without changing the status as this still seems better than doing nothing at all...

status changed; resolution deleted

jason@… — Sat, 11 Apr 2009 03:43:35 GMT

status closed → reopened
resolution worksforme

I am having the same problem. Building the example "quickstart" for Boost.Python hangs on the line described above. Versions: OS X - 10.5.6 gcc - i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5490) Boost 1.38.0 Boost.Jam 03.1.17

I've also tried using bjam -n -a to get at the commands being run and to run them manually, as outlined here: http://mail.python.org/pipermail/cplusplus-sig/2008-August/013580.html

I managed to run the command that hangs under bjam, successfully. But in the next step I ran into "darwin.link.dll: command not found" and that (alias?) didn't seem to be explicitly defined in the commands I saw, so I set this aside for the moment.

component changed

Steven Watanabe — Sun, 19 Apr 2009 02:56:21 GMT

component Building Boost → Python

darwin.link.dll is a Boost.Build rule, not a command that you can run from a shell.

Dave Abrahams — Thu, 04 Jun 2009 21:43:49 GMT

This is happening for me, too. I'm not sure where to start in trying to fix it, though!

status changed; owner set

Dave Abrahams — Thu, 04 Jun 2009 22:11:45 GMT

owner set to René Rivera
status reopened → new

Another strange fact: adding -d+2 doesn't cause the command to be printed out before Python is started. I even added an explicit fflush to the bjam sources to try to force it. If I do a "ps waux" I can clearly see that it has started running the Python command, though!

anonymous — Thu, 04 Jun 2009 23:42:35 GMT

Replying to dave:

Another strange fact: adding -d+2 doesn't cause the command to be printed out before Python is started. I even added an explicit fflush to the bjam sources to try to force it. If I do a "ps waux" I can clearly see that it has started running the Python command, though!

This was part of [38009]. It's annoying and should probably be disabled for single threaded builds.

anonymous — Fri, 05 Jun 2009 09:29:35 GMT

could somebody run bjam with '-l5' (undocumented) option and see if it helps?

attachment set

anonymous — Fri, 05 Jun 2009 14:50:25 GMT

attachment → execunix.c.patch

updated

Dave Abrahams — Fri, 05 Jun 2009 18:43:56 GMT

With -l5, (that's a lowercase "L", right?) I get:

#!/bin/sh
$ bjam -l5
...patience...
...patience...
...found 3236 targets...
...updating 2 targets...
capture-output ../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test
5 second time limit exceeded
    DYLD_LIBRARY_PATH=/System/Library/Frameworks/Python.framework/Versions/2.5/lib:/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config:/Users/dave/src/boost/bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug:/Users/dave/src/boost/bin.v2/libs/python/build/darwin-4.0.1/debug:$DYLD_LIBRARY_PATH
export DYLD_LIBRARY_PATH
    PYTHONPATH=../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug
export PYTHONPATH
 "/usr/bin/python2.5" "python_test.py"   > "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output" 2>&1
    status=$?
    echo >> "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output"
    echo EXIT STATUS: $status >> "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output"
    if test $status -eq 0 ; then
        cp "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output" "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test"
    fi
    verbose=0
    if test $status -ne 0 ; then
        verbose=1
    fi
    if test $verbose -eq 1 ; then
        echo ====== BEGIN OUTPUT ======
        cat "../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test.output"
        echo ====== END OUTPUT ======
    fi
    exit $status
...failed capture-output ../../../bin.v2/libs/parameter/test/python_test.test/darwin-4.0.1/debug/python_test...
...failed updating 1 target...
...skipped 1 target...
zreba:test dave$

I did this one in /libs/parameter/test because it has only one Python-related test. I don't know if you're suggesting it, but I'll try the patch next.

Dave Abrahams — Fri, 05 Jun 2009 18:51:15 GMT

Applying the patch makes no apparent difference to my results

severity changed

Dave Abrahams — Fri, 05 Jun 2009 20:16:20 GMT

severity Problem → Showstopper

I'm able to run these tests with CMake, so this is not a blocker for me, but it surely is for someone else!

attachment set

ilyasokol@… — Sun, 07 Jun 2009 10:31:07 GMT

attachment → execunix.c.2.patch

removes o_nonblock from both ends of pipe

ilyasokol@… — Sun, 07 Jun 2009 10:37:00 GMT

If the patch above doesn't work then I am out of ideas. I hope we are hitting this bug: http://lists.apple.com/archives/Darwin-kernel/2005/Dec/msg00058.html

Dave Abrahams — Sun, 07 Jun 2009 18:44:58 GMT

Makes no difference for me, I'm afraid. Note, though, that the 2nd hunk of your patch applied at an offset of 4 lines, which makes me think you're working with a different version of the source.

If you need it I can hook you up with a MacOS account to try this on.

anonymous — Sun, 07 Jun 2009 19:46:10 GMT

Replying to dave:

If you need it I can hook you up with a MacOS account to try this on.

That would be great. Please contact me by email ilyasokol<at>gmail<dot>com

Wed, 08 Jul 2009 21:20:24 GMT

Still seeing this with Boost 1.39.0 on Mac OS X 10.5.7. I will try adding the timeout described in http://article.gmane.org/gmane.comp.lib.boost.build/18302. To be continued...

severity changed

Thu, 09 Jul 2009 20:12:40 GMT

severity Showstopper → Problem

The hang actually occurs in test_extending.py. I am starting to doubt this is a bjam problem.

Thu, 09 Jul 2009 20:24:04 GMT

Situation is:

Configure dynamic libraries and PYTHONPATH from command line. Run "python test_extending.py". Test succeeds.

Run tests from bjam, with command "bjam test_ext". Test hangs.

Run tests from bjam, with timeout; command "bjam -l15 test_ext". Test fails by timeout.

It is probably some subtle environment-related bug, either in the Apple release of python 2.5.1 or in Boost.Python itself, but so far I am unable to determine which.

anonymous — Tue, 11 Aug 2009 23:48:30 GMT

I know it's been bumped down, but can this be bumped back up to showstopper? I't seem like one -- I can find no consistent workaround, and it's made boost.python unavailable to what appears to be a large segment of population for at least a year now. (I've spoken to two other people who have abandoned boost because of this bug.)

severity changed

anonymous — Wed, 12 Aug 2009 17:00:33 GMT

severity Problem → Showstopper

anonymus — Thu, 26 Nov 2009 13:08:47 GMT

apparently, somebody got it running here http://lists.boost.org/boost-users/2009/02/45404.php

go to later posts in this thread.

Marshall Clow — Mon, 30 Nov 2009 01:14:55 GMT

Additional data point: I just tried this on Mac OS X 10.6.2 / gcc 4.2.1 (Intel), and was unable to reproduce.

anonymous — Sat, 15 May 2010 18:34:40 GMT

FWIW, 'bjam toolset=darwin test' still hangs for me. However, 'extending' and 'embedding' both compile without problems, and I can import the generated extension module (after moving 'libboost_python.dylib' into a library path.) It sounds like 'test' works on Snow Leopard, but not on Leopard. Given that boost.python obviously works on Leopard, this isn't a showstopper for me, but it would have been really great if there had been a note somewhere, because I all but abandoned boost.python for over a year because I thought it simply didn't work on OS X (and others have said the same).

OS X 10.5.8

Developer Tools 3.1.4

gcc version 4.0.1 (Apple Inc. build 5493)

boost_1_43_0

Boost.Jam Version 3.1.17. OS=MACOSX

anonymous — Tue, 19 Oct 2010 10:00:08 GMT

hanging at:

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

OS X 10.6.4

darwin-4.2.1

boost_1_44_0

Boost.Jam 03.1.18

anonymous — Tue, 19 Oct 2010 10:00:15 GMT

hanging at:

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

OS X 10.6.4

darwin-4.2.1

boost_1_44_0

Boost.Jam 03.1.18

meowsqueak@… — Mon, 09 Jul 2012 22:36:59 GMT

I have this problem right now with:

$ bjam toolset=darwin --verbose-test test

It hangs after the line:

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

Ctrl-c and inspecting the bin/ directory shows some .o and .so files, but no .dylib files, in bin/darwin-4.2.1/debug/ but these don't seem to be 'importable' by Python.

This command (not sure if that's the right PYTHONPATH?) fails:

$ PYTHONPATH=bin/darwin-4.2.1/debug python test_extending.py ... ImportError: dlopen(/Users/david/Downloads/boost_1_50_0/libs/python/example/quickstart/bin/darwin-4.2.1/debug/extending.so, 2): Library not loaded: libboost_python.dylib

Not sure why bjam created a .so instead of a .dylib?

My platform: *OS X 10.6.8 64bit *Python 2.7.3 (built locally with pythonbrew) *i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3) *boost_1_50_0 *Boost.Build 2011.12-svn (bjam built locally from boost_1_50_0)

meowsqueak@… — Mon, 09 Jul 2012 22:54:45 GMT

Replying to meowsqueak@…:

This command (not sure if that's the right PYTHONPATH?) fails:

$ PYTHONPATH=bin/darwin-4.2.1/debug python test_extending.py ... ImportError: dlopen(/Users/david/Downloads/boost_1_50_0/libs/python/example/quickstart/bin/darwin-4.2.1/debug/extending.so, 2): Library not loaded: libboost_python.dylib

Just as a follow up, after hitting Ctrl-C at the hang, I was able to successfully run 'test_extending.py' with:

 $ export DYLD_LIBRARY_PATH=../../../../bin.v2/libs/python/build/darwin-4.2.1/debug
 $ export LD_LIBRARY_PATH=../../../../bin.v2/libs/python/build/darwin-4.2.1/debug
 $ export PYTHONPATH=bin/darwin-4.2.1/debug
 $ python test_extending.py
 ...
  9 passed and 0 failed.
 Test passed.

jamestwebber@… — Fri, 28 Sep 2012 00:46:18 GMT

I'm having the exact problem that meowsqueak reports: bjam hanged after

darwin.link.dll bin/darwin-4.2.1/debug/extending.so

But I can run test_extending.py fine after killing the command and export the relevant variables. No new wrinkles from what's been previously reported, but wanted to add another data point.

I'm using OS X 10.7.5 and python 2.7.3 built with homebrew, boost 1.50.0 and bjam built locally from boost.

Digging a bit into the commands, I can see as dave did that Python is running, with the command

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/bin/python" "test_extending.py"   > "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output" 2>&1

But when I run that command alone (after compiling) it runs fine. The test_ext.output looks fine except that it starts with the characters \E[?1034h which appears to be related to some weird behavior from readline (CAPTCHA isn't working so I can't post this link, but: search for "readline+1034h")? I have no idea if that might cause a problem--I'm not sure how it would.

If I kill the Python process it remains hanging, but killing the bash script stops the test (which then fails because the output is wrong).

Actually…

I tried running the testing script manually, by pasting this into the terminal:

/bin/sh -c
    DYLD_LIBRARY_PATH="/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib:/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config:/usr/local/boost_1_51_0/bin.v2/libs/python/build/darwin-4.2.1/debug:/usr/local/boost_1_51_0/libs/python/example/quickstart/bin/darwin-4.2.1/debug:$DYLD_LIBRARY_PATH"
export DYLD_LIBRARY_PATH
    PYTHONPATH="bin/darwin-4.2.1/debug"
export PYTHONPATH
 "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/bin/python" "test_extending.py"   > "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output" 2>&1
    status=$?
    echo >> "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output"
    echo EXIT STATUS: $status >> "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output"
    if test $status -eq 0 ; then
        cp "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output" "bin/test_ext.test/darwin-4.2.1/debug/test_ext"
    fi
    verbose=1
    if test $status -ne 0 ; then
        verbose=1
    fi
    if test $verbose -eq 1 ; then
        echo ====== BEGIN OUTPUT ======
        cat "bin/test_ext.test/darwin-4.2.1/debug/test_ext.output"
        echo ====== END OUTPUT ======
    fi
    exit $status

The result is that my terminal hangs--I have to close the window and open a new one. It does successfully run the tests, though, and outputs the results.

If I omit the last line (exit $status), it runs fine and doesn't hang. So perhaps this is happening because the shell script is exiting out of the user's bash shell, never returning control to bjam? This doesn't explain why I didn't see the test.output file before, I guess. And I would guess that the exit status is good to know. But this might be the cause of the problem.

anonymous — Fri, 28 Sep 2012 00:56:35 GMT

Replying to jamestwebber@…:

In hindsight, that is clearly a wrong explanation, because Python has already completed by the time the console hangs, and in the bug it is still running. So, uh, never mind...

jamestwebber@… — Tue, 19 Feb 2013 23:16:08 GMT

Sweet Fancy Moses, I got it to work!

(I've searched for answers to this for a long time, so I'm going to post my solution in multiple places for the sake of posterity)

I was able to get all Boost Python tests (in /test and in /example/quickstart) by uninstalling the Homebrew-installed readline. This strongly suggests to me that a bug in readline is causing these tests to fail.

While trying to figure this out, I noticed that only the tests using doctest were failing. doctest uses readline internally, and this might be causing an issue. As has been noted in other places, readline seems to have a bug in which it emits an extra character when first imported in python. At first this seemed unrelated but now I think it (or some other readline bug) is the cause of much headaches.

This is rather unfortunate because uninstalling readline breaks the Python REPL (e.g. arrow keys no longer work). So obviously this is not a long-term fix, but it does answer some questions. Weirdly, readline is not necessarily for doctests to function.

So, I don't have a patch or anything, but I think I have tracked down the bug to readline.

They didn't seem all that interested in patching this bug, but maybe this will give them a good reason to do so (or look for the real bug?).

jamestwebber@… — Wed, 20 Feb 2013 00:29:11 GMT

One quick update: I don't think the odd readline character is the cause of this problem (which is not terribly surprising, I don't know how it would be). But I do think something about readline is the issue.

status changed; resolution set

Dave Abrahams — Wed, 20 Feb 2013 01:10:03 GMT

status new → closed
resolution → invalid

Thanks for tracking down that readline bug!