./b2 fails with 'import error: length of source and target rule name lists don't match!'

[anonymous]

I attempted to follow the instructions in "Getting Started for Linux", but building with "b2" fails :

$ ./bootstrap.sh install --prefix=/usr Building Boost.Build engine with toolset gcc... tools/build/v2/engine/bin.linuxx86_64/b2 Detecting Python version... 2.7 Detecting Python root... /usr/lib64/python2.7.1 Unicode/ICU support for Boost.Regex?... /usr Backing up existing Boost.Build configuration in project-config.jam.1 Generating Boost.Build configuration in project-config.jam...

Bootstrapping is done. To build, run:

./b2

To adjust configuration, edit 'project-config.jam'. Further information:

• Command line help: ./b2 --help

• Getting started guide: http://www.boost.org/more/getting_started/unix-variants.html

• Boost.Build documentation: http://www.boost.org/boost-build2/doc/html/index.html

$ ./b2 --help /usr/OOO/boost_1_50_0/tools/build/v2/kernel/modules.jam:326: in boost-build import error: length of source and target rule name lists don't match!

source: peek poke record-binding target: modules.peek

/usr/OOO/boost_1_50_0/boost-build.jam:17: in module scope $ ./b2 /usr/OOO/boost_1_50_0/tools/build/v2/kernel/modules.jam:326: in boost-build import error: length of source and target rule name lists don't match!

source: peek poke record-binding target: modules.peek

/usr/OOO/boost_1_50_0/boost-build.jam:17: in module scope

Is this a python problem ? What version of python should I be using ?

What extra python modules do I need ?

Anyone got a GNU make(1) Makefile for Boost to build with Linux GCC toolchain, or one which does not use python ?

[Jason Vas Dias <jason.vas.dias@…>]

Sorry, I ran :

$ ./bootstrap.sh --prefix=/usr Building Boost.Build engine with toolset gcc... tools/build/v2/engine/bin.linuxx86_64/b2 Detecting Python version... 2.7 Detecting Python root... /usr/lib64/python2.7.1 Unicode/ICU support for Boost.Regex?... /usr Backing up existing Boost.Build configuration in project-config.jam.2 Generating Boost.Build configuration in project-config.jam...

Bootstrapping is done. To build, run:

./b2

To adjust configuration, edit 'project-config.jam'. Further information:

• Command line help: ./b2 --help

• Getting started guide: http://www.boost.org/more/getting_started/unix-variants.html

• Boost.Build documentation: http://www.boost.org/boost-build2/doc/html/index.html

$ ./b2 install /usr/OOO/boost_1_50_0/tools/build/v2/kernel/modules.jam:326: in boost-build import error: length of source and target rule name lists don't match!

source: peek poke record-binding target: modules.peek

/usr/OOO/boost_1_50_0/boost-build.jam:17: in module scope

[anonymous]

Replying to viboes:

where the change has taken place ? how to check it and correct it ?

[Jurko Gospodnetic]

Hi.

I remember someone once asking me about something like this - here's a response I gave them in a personal e-mail message (never got any response back nor did I see them post a related question to any of the Boost mailing lists):

---

I have used the following commands to install Boost but i am getting the following error :-

[priyanka@NGS boost_1_50_0]$ ./bootstrap.sh Building Boost.Build engine with toolset gcc... tools/build/v2/engine/bin.linuxx86_64/b2 Detecting Python version... 2.7 Detecting Python root... /usr ...

[priyanka@NGS boost_1_50_0]$ ./bjam --prefix= /~/Downloads link=static runtime-link=static stage install /home/priyanka/Downloads/boost_1_50_0/tools/build/v2/kernel/modules.jam:326: in boost-build import error: length of source and target rule name lists don't match!

source: peek poke record-binding target: modules.peek

/home/priyanka/Downloads/boost_1_50_0/boost-build.jam:17: in module scope

And I really do not know how this happened at your site. I checked

the source code lines listed in your error report and in modules.jam there is the following:

local globalize = peek poke record-binding ; IMPORT modules : $(globalize) : : modules.$(globalize) ;

which expands to:

IMPORT modules : peek poke record-binding : : modules.peek modules.poke modules.record-binding ;

In your case something seems to cause this to expand to:

IMPORT modules : peek poke record-binding : : modules.peek ;

Try adding some ECHO statements there like:

ECHO /$(globalize)/ ; ECHO /modules.$(globalize)/ ;

and see what you get.

---

Hopefully someone that can reproduce this issue can report back here (or even better - on the Boost Build mailing list) with some more information, otherwise, I do not see what to do about this.

Some ideas:

• What did the ECHO statements suggested above display?
• Does the same occur with Boost library release 1.51?
• Does the same occur when using the current Boost Build version from the trunk?
• Are you sure you are using the correct Boost Jam executable?

[rasmussen74@…]

Unfortunately I am able to reproduce this error and my build is therefore broken. Any help would be appreciated.

Boost library version 1.51

After unpacking the sources tarball I do:

./bootstrap.sh

which completes succesfully. But on the next command:

./b2 tools/bcp

I get

/home/krr/build/external/boost/src/external_boost/tools/build/v2/kernel/modules.jam:326: in boost-build import error: length of source and target rule name lists don't match!

source: peek poke record-binding target: modules.peek

/home/krr/build/external/boost/src/external_boost/boost-build.jam:17: in module scope

[Vladimir Prus]

What does line 326 in modules.jam say for you?

[anonymous]

It says

IMPORT modules : $(globalize) : : modules.$(globalize) ;

BTW I noticed that in bootstrap.log I have two warnings:

###
### Using 'gcc' toolset.
###
rm -rf bootstrap
mkdir bootstrap
gcc -o bootstrap/jam0 command.c compile.c constants.c debug.c function.c glob.c hash.c hdrmacro.c headers.c jam.c jambase.c jamgram.c lists.c make.c make1.c object.c option.c output.c parse.c pathunix.c regexp.c rules.c scan.c search.c subst.c timestamp.c variable.c modules.c strings.c filesys.c builtins.c pwd.c class.c native.c md5.c w32_getreg.c modules/set.c modules/path.c modules/regex.c modules/property-set.c modules/sequence.c modules/order.c execunix.c fileunix.c
jam.c: In function 'executable_path':
jam.c:653:12: warning: incompatible implicit declaration of built-in function 'strndup' [enabled by default]
./bootstrap/jam0 -f build.jam --toolset=gcc --toolset-root= clean
...found 1 target...
...updating 1 target...
[DELETE] clean
...updated 1 target...
./bootstrap/jam0 -f build.jam --toolset=gcc --toolset-root=
...found 50 targets...
...updating 2 targets...
[COMPILE] bin.linuxx86_64/b2
jam.c: In function 'executable_path':
jam.c:653:12: warning: incompatible implicit declaration of built-in function 'strndup' [enabled by default]
[COPY] bin.linuxx86_64/bjam
...updated 2 targets...

which are not present on another system where Boost build works fine. Or is this just a red herring?

[Jurko Gospodnetic]

rasmussen74 - Could you contact me over Skype chat, my login is jurkog? Then I can help you debug this a bit faster...

As for the bjam build compiler warning - do you know which header contains the strndup() function declaration on your system?

As for the original problem - could you send us lines 325-326 in your modules.jam file. They should read:

local globalize = peek poke record-binding ;
IMPORT modules : $(globalize) : : modules.$(globalize) ;

If that is so then please change those two lines to the following:

local globalize = peek poke record-binding ;
ECHO xxx1: /$(globalize)/ ;
ECHO xxx2: /modules.$(globalize)/ ;
IMPORT modules : $(globalize) : : modules.$(globalize) ;

and let us know what those 'xxx1:' and 'xxx2:' lines contain in your build output.

Best regards,

Jurko Gospodnetić

[anonymous]

This warning might be the real problem. Can you try r80097 ?

[Kim Rasmussen <rasmussen74@…>]

Replying to jurko:

rasmussen74 - Could you contact me over Skype chat, my login is jurkog? Then I can help you debug this a bit faster...

Sorry, Skype is no-go from here.

As for the bjam build compiler warning - do you know which header contains the strndup() function declaration on your system?

Will check.

As for the original problem - could you send us lines 325-326 in your modules.jam file.

The lines and surroundings are:

320 }
321 
322 
323 # These rules need to be available in all modules to implement module loading
324 # itself and other fundamental operations.
325 local globalize = peek poke record-binding ;
326 IMPORT modules : $(globalize) : : modules.$(globalize) ;                                                                                                                                  
327 
328 
329 rule __test__ ( )
330 {

They should read:

local globalize = peek poke record-binding ;
IMPORT modules : $(globalize) : : modules.$(globalize) ;

If that is so then please change those two lines to the following:

local globalize = peek poke record-binding ;
ECHO xxx1: /$(globalize)/ ;
ECHO xxx2: /modules.$(globalize)/ ;
IMPORT modules : $(globalize) : : modules.$(globalize) ;

and let us know what those 'xxx1:' and 'xxx2:' lines contain in your build output.

Output is:

xxx1: /peek/
xxx2: /modules.peek/

-Kim

[Jurko Gospodnetic]

Ok, I see you already sent the ECHO results to the Boost Build mailing list - I'll continue responding there...

As for the warning - that should not have anything to do with mem.h changed in r80097. It is complaining about not being able to find a declaration for strndup() - which is a bad thing but should be fixable by a simple header include.

Any other instant messenger? Yahoo/ICQ/IRC/Google/MSN?

Best regards,

Jurko Gospodnetić

[Kim Rasmussen <rasmussen74@…>]

Replying to jurko:

Ok, I see you already sent the ECHO results to the Boost Build mailing list - I'll continue responding there...

OK.

As for the warning - that should not have anything to do with mem.h changed in r80097. It is complaining about not being able to find a declaration for strndup() - which is a bad thing but should be fixable by a simple header include.

Didn't think that either.

Any other instant messenger? Yahoo/ICQ/IRC/Google/MSN?

Google would be okay (rasmussen74).

[Jurko Gospodnetic]

It would be great if some more people affected by this problem would come back to us so we can compare their environment with Kims.

And here's a status update after doing some interactive debugging with Kim.

The problem seems to have been introduced by some of the variable expansion changes made between Boost library 1.49 & 1.50 releases but the effects are *really really wierd*. My guess is that the changed code is somehow triggering a compiler/optimizer bug.

The problematic function in new code is expand(), located in the function.c module. In the original 1.49 Boost library release code-base, that function did not exist and its work seems to have been done by the var_string() function in the variable.c module (and possibly some other function in the later removed expand.c module - did not check).

There we have a 'while ( --i >= 0 )' loop around the middle of that function which never seems to get entered on the target system. Weird thing is that if we add the following code just before that while loop:

    if ( --i >= 0 ) { printf( "%d\n", i ); }
    ++i;

Then everything works fine.

On the other hand, if we leave in the same code but without the printf() call - then we're back to the original problem - most likely due to the optimizer recognizing and removing the dummy code.

We still have not done the following on the target system:

• tried using the debug Boost Jam build
• collected version information (Linux/gcc/glibc)

The 'warning' problem mentioned seems to be caused by the target system's include files (old version perhaps?) not declaring the strndup() function unless the _GNU_SOURCE symbol is defined, but that warning existed with Boost library 1.49 release and fixing it in the current trunk has no effect on the initial variable expansion problem. I'll fix that by rewriting the code to not use strndup() in the first place (will replace it with some malloc()/strncpy() combination).

For more information on the planned expand() function changes - see the Boost Build mailing list.

Hope this helps.

Best regards,

Jurko Gospodnetić

_ Unsubscribe & other changes: ​http://lists.boost.org/mailman/listinfo.cgi/boost-build

[anonymous]

Hi!

I have looked more into the problem and as far as I can see it is clearly related to compiler optimization. Below I have included all relevant information, so please accept my apologies for the lengthy comment.

Target system information:

$ uname -a
Linux munin-4 2.6.16.60-0.69.1_1.0102.5650.2.2.74-ss #1 SMP Tue Dec 21
17:20:07 CST 2010 x86_64 x86_64 x86_64 GNU/Linux

$ g++ -v
Using built-in specs.
COLLECT_GCC=/opt/gcc/4.6.0/bin/../snos/bin/g++
COLLECT_LTO_WRAPPER=/opt/gcc/4.6.0/snos/libexec/gcc/x86_64-suse-linux/4.6.0/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../xt-gcc-4.6.0/configure
--prefix=/opt/gcc/4.6.0/snos --disable-nls
--libdir=/opt/gcc/4.6.0/snos/lib --enable-languages=c,c++,fortran
--with-gxx-include-dir=/opt/gcc/4.6.0/snos/include/g++
--with-slibdir=/opt/gcc/4.6.0/snos/lib --with-system-zlib
--enable-shared --enable-__cxa_atexit x86_64-suse-linux
--with-mpc=/opt/gcc/mpc/0.8.1 --with-mpfr=/opt/gcc/mpfr/2.4.2
--with-gmp=/opt/gcc/gmp/4.3.2 --with-sysroot=
Thread model: posix
gcc version 4.6.0 20110325 (Cray Inc.) (GCC)

As suggested by Jurko earlier today when we did a bit of debugging, the following was tried on the trunk of Boost build. All commands below are executed within the v2/engine directory.

Now, the culprit appears to be the expand() function in function.c. For testing we use a file, let's call it jamtest.jam, containing only these lines:

local a = one two three ;
ECHO /$(a)/ ;

Then the following commands:

$ ./build.sh -d3 --release > /tmp/build-d3-release
$ ./bin.linuxx86_64/bjam -f jamtest.jam

gives

/one/
don't know how to make all
...found 1 target...
...can't find 1 target...

but

$ ./build.sh -d3 --debug > /tmp/build-d3-debug
$ ./bin.linuxx86_64.debug/bjam -f jamtest.jam

gives

/one/ /two/ /three/
don't know how to make all
...found 1 target...
...can't find 1 target...

which is what we expected -- so clearly, the 'release' version is faulty. (BTW, the build-d3-release and build-d3-debug files can be found in the pastebins pastebin.com/M2njiNEK and pastebin.com/038qcgFF respectively).

Now, if we insert the following lines just before the line "while ( --i >= 0)"

   if (--i >= 0) {
      side_effect = i;
   }
   ++i;

and add

   static int side_effect;

just before the function declaration, we then get:

$ ./build.sh -d3 --release > /tmp/build-d3-release
$ ./bin.linuxx86_64/bjam -f jamtest.jam
/one/ /two/ /three/
don't know how to make all
...found 1 target...
...can't find 1 target...

and

$ ./build.sh -d3 --debug > /tmp/build-d3-debug
$ ./bin.linuxx86_64.debug/bjam -f jamtest.jam
/one/ /two/ /three/
don't know how to make all
...found 1 target...
...can't find 1 target...

that is, correct in both release and debug modes.

To me, it smells compiler bug, but as this is a system where I cannot "just update" the compiler, I am wondering if there are other solutions. I have a feeling that it is the loop/while/goto construct that's complicating matters.

BTW, the problem is not present with Boost 1.49, but both in 1.51 and trunk. Of course, function.c is not present in 1.49, so that figures.

I'd really like to get on 1.51, so any suggestions on how to proceed are most welcome.

Best regards

[anonymous]

BTW, could somebody else with a GCC 4.6.0 try to reproduce?

[Jurko Gospodnetic]

We've confirmed this as a goto related compiler bug. :-)

I'm attaching a patched function.c module based on revision [80313] that fixes the bug on the target system. I'll not commit it until we can find out more precisely which systems are affected by the bug.

Best regards,

Jurko Gospodnetić

[Jurko Gospodnetic]

Modified function.c Boost Jam source file based on revision [80313] that fixes the issue on one system.

[Jason Vas Dias <jason.vas.dias@…>]

Sorry I haven't been much help in resolving the bug I raised, since I needed a boost build in a hurry I reran bootstrap.sh in my own Linux root but using the Python from a Fedora 14 chroot ; then magically the b2 commands started to work , and I was able to complete the build back in my own distro with gcc-4.6.0 . But I think the whole Boost build process should not depend on Python ! I would rather program with modern C++-11 and Boost than Python - why not remove the python dependencies and replace with C++ ? I just wrote bash shell brace-expansion and wildcard-matching code in C++-11 (only!) and found it a refreshing and rewarding experience - beside it the python equivalent looks kludgey, quirky and over-verbose - what do you expect from a language whose syntax changes with every major release ? Why not use one with a well-designed syntax ? Or just use bash / sh ?

[Jason Vas Dias <jason.vas.dias@…>]

Anyway, I'll try with 1.51 now - at least this issue seems to have exposed a genuine problem .

[Jurko Gospodnetic]

Replying to Jason Vas Dias <jason.vas.dias@…>:

Sorry I haven't been much help in resolving the bug I raised, since I needed a boost build in a hurry I reran bootstrap.sh in my own Linux root but using the Python from a Fedora 14 chroot ; then magically the b2 commands started to work , and I was able to complete the build back in my own distro with gcc-4.6.0 . But I think the whole Boost build process should not depend on Python !

I am not really sure what happened on your system. I can correctly build Boost on my Win7 system without having Python on the path at all (without building up Boost Python, ofcourse) so I do not really know what Python dependency you are talking about.

I would rather program with modern C++-11 and Boost than Python - why not remove the python dependencies and replace with C++ ?

The only Python dependency I am aware of is in the Boost Python library and some internal testing scripts. Could you help find any others?

I just wrote bash shell brace-expansion and wildcard-matching code in C++-11 (only!) and found it a refreshing and rewarding experience - beside it the python equivalent looks kludgey, quirky and over-verbose - what do you expect from a language whose syntax changes with every major release ? Why not use one with a well-designed syntax ? Or just use bash / sh ?

Actually, I'm quite fond of Python. :-) But, as I said before, I'm not actually sure what Python dependency you're referring to here...

However - could you try out the corrected source file I attached to this ticked and see if it fixes the issue? Just checkout Boost Build revision 80313, patch it with the attached file, build Boost Jam from there (run the 'tools/build/v2/engine/build.sh gcc' script) and then use the built Boost Jam executable to install the Boost libraries.

Best regards,

Jurko Gospodnetić

[Jason Vas Dias <jason.vas.dias@…>]

Replying to Jason Vas Dias <jason.vas.dias@…>:

Anyway, I'll try with 1.51 now - at least this issue seems to have exposed a genuine problem .

And after building gcc-4.7.1 and compiling with it ... Thanks!

[Jurko Gospodnetic]

Replying to Jason Vas Dias <jason.vas.dias@…>:

Anyway, I'll try with 1.51 now - at least this issue seems to have exposed a genuine problem .

And after building gcc-4.7.1 and compiling with it ... Thanks!

I'm not sure what you're referring to here - again. :-)

You mean - your problem went away after you started using gcc-4.7.1 but with gcc-4.6.0 it's failing?

[Kim Rasmussen <rasmussen74@…>]

Update:

I have tried compiling with gcc 4.6.1 and gcc 4.6.2 and the small case described earlier in this thread successfully compiles with both.

Subsequently I have successfully compiled the entire Boost 1.51 and the dependent project using gcc 4.6.2 on the target system, so that's solves it for me for now.

However, I think that it is clear that gcc 4.6.0 is broken for the loop/while/goto construct when using -Os, -O2 or -O3. It might be an idea to replace it with a while/while/break construct or flag 4.6.0 as broken.

Sidenote: On my desktop I have absolutely no problem using gcc 4.7.0 and Boost 1.51.

[Jurko Gospodnetic]

Fixed by revision [80319].