Boost C++ Libraries: Ticket #4864: Support for 64-bit ICU

Dave Abrahams — Wed, 17 Nov 2010 17:24:45 GMT

Note that even after accounting for that issue, ICU is still not successfully detected under x64:

msvc.link c:\work\build\boost\bin.v2\libs\regex\build\msvc-10.0express\debug\threading-multi\has_icu.exe
has_icu_test.obj : error LNK2019: unresolved external symbol _u_charFromName_44 referenced in function _main
c:\work\build\boost\bin.v2\libs\regex\build\msvc-10.0express\debug\threading-multi\has_icu.exe : fatal error LNK1120: 1 unresolved externals
        call "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" >nul
call "c:/Program Files/Microsoft SDKs/Windows/v7.1/Bin/SetEnv" /x86 >nul
link /NOLOGO /INCREMENTAL:NO /DEBUG /MACHINE:X86 /subsystem:console /out:"c:\work\build\boost\bin.v2\libs\regex\build\msvc-10.0express\debug\threading-multi\has_icu.exe"  /LIBPATH:c:/work/icu/lib64 @"c:\work\build\boost\bin.v2\libs\regex\build\msvc-10.0express\debug\threading-multi\has_icu.exe.rsp"
        if %ERRORLEVEL% NEQ 0 EXIT %ERRORLEVEL%

anonymous — Thu, 18 Nov 2010 17:06:40 GMT

Dave, can you tell me what's in has_icu.exe.rsp?

Also the bjam command line.

Thanks, John.

Dave Abrahams — Thu, 18 Nov 2010 17:17:33 GMT

sorry, John... been up for 3 days trying to get all this building/working and I didn't take the time to create a little test case. The command-line was undoubtedly just something like bjam address-model=64 in libs/regex/test. However, I don't have the other file. I could try to recreate this if you ping me in a couple of days.

anonymous — Thu, 18 Nov 2010 18:46:48 GMT

There's a tentative fix for this in Trunk, if you can let me know if this works once you catch breath, that would be great.

cc set

Katie Chan — Sat, 20 Nov 2010 05:01:26 GMT

cc Katie Chan added

Nope, it completely broke things now. :D

Before, x64 actually worked if one renamed the directory from lib64 to lib. I had

D:\libs\icu4c-4_4_2\        // x86
D:\libs\icu4c-4_4_2-x64\    // x64

where both contained a \bin & \lib. i.e. I renamed the bin64 & lib64 and everything worked.

The new Jamfile.v2 causes ICU not to be detected at all, even for x86.

msvc.link bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu.exe
LINK : fatal error LNK1181: cannot open input file 'icuucd.lib'
        call "D:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86 >nul
link /NOLOGO /INCREMENTAL:NO /DEBUG /MACHINE:X86 /subsystem:console /out:"bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu.exe"  /delayload:icudt44.dll /delayload:icuin44.dll /delayload:icuin44d.dll /delayload:icuuc44.dll /delayload:icuuc44d.dll delayimp.lib @"bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu.exe.rsp"
        if %ERRORLEVEL% NEQ 0 EXIT %ERRORLEVEL%
...failed msvc.link bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu.exe bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu.pdb...
...failed updating 1 target...
...updated 8 targets...
...failed updating 1 target...
Performing configuration checks
    - has_icu builds           : no

John Maddock — Sat, 20 Nov 2010 10:04:55 GMT

The new Jamfile.v2 causes ICU not to be detected at all, even for x86.

Dang, it works for me here on Win32 (Vista)... two questions, what's the bjam invocation, and what's the content of bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu.exe.rsp?

Thanks, John.

Katie Chan — Sat, 20 Nov 2010 23:53:15 GMT

Boost trunk r66651.

%ICU_PATH%=D:/libs/icu4c-4_4_2
%PATH%=....;%ICU_PATH%/bin

what's the bjam invocation

bjam --build-type=complete --with-regex stage

what's the content of bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu.exe.rsp?

"bin.v2\libs\regex\build\msvc-10.0\debug\threading-multi\has_icu_test.obj"
"icuucd.lib"
"icudt.lib"
"icuind.lib"

John Maddock — Sun, 21 Nov 2010 18:41:03 GMT

(In [66659]) Another attempt at fixing 64-bit ICU support. dll-path for 64-bit builds still isn't set correctly. Refs #4864.

anonymous — Sun, 21 Nov 2010 18:43:00 GMT

Can you please try again?

There's still an issue of dll-path not being set correctly for 64-bit builds, but otherwise, I *hope* it will now work. Unfortunately I can't test 64-bit support locally though...

Thanks, John.

Katie Chan — Sun, 21 Nov 2010 20:02:46 GMT

Well, the 32-bits detect ICU and build fine reporting it finished building all targets, however 8 of the 18 resulting files that are built for 1.45 wasn't built.

boost_regex-vc100-mt-gd.lib
boost-regex-vc100-mt.lib
libboost_regex-vc100-mt-gd.lib
libboost_regex-vc100-mt-s.lib
libboost_regex-vc100-mt-sgd.lib
libboost_regex-vc100-mt.lib
libboost_regex-vc100-s.lib
libboost_regex-vc100-sgd.lib

are missing.

Win64 build didn't work at all. With the build command

bjam --build-type=complete --with-regex -j2 architecture=x86 address-model=64 stage

error: No best alternative for libs/regex/build/icuuc
    next alternative: required properties: <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <address-model>64 <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <link>shared <runtime-link>shared <toolset>msvc <variant>debug
        matched
error: No best alternative for libs/regex/build/icudt
    next alternative: required properties: <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <address-model>64 <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <link>shared <runtime-link>shared <toolset>msvc
        matched
error: No best alternative for libs/regex/build/icuin
    next alternative: required properties: <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <address-model>64 <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <link>shared <runtime-link>shared <toolset>msvc <variant>debug
        matched
error: No best alternative for libs/regex/build/icudt
    next alternative: required properties: <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <address-model>64 <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <link>shared <runtime-link>shared <toolset>msvc
        matched
error: No best alternative for libs/regex/build/icuin
    next alternative: required properties: <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <address-model>64 <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <link>shared <runtime-link>shared <toolset>msvc <variant>debug
        matched
error: No best alternative for libs/regex/build/icuuc
    next alternative: required properties: <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <address-model>64 <link>shared <runtime-link>shared
        matched
    next alternative: required properties: <link>shared <runtime-link>shared <toolset>msvc <variant>debug
        matched
...............
has_icu_test.obj : error LNK2019: unresolved external symbol u_charFromName_44 referenced in function main
bin.v2\libs\regex\build\msvc-10.0\debug\address-model-64\architecture-x86\threading-multi\has_icu.exe : fatal error LNK1120: 1 unresolved externals
        call "D:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat" x86_amd64 >nul
link /NOLOGO /INCREMENTAL:NO /DEBUG /MACHINE:X64 /subsystem:console /out:"bin.v2\libs\regex\build\msvc-10.0\debug\address-model-64\architecture-x86\threading-multi\has_icu.exe"  /delayload:icudt44.dll /delayload:icuin44.dll /delayload:icuin44d.dll /delayload:icuuc44.dll /delayload:icuuc44d.dll delayimp.lib @"bin.v2\libs\regex\build\msvc-10.0\debug\address-model-64\architecture-x86\threading-multi\has_icu.exe.rsp"
        if %ERRORLEVEL% NEQ 0 EXIT %ERRORLEVEL%
...failed msvc.link bin.v2\libs\regex\build\msvc-10.0\debug\address-model-64\architecture-x86\threading-multi\has_icu.exe bin.v2\libs\regex\build\msvc-10.0\debug\address-model-64\architecture-x86\threading-multi\has_icu.pdb...
...removing bin.v2\libs\regex\build\msvc-10.0\debug\address-model-64\architecture-x86\threading-multi\has_icu.pdb
...failed updating 1 target...
...updated 10 targets...
...failed updating 1 target...

anonymous — Mon, 22 Nov 2010 09:33:26 GMT

Aaaah.... in cyberspace no one can hear you scream.... :-0

I had this working - or at least the bjam bit worked - and then I tidied up the Jamfile formatting before committing - and now I just can't get it working again :(

Re the missing files - this is an inevitable consequence of using ICU - it has to link against the dynamic runtime, so the static runtime variants aren't built.

Time to ask for help...

John Maddock — Tue, 23 Nov 2010 13:09:11 GMT

(In [66699]) Yes another attempt to fix the 64-bit paths issue. Refs #4864.

anonymous — Tue, 23 Nov 2010 14:57:13 GMT

Dave, Yet another attempt to fix this has gone into Trunk... if you haven't lost patience yet, can you give this one a try too?

Cheers, John.

Katie Chan — Tue, 23 Nov 2010 17:54:21 GMT

Um, I'm not Dave (*point to by line*) but sure. :D

This seems to work. Build without errors for both x86 & x64. All regression tests passed without errors for both as well. :)

KTC

Dave Abrahams — Sun, 19 Dec 2010 23:20:22 GMT

Replying to anonymous:

Dave, Yet another attempt to fix this has gone into Trunk... if you haven't lost patience yet, can you give this one a try too?

John, I'll check this out the next time I build an installer, which should be RSN™

Chris Newbold — Wed, 05 Jan 2011 20:31:30 GMT

We've had similar struggles trying to build Boost 1.44 with ICU on 64-bit Windows 7. In addition to the changes to boost/libs/regex/build/Jamfile.v2 which we lifted from Trunk, we also found a fundamental problem with the implementation of check-target-builds in Boost.Build.

The problem is that it does not pass properties such as variant, architecture and address-model from the property set of the "main" build into that for the test executable (has_icu in the case of Boost.Regex). This resulted in attempts to build a debug-mode, 32-bit has_icu in our release-mode 64-bit environment--none of which succeeded.

After a lot of hacking around, we finally made the changes in the attached patch to boost/tools/build/v2/build/configure.jam. I'm not sure these are entirely correct, but when combined with the Boost.Regex Jamfile.v2 from Trunk we were able to successfully build with ICU in both 32- and 64-bit Windows environments.

cc changed

Chris Newbold — Wed, 05 Jan 2011 20:32:05 GMT

cc cnewbold@… added

attachment set

Chris Newbold — Wed, 05 Jan 2011 20:35:10 GMT

attachment → check-target-builds-patch

Patch for check-target-builds to pass relevant properties along to the target executable

cc changed

John Maddock — Thu, 06 Jan 2011 11:22:14 GMT

cc john@… added

This seems to be mostly fixed in Trunk anyway - however not all properties are passed down - address-model seems to work OK, but variant debug/release isn't - so I'm adding Vladimir Prus to the CC list.

cc changed

John Maddock — Thu, 06 Jan 2011 11:24:34 GMT

cc ghost@… added; john@… removed

Katie Chan — Tue, 11 Jan 2011 18:54:41 GMT

Shouldn't what's there already which appears to be an improvement on previous release be merged to release for 1.46?

anonymous — Wed, 12 Jan 2011 17:32:52 GMT

Shouldn't what's there already which appears to be an improvement on previous release be merged to release for 1.46?

It has been, sorry forgot to reference the ticket in the commit.

Regards, John.

owner, component changed

John Maddock — Fri, 18 Mar 2011 16:56:09 GMT

owner changed from John Maddock to Vladimir Prus
component regex → build

Reassigning to Boost.Build as this is now solely an issue with the check-target-builds rule.

Please refer to the comment by cnewbold above to understand the issue.

Vladimir Prus — Sat, 19 Mar 2011 09:59:00 GMT

It seems to be that check-target-builds should actually access the list of relevant properties. In some cases, debug vs. release does not actually matter, in particularly on Linux. WDYT?

anonymous — Sat, 19 Mar 2011 12:10:16 GMT

How do you determine what properties are relevant, unless you pass all of them? Not a rhetorical question BTW :-)

Vladimir Prus — Sat, 19 Mar 2011 12:24:36 GMT

The plan is for me not to answer the above non-rhetorical question, by asking the user of check-target-builds to explicitly specify relevant properties if those are different from the default set (which have to documented, but that's separate matter).

anonymous — Sat, 19 Mar 2011 12:48:56 GMT

I don't see how that is a solution - how can the author of the Jamfile know what build properties may be specified on the command line - for example I don't see how this addresses the original issue above - that variant, address-model and architechture options don't get passed to the config check which causes incorrect configuration.

To take another example - what happens if the user is building both 32 and 64 bit binaries, and the configuration for each is different?

Vladimir Prus — Sat, 19 Mar 2011 13:01:37 GMT

First, not that address-model and architecture are now passed.

Second, it's not a random Jamfile author -- it's somebody who implements a configuration check, and while that check is in Jamfile in this specific case, he might as well be in a special module. Given that you're implementing a configuration check, you should know what properties determine its outcome. In case of ICU, you should know whether debug or release variant determines the outcome. The only other alternative is to re-run a check for every possible set of properties ever used anywhere, which might waste time.

anonymous — Tue, 22 Mar 2011 17:15:09 GMT

Well OK, pretty much all the properties may have an impact - certainly include, search, cxxflags, linkflags, also release and debug (on windows because both ICU lib variants may not be built, and only release mode versions are in the binary ICU distro), probably also define on MSVC (because some defines change the std lib ABI), and I'm sure others I haven't thought of yet... And that's really the problem, on some given platform, I really don't know the complete list of - possibly-compiler-specific - options that will have an effect.

For that matter, how on earth do I force a check-target-builds rule to use the same properties as the target that's using it anyway? Remember also that the target may be built more than once with different property sets - and therefore need different configurations.

Cheers, John.

cc changed

dtrebbien@… — Mon, 11 Apr 2011 18:16:42 GMT

cc dtrebbien@… added

I just discovered this ticket. Using Microsoft Visual C++ 10.0, I was able to build Boost.Regex (Boost 1.46.1) with ICU 4.6.1 support by modifying the has_icu.exe.rsp file and re-running the link command.

The contents of the has_icu.exe.rsp file were originally:

"bin.v2\libs\regex\build\msvc-10.0\debug\address-model-64\architecture-x86\threading-multi\has_icu_test.obj"
"icudt.lib"
"icuind.lib"
"icuucd.lib"

Notice the trailing "d" on the last two library names. By removing those two "d"s and re-running the link command, I was able to build has_icu.exe.

anonymous — Tue, 12 Apr 2011 08:38:27 GMT

Notice the trailing "d" on the last two library names. By removing those two "d"s and >re-running the link command, I was able to build has_icu.exe.

This isn't really a 64-bit issue - it's a debug build vs release build issue.

The ICU dll's built against the release MSVC runtime, don't have the "d" suffix, the debug ones do (but you have to build them yourself from the project files). Note that while you can link the release ICU versions to a debug build, doing so will cause two different runtime libraries to be in use, and that almost always results in a crash or heap curruption at some point :(

To put it another way - the build script is correct - when building debug regex libraries it looks for the debug ICU dll's because those are the only ones that are guaranteed to work correctly. It's just that ICU doesn't supply those in it's binary release .zips, so you have to build them yourself.

HTH, John.

dtrebbien@… — Tue, 12 Apr 2011 22:22:40 GMT

This isn't really a 64-bit issue - it's a debug build vs release build issue.

The ICU dll's built against the release MSVC runtime, don't have the "d" suffix, the debug ones do (but you have to build them yourself from the project files). Note that while you can link the release ICU versions to a debug build, doing so will cause two different runtime libraries to be in use, and that almost always results in a crash or heap curruption at some point :(

To put it another way - the build script is correct - when building debug regex libraries it looks for the debug ICU dll's because those are the only ones that are guaranteed to work correctly. It's just that ICU doesn't supply those in it's binary release .zips, so you have to build them yourself.

Oh, I know that the "d" means "debug". The problem is that I am not building debug Boost.Regex; I am only building the release variant (in the bjam line I included variant=release).

anonymous — Wed, 13 Apr 2011 07:40:54 GMT

Ah yes, this is the Boost.Build bug I outlined above, Vladimir?

Fri, 15 Apr 2011 12:23:16 GMT

Unfortunately I have a complication to add to the subject.

I compiled ICU 4.6.1 on AIX 6.1 using the following line:

./configure --prefix=/some/icupath --enable-rpath --disable-debug --disable-tracing --with-library-bits=64

The latter was necessary because the default behaviour of ICU's configure on AIX is to build 32bit binaries even while the OS is 64bit.

The resulting libraries and binaries reside in the subdirectories /some/icupath/bin and /some/icupath/lib (no "64" in the paths).

When I try to compile Boost 1.46.1, like so:

./bootstrap.sh --prefix=/some/path --with-toolset=vacpp --with-icu=/some/icupath --with-libraries=regex
./bjam address-model=64

Boost concludes that:

vacpp.compile.c++ bin.v2/libs/regex/build/vacpp/debug/address-model-64/has_icu_test.o
vacpp.link bin.v2/libs/regex/build/vacpp/debug/address-model-64/has_icu
ld: 0711-317 ERROR: Undefined symbol: .u_charFromName_46
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
    xlC -brtl -bnoipath -q64 -g -qfullpath -o "bin.v2/libs/regex/build/vacpp/debug/address-model-64/has_icu" -L/PAZ/contrib_test/icu-4.6.1/bin64 -L/PAZ/contrib_test/icu-4.6.1/lib64  "bin.v2/libs/regex/build/vacpp/debug/address-model-64/has_icu_test.o"   -licuuc -licui18n -licudata

Long story short, ICU uses lib and bin, while Boost's ICU tests looks into lib64 and bin64...

Dave Abrahams — Wed, 29 Aug 2012 19:51:15 GMT

John,

Do you need access to a Win64 system where you can test this?

P.S. If you don't want to sign in, just enter your username (johnmaddock) in the Author ticket at the bottom when you comment

John Maddock — Thu, 30 Aug 2012 08:25:32 GMT

Do you need access to a Win64 system where you can test this?

I think it's moved far beyond Win64 now - actually I think those issues are mostly solved? But I assume we still have problems with Boost.Build not passing all the build variants on to the config check.

The resulting libraries and binaries reside in the subdirectories /some/icupath/bin and /some/icupath/lib (no "64" in the paths)

I think we may have reached the point at which we can't please everybody all of the time :-(

For a specific setup like that, I'd suggest building with:

export LD_LIBRARY_PATH=some-path/bin bjam include=/some-path/include linkflags=-Lsome-path/lib

So everything is "just there" in the compilers paths.

Dave Abrahams — Mon, 10 Sep 2012 13:54:33 GMT

Replying to johnmaddock:

Do you need access to a Win64 system where you can test this?

I think it's moved far beyond Win64 now - actually I think those issues are mostly solved?

I don't know how to tell whether those issues are solved or not.

But I assume we still have problems with Boost.Build not passing all the build variants on to the config check.

I don't know how to tell that either.

The resulting libraries and binaries reside in the subdirectories /some/icupath/bin and /some/icupath/lib (no "64" in the paths)

I think we may have reached the point at which we can't please everybody all of the time :-(

Personally I don't care about the path to the resulting libraries. I'm looking to create a 64-bit build with ICU on Windows.

John Maddock — Mon, 17 Sep 2012 11:24:18 GMT

Personally I don't care about the path to the resulting libraries. I'm looking to create a 64-bit build with ICU on Windows.

I believe that part should have been fixed - do you still see issues?

John.

Dave Abrahams — Tue, 18 Sep 2012 12:37:25 GMT

Yes, I still see issues. I don't remember the exact issues but there were problems when I tried to do this recently or I wouldn't have bumped this ticket. I'll try to find some time to repeat the experiment and write down the specifics.