boost >= 1.54 failes to compile with gcc-4.8.2 and LTO enabled

[steffen@…]

I tried to compile boost on my gentoo systemd and it failes while linking libboost_log.so with the following error:

gcc.link.dll bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/libboost_log.so.1.55.0

"x86_64-pc-linux-gnu-g++" -o "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/libboost_log.so.1.55.0" -Wl,-h -Wl,libboost_log.so.1.55.0 -shared -Wl,--start-group "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/dump_ssse3.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/dump_avx2.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/attribute_name.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/attribute_set.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/attribute_value_set.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/code_conversion.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/core.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/record_ostream.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/severity_level.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/global_logger_storage.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/named_scope.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/process_name.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/process_id.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/thread_id.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/timer.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/exceptions.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/default_attribute_names.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/default_sink.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/text_ostream_backend.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/text_file_backend.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/syslog_backend.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/thread_specific.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/once_block.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/timestamp.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/threadsafe_queue.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/event.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/trivial.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/spirit_encoding.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/format_parser.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/date_time_format_parser.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/named_scope_format_parser.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/unhandled_exception_count.o" "bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/dump.o" "bin.v2/libs/thread/build/gcc-4.8/gentoorelease/pch-off/threading-multi/libboost_thread.so.1.55.0" "bin.v2/libs/filesystem/build/gcc-4.8/gentoorelease/pch-off/threading-multi/libboost_filesystem.so.1.55.0" "bin.v2/libs/date_time/build/gcc-4.8/gentoorelease/pch-off/threading-multi/libboost_date_time.so.1.55.0" "bin.v2/libs/chrono/build/gcc-4.8/gentoorelease/pch-off/threading-multi/libboost_chrono.so.1.55.0" "bin.v2/libs/system/build/gcc-4.8/gentoorelease/pch-off/threading-multi/libboost_system.so.1.55.0" -Wl,-Bstatic -Wl,-Bdynamic -lrt -Wl,--end-group -Wl,-O1 -Wl,--as-needed -march=native -O2 -pipe -ggdb -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -fira-loop-pressure -ftree-vectorize -ftree-loop-linear -flto=5 -fuse-linker-plugin -Wl,-znow -Wl,--sort-common -Wl,--hash-style=gnu -Wl,--enable-new-dtags -pthread -lrt -lpthread

/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/avx2intrin.h: In function ‘_ZN5boost3log11v2_mt_posix3aux20dump_data_wchar_avx2EPKvmRSt13basic_ostreamIwSt11char_traitsIwEE.part.4’: /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/avx2intrin.h:737:62: error: ‘builtin_ia32_psrlwi256’ needs isa option -m32

return (m256i)builtin_ia32_psrlwi256 ((v16hi)A, B);

[snip a lot more of these isa option errors] /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/avx2intrin.h:585:23: error: ‘builtin_ia32_pshufb256’ needs isa option -m32

(v32qi)Y);

make: /home/misc/gentoo/tmp/portage/dev-libs/boost-1.55.0-r1/temp/ccobKVrF.ltrans0.ltrans.o Error 1 make: Waiting for unfinished jobs.... lto-wrapper: make returned 2 exit status /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/../../../../x86_64-pc-linux-gnu/bin/ld: fatal error: lto-wrapper failed collect2: error: ld returned 1 exit status ...skipped <pstage/lib>libboost_log.so.1.55.0 for lack of <pbin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi>libboost_log.so.1.55.0... gcc.compile.c++ bin.v2/libs/log/build/gcc-4.8/gentoorelease/log-api-unix/pch-off/threading-multi/filter_parser.o

I've attached the complete gentoo build.log. Here are some additional details about my system environment:

Portage 2.2.8-r1 (default/linux/amd64/13.0, gcc-4.8.2, glibc-2.18-r1, 3.13.5-HAUIHAU x86_64) ================================================================= System uname: Linux-3.13.5-HAUIHAU-x86_64-Intel-R-_Core-TM-_i7-2620M_CPU_@_2.70GHz-with-gentoo-2.2 KiB Mem: 7974844 total, 2094852 free KiB Swap: 8388604 total, 7620020 free Timestamp of tree: Mon, 10 Mar 2014 10:30:01 +0000 ld GNU gold (GNU Binutils 2.24) 1.11 ccache version 3.1.9 [disabled] app-shells/bash: 4.2_p45-r1 dev-java/java-config: 2.2.0 dev-lang/python: 2.7.6, 3.3.4 dev-util/ccache: 3.1.9-r3 dev-util/cmake: 2.8.12.2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.11.6, 1.14.1 sys-devel/binutils: 2.24-r2 sys-devel/gcc: 4.8.2-r1 sys-devel/gcc-config: 1.8 sys-devel/libtool: 2.4.2 sys-devel/make: 4.0-r1 sys-kernel/linux-headers: 3.13 (virtual/os-headers) sys-libs/glibc: 2.18-r1

CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe -ggdb -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -fira-loop-pressure -ftree-vectorize -ftree-loop-linear -flto=5 -fuse-linker-plugin" CHOST="x86_64-pc-linux-gnu" CXXFLAGS="-march=native -O2 -pipe -ggdb -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -fira-loop-pressure -ftree-vectorize -ftree-loop-linear -flto=5 -fuse-linker-plugin" LDFLAGS="-Wl,-O1 -Wl,--as-needed -march=native -O2 -pipe -ggdb -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -fira-loop-pressure -ftree-vectorize -ftree-loop-linear -flto=5 -fuse-linker-plugin -Wl,-znow -Wl,--sort-common -Wl,--hash-style=gnu -Wl,--enable-new-dtags"

If you need further information, please let me know.

[steffen@…]

boost 1.55 build log

[steffen@…]

No one ever looked at this issue?

[viboes]

Try moving it to Log library

[Andrey Semashev]

Can you provide the b2 or bjam command line so I could reproduce the error? BTW, I compile with gcc 4.8.2 on Kubuntu without problems.

[anonymous]

Hi,

this is the command, Gentoo's ebuild used to build boost:

b2 gentoorelease -j5 -q -d+2 --user-config=/home/misc/gentoo/tmp/portage/dev-libs/boost-1.55.0-r1/work/boost_1_55_0/user-config.jam -sICU_PATH=/usr --without-mpi --without-python --without-context --without-coroutine pch=off --boost-build=/usr/share/boost-build --prefix="/home/misc/gentoo/tmp/portage/dev-libs/boost-1.55.0-r1/image/usr" --layout=system threading=multi link=shared

and this is the content of user-config.jam:

using gcc : 4.8 : x86_64-pc-linux-gnu-g++ : <cflags>"-march=native -O2 -pipe -ggdb -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -fira-loop-pressure -ftree-vectorize -ftree-loop-linear -flto=5 -fuse-linker-plugin" <cxxflags>"-march=native -O2 -pipe -ggdb -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -fira-loop-pressure -ftree-vectorize -ftree-loop-linear -flto=5 -fuse-linker-plugin -std=gnu++98" <linkflags>"-Wl,-O1 -Wl,--as-needed -march=native -O2 -pipe -ggdb -floop-interchange -floop-strip-mine -floop-block -ftree-loop-distribution -fira-loop-pressure -ftree-vectorize -ftree-loop-linear -flto=5 -fuse-linker-plugin -Wl,-znow -Wl,--sort-common -Wl,--hash-style=gnu -Wl,--enable-new-dtags" ;

Please let me know if you need further information.

[anonymous]

I can see you're using LTO. That may be the source of the problem. There are several source files in Boost.Log which are compiled with different compiler flags. In particular, it's dump_avx2.cpp, which contains the failing function you seen in the compiler errors. To verify that try disabling LTO and rebuilding the library.

I have no experience with GCC LTO, but from what I know I can't use all the flags on the linking stage since that would make the whole resulting binary AVX2-only instead of only just the optimized part in dump_avx2.cpp. I could try adding attributes to the particular functions in dump_avx2.cpp so that they contain the required compiler options. But really, this has to be done by the compiler since I already pass the options on the compilation stage.

[anonymous]

Have you been able to reproduce the failure with my flags? Disabling LTO fixes the issue, sorry I forgot to mention that in my initial bug report. I guess this bug should be reported upstream to the GCC devs? Can you do this or do I have to deal with it? I'm wondering why I haven't found any other guys around who fail to compile boost >=1.54 with LTO enabled.

[Andrey Semashev]

Replying to anonymous:

Have you been able to reproduce the failure with my flags?

No, I have tried to compile git develop branch with your flags and it succeeded without errors. As I said, I have gcc 4.8.2 on Ubuntu. I'm not sure how to verify that LTO was actually used. The object files (*.o) have the __gnu_lto_v1 symbol, if that's an indication. OTOH, objdump produces disassembly for these files, which I find surprising.

Disabling LTO fixes the issue, sorry I forgot to mention that in my initial bug report. I guess this bug should be reported upstream to the GCC devs? Can you do this or do I have to deal with it? I'm wondering why I haven't found any other guys around who fail to compile boost >=1.54 with LTO enabled.

I suppose, LTO is not used very often yet in Linux world. Since I cannot reproduce the failure, it's probably easier for you to communicate with gcc devs (or package maintainers in Gentoo) regarding this issue.

[steffen@…]

I tried to compile it under OpenSUSE 13.1 (gcc 4.8.1) on another system (also Core i7 Sandy Bridge) and it failed there with the same error.

Then I tried to compile it on my older computer under Ubuntu 14.04 (AMd Athlon X2 4600), and it failed with an internal compiler error, I guess due to the lack of AVX support in that old CPU:

libs/log/src/dump_avx2.cpp: In function ‘_ZN5boost3log11v2_mt_posix3aux20dump_data_wchar_avx2EPKvmRSt13basic_ostreamIwSt11char_traitsIwEE.part.4’:
libs/log/src/dump_avx2.cpp:260:6: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
 void dump_data_wchar_avx2(const void* data, std::size_t size, std::basic_ostream< wchar_t >& strm)
      ^
libs/log/src/dump_avx2.cpp:260:6: internal compiler error: in convert_move, at expr.c:316
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.

Did you use my user-config.jam? Then bjam should have called g++ with "-flto=5" option. You can see the compiler options in the output lines of bjam. I'll file a bug report for this to GCC and give you the id.

[Andrey Semashev]

Replying to steffen@…:

Then I tried to compile it on my older computer under Ubuntu 14.04 (AMd Athlon X2 4600), and it failed with an internal compiler error, I guess due to the lack of AVX support in that old CPU:

Host CPU features should not influence the code generation, unless you use -march=native. BTW, you shouldn't really use -march and similar flags, b2 has instruction-set property, which has the same meaning but may also be picked up by Boost libraries build scripts.

Did you use my user-config.jam?

Yes, I only had to change the compiler executable name and correct directories and similar stuff in the command line. I'll try experimenting more though.

Then bjam should have called g++ with "-flto=5" option. You can see the compiler options in the output lines of bjam.

Yes, all options were there, both at compilation and linking stages. I wonder if it could be related to -march=native flag since I tried this on a Haswell machine. I'll try on another machine when I have time.

I'll file a bug report for this to GCC and give you the id.

Thanks, I appreciate that.

[Andrey Semashev]

After all my experiments I still fail to reproduce the error. I even tried to replay the compilation with the command lines from your build log. I've discovered one nasty detail though - the resulting binaries in my case are AVX2-only. I guess my compiler somehow applies the flags for dump_avx2.cpp to all object files. Needless to say that if that's how LTO is supposed to work, it cannot be used with projects such as Boost.Log.

However, I have a patch I'd like to test. Basically, it adds attributes to all SSE/AVX functions so that the compiler knows what CPU they are targeted for. I'd appreciate if you could perform the following experiment:

1. Apply the patch from this ticket.
2. Perform a clean rebuild of Boost.
3. If Boost.Log compiles this time, run the following command:

objdump -dS libboost_log.so.1.55.0 >libboost_log.so.1.55.0.S

(add the directory name to the compiled library, if necessary).

4. Compress libboost_log.so.1.55.0.S and attach it to this ticket.

The libboost_log.so.1.55.0.S is the disassembled library; I'd like to inspect the resulting code for whether it's AVX2-only or not.

[Andrey Semashev]

The patch adds target CPU attributes to the optimized functions.

[steffen@…]

I have reported it upstream. Here is the link: gcc.gnu.org/bugzilla/show_bug.cgi?id=60964. Seems like your suggestion was really close with -march=native (I used this as it was suggested in wiki.gentoo.org/wiki/CFLAGS). They asked for a test case. May you provide one based on your log library?

I also tried to compile with -march=corei7-avx and that solved the issue for me. I may also try to compile with your patch.

[Andrey Semashev]

May you provide one based on your log library?

I'm still not able to reproduce the failure, sorry.

[anonymous]

Could you please give some detail about your used platform? Which Ubuntu release/GCC version are you using? This bug is only present in GCC 4.8 and newer. I've now checked also Fedora 20 and had the same issue as under Gentoo and current OpenSUSE.

I'll try to compile boost with your patch and -march=native next week as I'm on business trip until next week.

[Andrey Semashev]

I'm on Kubuntu 13.10 x86_64, gcc -v gives:

gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc-4.8.real
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.8.1-10ubuntu9' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu9)

It's the standard compiler installed from the Ubuntu packages.

[Andrey Semashev]

I managed to reproduce the error, after I upgraded to Kubuntu 14.04 and tried to compile Boost with -march=native on a Sandy Bridge machine. The patch I attached earlier does not help. For now I have no further ideas how to work it around and make -march=native work, the issue has to be fixed in gcc.

Considering that even when the compilation succeeds (i.e. -march=native is not used) the resulting binaries are incorrect, I'll document gcc LTO as unsupported.

[Andrey Semashev]

BTW, I reported another LTO bug: ​http://gcc.gnu.org/bugzilla/show_bug.cgi?id=61043

[steve@…]

This isn't a GCC bug. Use of -march is expected to produce code suitable for *only* the target CPU architecture. Usage of -march or other -m flags to enable features from within build-systems is incorrect usage of the compiler. The build-system should detect which features are available on the configured compiler, not the other way around. Generic code should be generated with with a specific minimum requirement in mind, but should always be limited to whatever the build toolchain is constrained by.

[Andrey Semashev]

Read the discussion and the referred bug. The architecture-specific flags are passed for compilation of the files, which are executed only when the relevant CPU features are present in run time. The compiler bug is that it produces binary code as if the flags were specified for all files rather than those few special ones.

[steve@…]

My point is though that even when generating generic code with run-time detection it's inappropriate to override -march. If generic code is required -march can be used to specify a generic target (such as x86_64, or i686) and then use feature -m switches. If building not building generic code, it should generally be constrained to the specific set of enabled features, and certainly shouldn't override the per target code generation optimization. The GCC bug recommends using attribute(target("...")) which allows specific functions to use features not necessarily available for the given -march, and which doesn't fail for LTO, and actually produces the desired behaviour. See: ​https://gcc.gnu.org/onlinedocs/gcc/Function-Specific-Option-Pragmas.html

[Andrey Semashev]

Specifying different compiler options for different translation units is a common practice, I'm not sure what you're arguing for.

The bug report also mentions that the attributes don't work up until gcc 4.9. And the bug is fixed proper in 5.1. I do not think mangling the code just for 4.9 is worth the effort.

[anonymous]

Actually, I'm using gcc-5.3, and it still causes build failure on amdfam10 (AMD Phenom II). I'm arguing that overriding -march isn't one of the compiler options that should be set for different translation units. This is something that often crops up in Gentoo, where -march is set system-wide, and then gets overridden by upstream build-systems not following best practises. There are plenty of patches in Portage to work around these issues.

The simplest solution is to just make the optimized functions individually build-time configurable, that way a generic distributor build can chose a suitable generic -march, along with enabling run-time CPU optimizations. Conversely, a targeted build can just include support and optimization as appropriate.

I brought up the function attributes because that's the solution GCC has implemented to support this type of situation as was discussed in the GCC bug report, but as you say, it's only good for GCC>4.9.

[Andrey Semashev]

Actually, I'm using gcc-5.3, and it still causes build failure on amdfam10 (AMD Phenom II).

Could you provide more details on the failure? Is the error reproducible with the latest vanilla Boost release?

The simplest solution is to just make the optimized functions individually build-time configurable

That is too difficult to maintain.