Allow using libraries without separately compiling them

[Ulrich Eckhardt]

Currently, there are two types of libraries in Boost, one where the full implementation is inline (often templates) and others which require a separately compiled library. This feature request is about providing a third one, where the library's code is provided via include files but is not inline.

There already is an implementation based on 1.35 and 1.36 in the ​sandbox, including a readme that explains the motives for this feature a bit further and example code. A port of 1.37 is also in work there, but not finished. Later revisions and trunk haven't been started yet for lack of time.

See also:

• ​https://svn.boost.org/trac/boost/browser/sandbox/compile-in-place
• Bug #2964 (Making mutexes and lock header-only)
• ​http://lists.boost.org/Archives/boost/2006/03/102525.php
• ​http://lists.boost.org/Archives/boost/2007/04/119622.php

[Vladimir Prus]

Are you aware of any single project that uses this scheme? I don't, and I don't think Boost should be introducing new schemes, where old ones worked for years.

[Ulrich Eckhardt]

We use it successfully at the company (Sator Laser GmbH) I work for, it helps us to maintain libraries that are too small/expensive to be maintained as separate DLLs. Also, please read the rationale in the README, it really explains several motivations to this approach!

[...] I don't think Boost should be introducing new schemes, where old ones worked for years.

Take a look at e.g. #2964. This bug shows that at least one more person than me wants to use something without separately compiling a library. On the other hand, putting everything inline into a header puts much more stress on compilers and linkers, because the same code will be compiled over and over again, uselessly, so that approach is frowned upon, too. So much for the claim that the old schemes worked for years.

Further, Boost has innovation on its agenda, so finding and introducing new and better schemes is what we should do. Actually, I proposed a similar change to another project once, and the answer was like "Meh. That's not common. We don't want that. If a major project like Boost adopted it we could think about it again."!

[Jonathan Wakely <jwakely.boost@…>]

"it's up to the user to make sure that it's exactly done once"

This seems like a supremely bad idea to me, and so does asking users to #include "something.cpp"

What is the advantage of asking users to #include a file and then compile it, rather than just compiling it?

Wouldn't it achieve the same goals, without being needlessly unconventional, to make it easier for users to compile the sources into their app directly?

e.g. provide a makefile fragment with the commands necessary to turn src/libs/thread/src/thread.cpp into thread.o which could be linked directly into the application. Or make bjam print suitable commands (without running them) so that users can copy the commands into their own build system.

[Ulrich Eckhardt]

"it's up to the user to make sure that it's exactly done once" This seems like a supremely bad idea to me, and so does asking users to #include "something.cpp"

Why? It's not complicated, it doesn't give erratic and hard-to-diagnose runtime behaviour when done wrong.

What is the advantage of asking users to #include a file and then compile it, rather than just compiling it?"

There is none per se. One important thing however is that this file ideally doesn't change name or location, which can't be said about the implementation files of a library within Boost.

Wouldn't it achieve the same goals, without being needlessly unconventional, to make it easier for users to compile the sources into their app directly?"

Actually, I don't care about being unconventional, I call that innovation. What I do care about is making it easy for users to compile the sources into their application, and I can't find an easier way than "#include <boost/foo/compile_in_place.cpp>". Seriously, if you can propose a way that is as universal and as easy to use I'm happy to redraw my request, but up to now I don't see one.

Further:

• The proposal to use a makefile fragment is also not enough. Some people don't even know how to run their compiler via commandline, let alone what a makefile is, because they are using a different build system.
• Using bjam suffers the same problems and I personally haven't found my way around bjam yet, even after using it it still seems very intimidating and complicated to me. Maybe that's just me though.
• Providing build integration fragments and instructions for everything including the kitchen sink is a maintenance nightmare because it is next to impossible to automatically test it.
• Copying the commands into their own build system is also not really easy. The problem is that you need to refer to some files that are found via the include path and then fed to the compiler. Several IDE build systems don't support variable substitution let alone accessing the configured paths of the builtin compiler. #include on the other hand does without any further work.
• Why require a second build step for a separately compiled file? With my proposal, I can compile a program using e.g. Boost.Thread with a single compiler invocation and the only thing I have to adjust is the include path, which I have to adjust anyways and add a well-known stanza to the source file.

Note to self: Add an FAQ section in the README!

[Jonathan Wakely <jwakely.boost@…>]

Replying to eckhardt:

"it's up to the user to make sure that it's exactly done once" This seems like a supremely bad idea to me, and so does asking users to #include "something.cpp"

Why? It's not complicated, it doesn't give erratic and hard-to-diagnose runtime behaviour when done wrong.

Including a .cpp file is unnecessarily unconventional. It's not innovative, it's just confusing and makes people wonder what's going on. I would have fewer objections if you suggested calling it compile_inplace.inl or compile_inplace.hpp or almost *anything* else.

What is the advantage of asking users to #include a file and then compile it, rather than just compiling it?"

There is none per se. One important thing however is that this file ideally doesn't change name or location, which can't be said about the implementation files of a library within Boost.

If there's a single file which users can compile directly and its name changes, you need to change your #include directive to use the new name. Including it doesn't insulate you from that name change. Besides, if there's a single source file intended for users to compile themselves then I would expect it to be fairly stable, like the headers users are already meant to include.

Wouldn't it achieve the same goals, without being needlessly unconventional, to make it easier for users to compile the sources into their app directly?"

Actually, I don't care about being unconventional, I call that innovation. What I do care about is making it easy for users to compile the sources into their application, and I can't find an easier way than "#include <boost/foo/compile_in_place.cpp>". Seriously, if you can propose a way that is as universal and as easy to use I'm happy to redraw my request, but up to now I don't see one.

What command should be used to compile the file that has that include? Why not just use that command to compile the file directly? Including it in another file first doesn't insulate you from having to know exactly how to build the file.

Further:

• The proposal to use a makefile fragment is also not enough. Some people don't even know how to run their compiler via commandline, let alone what a makefile is, because they are using a different build system.

I don't care about people who don't bother to lean about their build system :-)

I gave the example of a makefile fragment as an example of how it would work for people using makefiles. For people using a different build system, different methods would be needed for building and linking to the object.

• Using bjam suffers the same problems and I personally haven't found my way around bjam yet, even after using it it still seems very intimidating and complicated to me. Maybe that's just me though.

I don't like bjam, but it's already used by boost and could be extended to generate the necessary commands as well as (or instead of) building the separate libraries. Whether bjam gets runs directly, or via cmake, or some other method, doesn't matter. It could print out the necessary commands and they could be incorporated into users' build systems, even if they have to ask a responsible adult to help them.

• Providing build integration fragments and instructions for everything including the kitchen sink is a maintenance nightmare because it is next to impossible to automatically test it.

But it's no worse than your proposal, which offers no way for users to know what commands to use to build compile_inplace.cpp (or the file that includes it, in your proposal.)

• Copying the commands into their own build system is also not really easy. The problem is that you need to refer to some files that are found via the include path and then fed to the compiler. Several IDE build systems don't support variable substitution let alone accessing the configured paths of the builtin compiler. #include on the other hand does without any further work.

I don't care about crappy IDEs any more than I care about programmers who don't know how to compile code :-)

However, I finally see a benefit to your proposal: users who do not know where the boost headers are located, they can find the {{{compile_inplace.cpp} file via the existing include paths in their build system.

• Why require a second build step for a separately compiled file? With my proposal, I can compile a program using e.g. Boost.Thread with a single compiler invocation and the only thing I have to adjust is the include path, which I have to adjust anyways and add a well-known stanza to the source file.

That's not true. If the compile_inplace.cpp file needs particular compiler settings (e.g. to increase template instantiation depth, or to enable pthreads, or to undefine MIN/MAX macros, etc.) then it is not sufficient just to include it in another file and build as normal - the normal build process might not be suitable for the boost library.

[Ulrich Eckhardt]

Replying to Jonathan Wakely <jwakely.boost@kayari.org>:

Replying to eckhardt:

"it's up to the user to make sure that it's exactly done once" This seems like a supremely bad idea to me, and so does asking users to #include "something.cpp"

Why? It's not complicated, it doesn't give erratic and hard-to-diagnose runtime behaviour when done wrong.

Including a .cpp file is unnecessarily unconventional. It's not innovative, it's just confusing and makes people wonder what's going on. I would have fewer objections if you suggested calling it compile_inplace.inl or compile_inplace.hpp or almost *anything* else.

The suffix 'inl' suggest to me that it contains some definitions of inline functions to me, the suffix 'hpp' suggests declarations, but neither is is the case. It really is a source file (you could compile it separately), so the 'cpp' is IMHO deserved. Also, if people stumble across an included 'cpp' file and as a result start to think, that is a good thing, because noticing that this is not just one more include is important for using it correctly. The point is that the thing does something unconventional and the name of the file properly reflects that.

One important thing however is that this file ideally doesn't change name or location, which can't be said about the implementation files of a library within Boost.

If there's a single file which users can compile directly and its name changes, you need to change your #include directive to use the new name. Including it doesn't insulate you from that name change.

There is thread.h and thread.cpp in version X. In version X+1 there is thread.h and thread_posix.cpp and thread_win32.cpp. Using a separate source file to include all real sources allows library maintainers to rearrange their code as they see necessary without breaking that. IOW, this single source file provides a level of indirection.

What command should be used to compile the file that has that include? Why not just use that command to compile the file directly? Including it in another file first doesn't insulate you from having to know exactly how to build the file.

You should get an #error if you do it wrong, along with a comment that tells you how that is wrong. Actually, there was very little to do in order to make this work. Also, and that is an important point, I want to build the sources in exactly the same configuration as the user's sources. The reason is that this makes different ABIs a non-issue. Testing a new version of Boost, a different compiler or just incompatible compiler settings doesn't require you to do anything further to assure binary compatibility.

[bjam] could be extended to generate the necessary commands as well as (or instead of) building the separate libraries.

bjam doesn't support my compiler (I'm not kidding, I'm working with a cross compiler for an embedded target!) or I'm not using compatible settings for my code, so now I have to find out how to tell bjam about these specialties I'm using.

[Vladimir Prus]

Replying to eckhardt:

bjam doesn't support my compiler (I'm not kidding, I'm working with a cross compiler for an embedded target!)

What compiler is that?

[Jonathan Wakely <jwakely.boost@…>]

Replying to eckhardt:

There is thread.h and thread.cpp in version X. In version X+1 there is thread.h and thread_posix.cpp and thread_win32.cpp. Using a separate source file to include all real sources allows library maintainers to rearrange their code as they see necessary without breaking that. IOW, this single source file provides a level of indirection.

Yes, obviously it provides a level of indrection, but that's not an argument in favour of using #include for a .cpp file. Compiling the .cpp file directly would still allow the sources to be moved around.

What command should be used to compile the file that has that include? Why not just use that command to compile the file directly? Including it in another file first doesn't insulate you from having to know exactly how to build the file.

You should get an #error if you do it wrong, along with a comment that tells you how that is wrong. Actually, there was very little to do in order to make this work. Also, and that is an important point, I want to build the sources in exactly the same configuration as the user's sources. The reason is that this makes different ABIs a non-issue. Testing a new version of Boost, a different compiler or just incompatible compiler settings doesn't require you to do anything further to assure binary compatibility.

You can only use #error for something detectable by the preprocessor, so won't work for many compiler switches (maybe most of them.)

As I said, the build options used for the user's sources may not be suitable for the library. I don't just mean ABI-changing options, Boost.Graph has these settings:

    # # Intel compiler ICEs if we turn optimization on
    <toolset>intel-vc71-win-9.1:<optimization>off
    # Without these flags, MSVC 7.1 and 8.0 crash
    <toolset>msvc-7.1:<cxxflags>-GR-
    <toolset>msvc-8.0:<cxxflags>-GR-

Or have a look at src/iostreams/build/Jamfile.v2 and tell me that making a user duplicate that logic in their own build system makes it easier, bearing in mind that you have been arguing the case for users who can't even find and compile a source file, but only use #include!

[Ulrich Eckhardt]

The compiler not supported is MSVC when cross-compiling for MS Windows CE, at least it wasn't supported some time ago (at around 1.34 ... yes, I know...).

[Ulrich Eckhardt]

(In [54605]) Added FAQ section (refs #3103).

[Vladimir Prus]

Ulrich,

this issue was on my weekly issue review page for quite some time, and I guess I should do something at last. I think you'll agree that this issue is somewhat controversial, and did not gain, at least yet, much support. In fact, it does not seem anybody has agreed with this specific approach. Therefore, it does not seem right to track this issue as a defect/enhancement that "Boost community" wishes to solve. In other words, this issue does not constitute an action item either for me, or anybody except for you. Then, it's better kept on your personal todo list -- and then, if you prototype enough of this and convince boost folks this should be implemented, it can be re-added back to "Boost todo list".

Because of the above, I am closing this issue as wontfix.

Thanks, Volodya