Change in default build variants broke "Getting Started"

[tychom@…]

Boost.Build no longer builds the same library variants that it used to, which caused the instructions in the Getting Started guide to be wrong.

Original description:

[getting started] No information on building static libraries with bjam under Windows

Problem relates to the boost 1.35 with VC++ 8.0.

When using bjam to build the Boost libraries from source there is no information provided in an easily accessible manner describing how to build static versions of those libraries for which such a thing is appropriate - threads, regex etc. As a number of libraries attempt to link statically by default this is quite a pain.

The getting started guide for Windows found at 'more/getting_started/windows.html#or-build-binaries-from-source' provides no information on the bjam setting 'link=static'. In order to come across this information currently one needs to follow the link 'Boost.Build reference manual' close to the very bottom of the 'Getting Started' page, then take the 'Documentation' link and finally 'Static and shared libraries'.

If additional information could be put in the bjam section of the getting started guide then this will 1) ensure less people wonder why the static libraries were not built in the first place 2) ensure less people have to hunt through the necessary links to find information on building static libraries.

[Dave Abrahams]

the default build process used to create both static and shared binaries. Has that changed? if you can suggest specific changes I can make to the text, that would save me some time. Thanks

[olenz@…]

I'm using a standard Linux distribution (kubuntu 8.04), and boost 1.35, and I do have the same problem, i.e. it is not Windows-specific. The static libraries are not built when using the instructions given in the "Getting started" documentation:

http://www.boost.org/doc/libs/1_35_0/more/getting_started/unix-variants.html

Effectively, this means also, that the example from "Build a Simple Program Using Boost" does not work, as the static libraries do not exist.

[olenz@…]

An additional comment: after some research, I found out that the way to build static libraries (on Unix, if you do not have bjam installed) is

make BJAM_CONFIG='link=static' install

which should probably go into the "Getting started" documentation, if you do not want to include an extensive paragraph about dynamic libraries and LD_LIBRARY_PATH...

[olenz@…]

And one more...

Finally, I also had to include "-lpthread" into the compilation command line to be able to successfully link the example program (oh, btw, I was always talking about the example in "How to Use a Boost Library Binary", not the "Header-only" one). This is by far not obvious for inexperienced users.

I would guess that this is because I'm using a dual core machine, but as these are standard nowadays...

[Dave Abrahams]

I'm really not sure what to do about this one because it seems to me that someone must've changed a Boost.Build behavior to cause the Getting Started Guide to be broken. I certainly didn't make significant changes to the guide. So I'm reassigning this one to Rene and Cc'ing Volodya, without being sure that they're the right parties, to stimulate a discussion. Guys, if this has nothing whatever to do with you, please assign it back to me, but try to help me understand what happened first!

Thanks

[René Rivera]

What happened is that as agreed, once upon a time, we changed to only build a single variant for the install. It happened that this single variant, the link=shared/runtime-link=shared/threading=multi/release combination, is neither what the getting started refers to nor what the autolink defaults to. There are two options for "fixing" this problem:

• Change the docs to refer to the current single variant and change autolink to default to the current single variant.
• Build two variants by default, the above, and link=static/runtime-link=shared/threading=multi/release.

I don't know how complicated changing the autolink code would be, nor the repercussions to users. I should say that most users are happy that the default build is down to one variant. It's only been inexperienced first-time users that have had problem because of the doc & autolink discrepancy.

[Dave Abrahams]

Replying to grafik:

What happened is that as agreed, once upon a time, we changed to only build a single variant for the install. It happened that this single variant, the link=shared/runtime-link=shared/threading=multi/release combination, is neither what the getting started refers to nor what the autolink defaults to.

Well, that's an embarrassment!

There are two options for "fixing" this problem:

• Change the docs to refer to the current single variant and change autolink to default to the current single variant.
• Build two variants by default, the above, and link=static/runtime-link=shared/threading=multi/release.

What about

• Change the single variant we build and the autolink code to be consistent with the docs
• Change the single variant we build and the docs to be consistent with the autolink code

?

I don't know how complicated changing the autolink code would be, nor the repercussions to users. I should say that most users are happy that the default build is down to one variant. It's only been inexperienced first-time users that have had problem because of the doc & autolink discrepancy.

I guess it goes without saying that inexperienced first-time users need the most help.

[mklein1]

I've run into a similar issue in that when building for multiple variants, the static libraries do get built, they just don't get installed.

[Dave Abrahams]

Replying to mklein1:

I've run into a similar issue in that when building for multiple variants, the static libraries do get built, they just don't get installed.

Goot to know. Since this ticket is filed against the Getting Started Guide, would you please report that as a bug against Boost.Build or "Building Boost" or another appropriate component?

Thanks

[René Rivera]

Replying to dave:

Well, that's an embarrassment!

To say the least :-(

What about

• Change the single variant we build and the autolink code to be consistent with the docs
• Change the single variant we build and the docs to be consistent with the autolink code

?

Because the link=static variant is not the most widely used one, AFAIK. Note, I have no real personal opinion as to which one to default to other than trying to satisfy the majority ;-)

I don't know how complicated changing the autolink code would be, nor the repercussions to users. I should say that most users are happy that the default build is down to one variant. It's only been inexperienced first-time users that have had problem because of the doc & autolink discrepancy.

I guess it goes without saying that inexperienced first-time users need the most help.

Very true. And this is why we decided the single default to be link=shared, because the variant that seemed to be used by most third party applications and libraries that use Boost. Which, from the experience answering how to build, is what I got asked about most often. I.e. the most common question I get is "I'm trying to build X which wants Boost library Y.dll/so. How do I build that?". And this is from sitting on IRC for two years now. Which is where many such new users end up asking.

[Dave Abrahams]

Replying to grafik:

Replying to dave:

Well, that's an embarrassment!

To say the least :-(

What about

• Change the single variant we build and the autolink code to be consistent with the docs
• Change the single variant we build and the docs to be consistent with the autolink code

?

Because the link=static variant is not the most widely used one, AFAIK. Note, I have no real personal opinion as to which one to default to other than trying to satisfy the majority ;-)

Well, me neither. And since you happen to have some sense of what the majority needs, I'd be happy to go with whichever your gut says is the better bet... or even whichever one you think causes the least risk. However, please consider that if we decide to change the docs, it would be very desirable to avoid adding complexity (if possible).

[Dave Abrahams]

Replying to grafik:

Replying to dave:

Well, that's an embarrassment!

To say the least :-(

What about

• Change the single variant we build and the autolink code to be consistent with the docs
• Change the single variant we build and the docs to be consistent with the autolink code

Because the link=static variant is not the most widely used one, AFAIK.

Wait a sec; that reasoning doesn't address the 2nd bullet in my list of two, does it? I presume that autolink doesn't specify static libs by default?

[René Rivera]

Replying to dave:

Replying to grafik:

Replying to dave:

What about

• Change the single variant we build and the autolink code to be consistent with the docs
• Change the single variant we build and the docs to be consistent with the autolink code

Because the link=static variant is not the most widely used one, AFAIK.

Wait a sec; that reasoning doesn't address the 2nd bullet in my list of two, does it? I presume that autolink doesn't specify static libs by default?

You presume incorrectly ;-) Currently autolink defaults to link=static. And hence matches the docs. Or rather the docs match autolink. Which is why I asked about the risk of changing the autolink code at this point. I think it's a high risk to do so now. And it's why I'm suggesting building both link=static and link=shared as the "quick fix".

Hm, I should check precisely what is the default for autolink.

[Dave Abrahams]

Replying to grafik:

Currently autolink defaults to link=static. And hence matches the docs. Or rather the docs match autolink. Which is why I asked about the risk of changing the autolink code at this point. I think it's a high risk to do so now. And it's why I'm suggesting building both link=static and link=shared as the "quick fix".

Sounds reasonable.

[René Rivera]

OK, default minimal build changed to also build link=static. See changesets [46953] and [46957]. Some tasks for changing this behavior in the next release to come.

[Vladimir Prus]

This bug report talks about Windows. Why exactly do we need to build static libraries on Linux?

[René Rivera]

Replying to vladimir_prus:

This bug report talks about Windows. Why exactly do we need to build static libraries on Linux?

Because there's no way to limit the default build variants based on target-os. Or is there?

[John Maddock]

Sorry to come late to the party, but can I just point out that the "fix" does not actually fix anything for Windows users:

• For VC++ there are either 4 or 6 possible runtime library variants (depending which version you have), as well as a further 2 dymanic vs static variants, and unless you build all of those variants, then the user absolutely will experience linker errors from the auto-link code at some point.

• In particular, as things now stand, the user may well have static variants of the libraries, but they are *release* variants which won't be binary compatible with the default IDE *debug* build, so IMO they're actually of no use whatsoever to a first time user.

Frankly anything other than --build-type complete is utter useless for VC++ users, sorry to be so blunt, but as you know I was against the change to build only a single variant and still am: I've already lost count of the number of times I've had to explain this, which is rather annoying as the auto-link header was introduced to avoid having to explain the whole build variant thing!

Regards, John.

[René Rivera]

I would love making it such that build-type=complete is the default on MSVC (or other Windows autolink toolsets). But BBv2 doesn't have that level of control, AFAIK.

[Dave Abrahams]

If not, a sensible policy would be that users of Cygwin (and any other toolsets that don't auto-link on windows -- Mingw?) get a complete build unless they explicitly request otherwise. MSVC and its brethren make up an overwhelming majority of the windows compiler market.

[Vladimir Prus]

Dave, I'm confused about your last change of this issue -- what build changes are called for? Is the conclusion to implement complete build by default on MSVC? Was the Linux version of getting started adjusted to document (the desired and agreed-on) change in the set of built variants?

[Dave Abrahams]

Replying to vladimir_prus:

Dave, I'm confused about your last change of this issue -- what build changes are called for? Is the conclusion to implement complete build by default on MSVC?

Is everyone somehow waiting for me to decide? As far as I can tell, the workable solution above seems to be that BB creates all variants by default when target-os=windows, but frankly once it was understood that changes by others to BB broke the "getting started" guide I stopped trying to figure out exactly what the best solution to the problem was.

Was the Linux version of getting started adjusted to document (the desired and agreed-on) change in the set of built variants?

No, but I'm not yet sure what change is needed. s/libboost_regex-gcc34-mt-d-1_35.a/libboost_regex-gcc34-mt-d-1_35.so/ ?

[Dave Abrahams]

Replying to grafik:

Replying to dave:

Well, that's an embarrassment!

To say the least :-(

What about

• Change the single variant we build and the autolink code to be consistent with the docs
• Change the single variant we build and the docs to be consistent with the autolink code

?

Because the link=static variant is not the most widely used one, AFAIK. Note, I have no real personal opinion as to which one to default to other than trying to satisfy the majority ;-)

I guess I was thrown by Rene's answer here, which seemed like an answer to a "why...?" question...

I guess it goes without saying that inexperienced first-time users need the most help.

Very true. And this is why we decided the single default to be link=shared, because the variant that seemed to be used by most third party applications and libraries that use Boost. Which, from the experience answering how to build, is what I got asked about most often. I.e. the most common question I get is "I'm trying to build X which wants Boost library Y.dll/so. How do I build that?". And this is from sitting on IRC for two years now. Which is where many such new users end up asking.

...which reminds me why the GSG used static libs when we were building both kinds: one less chance of problems such as failure to set LD_LIBRARY_PATH. What additional changes will be needed to make the instructions work if it's a .so? Various *nixes have different variables for shared library search IIRC.

[Vladimir Prus]

I don't think we should target folks who installed Linux yesterday. Anybody doing development on Linux knows how to link to libraries, and about LD_LIBRARY_PATH. I don't think GSG should, among other thing, be "teach yourself Linux programming in 10 mins" document.

[Dave Abrahams]

Replying to vladimir_prus:

I don't think we should target folks who installed Linux yesterday. Anybody doing development on Linux knows how to link to libraries, and about LD_LIBRARY_PATH. I don't think GSG should, among other thing, be "teach yourself Linux programming in 10 mins" document.

This attitude really rubs me the wrong way. 'Nuf said on that.

I wrote the current GSG, so I get to decide what its purpose is. It should be a "give people foolproof instructions so that they don't continually come to us asking why they can't get started with the libraries, or worse yet, give up without asking document."

Some people installing on Linux aren't doing development there. They're just configuring the libraries so the real developers can use them, and it's not uncommon that these people are only really familiar with Windows. It would be great if everybody were experienced enough to get over these humps by themselves, but in reality they're not. Also, with 10 or so new things to get right in order to make the process work, it's easy for even experienced users to overlook something. When things don't work for me, I go back over the instructions step by step and make sure I did everything right. If we don't provide instructions that work, people will have little reason to trust the rest of our documentation.

Reassigning to Beman because I consider it absolutely critical that 1.36 not ship with this ticket still open. Sorry, Beman, but we appear to be stalled over here. Can you help us get unstuck?

[Vladimir Prus]

Oh, well, if you want to invent work for yourself, you can document LD_LIBRARY_PATH, or anything. I remain convinced that building only shared, MT libraries on Linux is the right thing. And I likewise remain convinced that it's not the job of Boost to explain how do use gcc, or ld either -- much like it's not job of any other project except for gcc and binutils themself.

The only behavioral change suggested in this issue is to build both shared and static variants on windows -- either always on windows, or only for msvc. I don't get the impression there's clear concensus on that, and can implement anything anybody wants.

And this means, I think, that getting started can be adjust to document the current behaviour right away, at least for Linux.

[Dave Abrahams]

Replying to johnmaddock:

Frankly anything other than --build-type complete is utter useless for VC++ users, sorry to be so blunt, but as you know I was against the change to build only a single variant and still am: I've already lost count of the number of times I've had to explain this, which is rather annoying as the auto-link header was introduced to avoid having to explain the whole build variant thing!

So would someone remind me why we didn't change it to --build-type=complete by default for Windows users? This appears to be the only open issue remaining on this ticket. Is the GSG still broken? If the GSG now works for everyone, but the default build type is still useless to Windows users, then perhaps John should open a new ticket for that?

[John Maddock]

I believe it's still broken-ish: there is a mention of --build-type=complete in an example command line, but no indication of what it does, or that it's required for most Windows compilers.

No idea as for why we haven't made it the default for Windows compilers, IMO it should be,

John.

[Dave Abrahams]

John,

what you described doesn't amount to breakage. The GSG is only broken if someone following its directions will not get the results it promises. If there's an improvement we could make, please open a new ticket for it. In particular, if --build-type=complete ought to be the default on windows, please open a ticket against the appropriate component that says so, and if you feel --build-type=complete should be described in the GSG, open a ticket for that, too.

[John Maddock]

Sorry Dave but I disagree.

Consider an MSVC user that Builds without with currently undocumented --build-type=complete flag, they get static and dll libraries built against the msvc release runtimes.

Then they create a new IDE project (debug build by default) and follow step 6.1... and get a linker error because the necessary binary does not exist. Same if they follow 6.2.

IMO it's still broken, just differently than before (but the right fix is to change the build process IMO - fixing the getting started guide is a stopgap).

John.

[Dave Abrahams]

The flag, if it is required, doesn't need to be explained in the GSG as long as the reader is instructed to use it, and in fact I might be willing to assert that it shouldn't be explained there. This document is supposed to be a gentle introduction that leaves out many details.

That said, I hadn't realized someone put square brackets around the flag in the example (and in italics, to boot, which is totally inconsistent with the conventions used elsewhere in that document). That seems to suggest that the flag is optional. If, as you say, the GSG can't be completed without that flag, then by all means it is broken. We need to make a 2-character change (remove the brackets). So I will do that and close this ticket. Please open new tickets against any other components you think are broken, OK? There's nothing wrong with referencing this ticket from those others.

[Vladimir Prus]

I have asked on mailing list about appropriate defaults on Windows and will switch to static linking by default, so as to match autolink, for 1.39.

[Vladimir Prus]

Static linking is now the default on windows.