| 1521 | | If the last section was somewhat speculative due to the present uncertainties about the future developments in C++, these remaining two sections are almost entirely discussion pieces as they have no known good answers. Consider them therefore more as food for thought rather than recommendations, and they are obviously my (Niall Douglas) personal perspective and interpretation on things. I'll make the following discussion entirely Boost centric, but lots of corporations out there (Google is the most famous example) have truly enormous monolithic C++ codebases where this discussion equally applies if not far more so, so please consider find and replacing all mentions of Boost with ''<insert your large C++ code base here>''. |
| | 1521 | If the last section was somewhat speculative due to the present uncertainties about the future developments in C++, these remaining two sections are almost entirely discussion pieces as they have no known good answers. Moreover, this section and the next section are both essays arguing the opposite from one another on the basis of plenty sound good reasons, so the intent is really to make you ''think'' and ''reflect'' before you design your (C++) library. |
| | 1522 | |
| | 1523 | Consider them therefore more as food for thought rather than recommendations, and they are obviously my (Niall Douglas) personal perspective and interpretation on things. I'll make the following discussion quite Boost-centric as contractual NDAs prevent me discussing problems in commercial code bases I have worked upon, but lots of corporations out there (Google is the most famous example) have truly enormous monolithic C++ codebases where this discussion equally applies if not far more so, so please consider find and replacing all mentions of Boost with ''<insert your large C++ code base here>''. |
| | 1524 | |
| | 1525 | And finally, this section is going to look quite eccentric to most engineers as I will use arguments unfamiliar to most in software. Part of this is because I am equally trained (in terms of degrees) in Economics and Management as I am in Software, plus I am an affiliate researcher with the [http://www.wici.ca Waterloo Institute for Complexity and Innovation] Research Institute so this analysis considers software, and the human systems surrounding it, as the one and same [http://en.wikipedia.org/wiki/Complex_system complex system]. |
| | 1526 | |
| | 1527 | === Modular vs Monolithic === |
| | 1528 | |
| | 1529 | As we all know after a few years of experience, there is a constant tension in large scale software development between '''MODULARITY''' (tightly specified at points of coupling between structures) and '''MONOLITHICITY''' (loose or unspecified at points of coupling between structures) where structures include the human organisations which do the programming. This tradeoff is fundamental to the universe and appears throughout mathematics and physics because of how ''rigidity'' works: the choice between instability and rigidity is always a zero sum tradeoff, so you get to choose which parts of a system you would like to make flexible only by making other parts of the same system inflexible. |
| | 1530 | |
| | 1531 | Modular software has very strongly defined interface contracts between modules. This allows teams working within that module to change the module with any change ripples hopefully never propagating outside the module boundary, and therefore what one team does should not accidentally affect the work of another team if a module boundary is between them. As [http://en.wikipedia.org/wiki/Putnam_model effort required to modify software approximates its size cubed], for very large organisations this is an '''enormous''' productivity gain because it enables your development to occur as if you were working on a much smaller codebase, and [http://en.wikipedia.org/wiki/Component_Object_Model the most popular and successful modularity framework for C++ by far is Microsoft COM] (the latest iteration of which is called [http://en.wikipedia.org/wiki/Windows_Runtime Windows Runtime]). Unfortunately, very strictly defining coupling between modules has an enormous cost in what is allowed to traverse between modules, and even to this day nothing may pass through Microsoft COM/WinRT which cannot be perfectly represented in C, so no C++ feature not perfectly representable in C can traverse a COM boundary. |
| | 1532 | |
| | 1533 | Monolithic software has loose or unspecified (anything goes) interface contracts between parts. This lets programmers do what they want when they want, and that usually means a much more enjoyable experience for the programmer because of the feeling of power and control they get over the code (even if experience and self discipline mean they rarely exercise it). This approach encourages experimentation, fun, prototyping and more often than not is how volunteer-led open source software ends up being organised at a code level, partially because as a hobby you don't want to bother with all the boring compliance and worrying about other teams stuff you have to do at work. However just because an open source codebase may be monolithic internally doesn't mean that open source software is monolithic ''between'' codebase, if anything the extreme modularisation of cloud and web services into individual single purpose solutions supplied at a network socket which others mash together has been the truly defining innovation of the past twenty years of software. |
| | 1534 | |
| | 1535 | === C++ in the 21st century === |
| | 1536 | |
| | 1537 | As most younger C++ programmers, or indeed non-C++ programmers will ruefully note, C++ as a language and an ecosystem has almost entirely ignored the trends of the past twenty years towards modularity. The last big innovation in modularity was Microsoft COM back in 1993, and it formalised the C++ made available by [http://en.wikipedia.org/wiki/Cfront Cfront] because that was the most standardised implementation available in 1989 when COM was originally being proposed. To use Microsoft COM is therefore to restrict oneself to C++ as it was approximately in 1989. |
| | 1538 | |
| | 1539 | I don't know this for sure, but I suspect much of the recent investment by the tech majors in new systems programming languages ([http://en.wikipedia.org/wiki/Swift_%28programming_language%29 Swift] by Apple, [http://en.wikipedia.org/wiki/Go_%28programming_language%29 Go] by Google, and probably the biggest upcoming threat to C++, [http://en.wikipedia.org/wiki/Rust_%28programming_language%29 Rust] by Mozilla) stems from the continuing abject failure of C++ to finish supplanting C as the best general purpose systems language around. |
| | 1540 | |
| | 1541 | A lot of that continuing abject failure to finish supplanting C stems in my opinion from the followings causes: |
| | 1542 | |
| | 1543 | 1. '''A deliberate refusal by C++ to finish becoming a better systems glue language''' |
| | 1544 | |
| | 1545 | Investing in the development of a real modular component framework substantially improving on Microsoft COM (C++ 1z Modules is actually a build performance feature and has little to nothing to do with actual modularisation - this occurred due to repeated watering downs of the Modules proposal), standardising the C++ ABI, or even providing a C reflection library allowing C code to inspect some unknown C++ binary in a useful way, would go a long way to helping persuade language runtimes such as Python and the other new languages to use the C++ instead of C ABI. Indeed, such is the continuing failure of C++ as a community here that increasingly [http://llvm.org/docs/ExceptionHandling.html LLVM is becoming the next generation "better than C with just enough C++-esque support" ABI]. |
| | 1546 | |
| | 1547 | The thing which annoys me and many others is that [http://arxiv.org/abs/1405.3323 a much enhanced replacement for Microsoft COM is not technically challenging], it just requires someone to supply the sustained funding for the necessary two or three years to make it happen. This is exactly what [https://isocpp.org/about the Standard C++ Foundation] is supposedly there to fund, but I have observed almost zero interest from the C++ community to invest in becoming a better neighbour to other programming languages. I suspect, sadly, C++ will need to be mortally threatened by something like Rust to get its act together and get its head out of its ivory tower. |
| | 1548 | |
| | 1549 | 2. '''A tiredness and fear of disruptive change in the C++ thought leadership''' |
| | 1550 | |
| | 1551 | Internally C++ has undergone the wrenching change of getting the enormously delayed and overdue C++ 11 standard out of the door, and to achieve that the C++ leadership essentially had to beat itself up to such an extent that quite a number of the leading lights which were so historically influential in the leadership have since left C++ altogether either due to exhaustion, bitterness or their wives suggesting that divorce would be imminent unless they stopping putting C++ before their families. What I think has happened is that it has left the C++ leadership quite tired of disruptive change, and enormously defensive of any suggestion that they are not mostly correct on any given technical choice as I have often observed personally just on the Boost Developers mailing list, let alone at C++ conferences. People I know who attend WG21 meetings find them overwhelmingly negative and exhausting, and more than one has observed that the only kind of person who now thrives there is at best of a sociopathic disposition (and don't get me wrong, [http://uk.businessinsider.com/linux-foundation-reigns-in-linus-torvalds-2015-3?r=US the right kind of sociopath] even psychopath can be crucial to the success of any technically challenging and complex project, but for it to work you generally want at most just one sociopath in the room at once, collecting many of them into the same room never produces a positive outcome welcomed by all). Setting aside the consequences of the brain drain of good engineers forced out by the C++ process and culture, I think this was and is a great shame given what it could be instead, and I'll elaborate a bit more in the detailed treatment of what went wrong in the section on the Boost C++ libraries below. |
| | 1552 | |
| | 1553 | 3. '''A rejection of the business side of open source''' |
| | 1554 | |
| | 1555 | Most of the C++ leadership are - how shall I put it - of a certain age and job security not shared by younger C++ programmers or programmers in the newer languages. Back when they were younger programmers, open source software was something you did as a hobby, a charity and as a noble pursuit outside of normal work hours of achieving an excellence in your software that your employer wouldn't permit for business reasons. Employment was very secure, benefits and perks were high, and you had a strong chance of lifelong employment at the same employer if you so wished. Thanks to you working pretty much in the same field and technologies continuously, both within open source and within your employment your authority as an engineer had a good correlation with your years of experience which were many. You knew your position in the hierarchy of things, and you had worked and sacrificed to reach that position over a long career. Unsurprisingly, this generation of software engineer likes permanence, dislikes radical change, and really hates throwing away known-good code even when it is no longer fit for purpose. They also dislike the idea that one should make money from open source, and especially that one ought to ''leverage'' the dependency of firms on your free of cost open source software to extract rents (they call it blackmailing). |
| | 1556 | |
| | 1557 | The younger programmer, and increasingly the programmer not recognised as in the global engineering elite by the industry, has a very different experience and understanding of open source software. This is because all the big open source software projects invest heavily in the business side of acquiring and dispersing funding for vital work on the software, and this trend of seeing the open source software organisation as primarily one of actualising its business side instead of being just some central repo and web presence is the most prevalent in the newest open source projects simply through them being newer than older projects, and therefore adopting what they felt was the state of the art at the time. |
| | 1558 | |
| | 1559 | In case you are one of those engineers of a certain age and don't know what I'm talking about here, let me compare two open source projects: [http://www.boost.org/ the Boost C++ Libraries] and [https://www.djangoproject.com/ the Django web framework]. Django was first released in 2005, and it took just three years to establish [https://www.djangoproject.com/foundation/ the Django Software Foundation] which operates the business side of Django the open source software project, specifically as a charity which promotes, supports and advances the Django web framework by: |
| | 1560 | * Supports development of Django by sponsoring sprints, meetups, gatherings and community events. |
| | 1561 | * Promote the use of Django amongst the world-wide Web development community. |
| | 1562 | * Protect the intellectual property and the framework's long-term viability (i.e. invest in blue sky new development). |
| | 1563 | * Advance the state of the art in Web development. |
| | 1564 | In practice that means a constant cycle of acquiring and ''maintaining'' regular donations from the firms and users who use your free of cost open source software, whether from fee paying training courses and conferences, or simply through a donate button on the website, but often through investing effort to build networks and relationships with your biggest commercial users and ''leveraging'' those networks and relationships into a regular donations stream. Concomitant with that leveraging is a reverse leveraging by those sponsors on the future strategic direction of the open source project, so when this funding process is working well you the open source org gets funds to dispose on a future vision of the open source project previously agreed (or at least discussed) with your major sponsors. This is why leveraging your open source project for funding is not blackmail, despite what many in the C++ leadership might think. |
| | 1565 | |
| | 1566 | Now let's compare that to Boost which is probably the leading repository of open source C++ libraries. Boost was [http://www.boost.org/users/proposal.pdf originally conceived as a proving ground for new C++ standard library ideas] making better use of the 1998 C++ ISO standard, but from [https://web.archive.org/web/20000816132250/http://www.boost.org/libs/compiler_status.htm August 2000 onwards] it also began to turn into a broken compiler workarounds layer which whilst initially was great for those with broken compilers, it incurred an enormous technical debt into the codebase which still weighs heavily upon anyone trying to change anything substantial affecting more than one library, including cultural beliefs in the importance of support of broken toolsets due to those with a vested interest in the support of broken and legacy toolsets. Due to its roots as a playpen and proving ground for standard C++ libraries, Boost is unusually similar to the C++ standard library which is both good and bad: good in terms of the quality of design, implementation and testing -- bad in terms of monolithicity, poor coupling management, and over-reliance on a single person in charge of each library (great for the library if that maintainer is active, not so great across libraries when maintainers must work together, terrible if a maintainer vanishes or departs). Boost does not believe in there being a business side to itself apart from its annual conferences which began in 2006, and [https://sites.google.com/a/boost.org/steering/ its steering committee website] has this wonderful paragraph which was written after I caused a fuss about how little steering the steering committee does: |
| | 1567 | |
| | 1568 | ''"In the Boost community decisions have always been made by consensus and individual members have shown leadership by stepping forward to do what they felt needed to be done. Boost has not suffered from a lack of leadership or volunteer participation. It is not the role of the Steering Committee to inhibit this kind of spontaneous leadership and action, which has served Boost and the wider C++ community so well. On the contrary, it is the role of the Steering Committee to facilitate community-based leadership and decision making. The role of the Committee is to be able to commit the organization to specific action either where funds are required or where consensus cannot be reached, but a decision must be made."'' |
| | 1569 | |
| | 1570 | One instantly notices the big disclaimer of taking any strategic decisions whatsoever, unless vitally necessary, and the ability to approve funding for unsolicited proposals by the community. This effectively means that the Boost steering committee is really a board of trustees who have very little direct interaction with the thing they approve funding for, and that is pretty much what happens in practice. |
| | 1571 | |
| | 1572 | So what? one may now ask? Well, you've got to imagine what it is like trying to achieve anything in Django versus Boost. In Django, you pitch your idea for some change, whether radical or not, to a small, contained authoritative leadership where the power to decide and fund an initiative is clearly demarcated. They more often or not are acting really as a filter for appropriately repackaging and presenting ideas to the sponsors for funding, though they usually have some slush money around for the really radical stuff (if it is cheap). The process for enacting change is therefore extremely well specified, and moreover, it actually happens. |
| | 1573 | |
| | 1574 | Compare this to Boost. Firstly you must achieve consensus in the community, which involves persuading a majority on boost-dev -- or rather, persuading a majority of those who ''respond'' on boost-dev that what you proposed is not a terrible idea, often by you investing dozens of hours of your free unpaid time into some prototype you can show people. This part also usually involves you fending off trolls, those with chips on their shoulder, those with vested interests, those threatened by any form of change etc. It also usually involves ''years'' of campaigning, because to get a dispersed heterogeneous community with few common interests to reach consensus on some breaking change even with a proven working prototype takes a minimum of a year, and usually many years. If this begins to sound as dis-spiriting and as exhausting as attending WG21 meetings -- except this effort is entirely unpaid -- you would be right. Anyway, if after years of unpaid effort and toil you finally reach consensus, ''then'' you can apply to the steering committee for funding to implement and deliver your change, by which stage -- to be blunt -- the funding is but a tiny fraction of the time, blood and sweat you have personally already invested. |
| | 1575 | |
| | 1576 | |
| | 1577 | TODO AFTER THIS |
