Boost C++ Libraries: Ticket #4842: "pure virtual method called; terminate called without an active exception" on shutdown

attachment set

jan.boehme@… — Wed, 24 Nov 2010 11:32:13 GMT

attachment → ticket4842.tar.bz2

sample project to reproduce the problem

jan.boehme@… — Wed, 24 Nov 2010 11:43:53 GMT

I attached a packed project containing a Jamroot defining two rules, one to show the crash at the mentioned location, another which works around the problem by changing the libraries link order. Of course a workaround like this isn't appropriate for real projects. This sample should help to identify the problems background. I think it's because a dll was already unloaded which brings a later delete of a belonging extended_type_info into trouble.

Robert Ramey — Thu, 25 Nov 2010 17:02:33 GMT

Can you verify that DLLS are unloaded in the reverse sequence of that which they are loaded?

Are they loaded/unloaded explicitly or automatically via import libraries?

Robert Ramey

jan,boehme@… — Thu, 25 Nov 2010 20:01:36 GMT

Sorry, but I have no chance to verify the DLL load/unload sequence as I can't place debug code inside this automatic mechanism.

As you can see in the sample they are loaded automatically.

BTW this is an only UNIX issue, which occurs since the so called "Bogdan fix" which in turn solved similar problems under Windows.

Jan

Robert Ramey — Mon, 29 Nov 2010 20:40:24 GMT

What is the "Bogdan fix"?

jan.boehme@… — Tue, 30 Nov 2010 08:19:05 GMT

Sorry, it contains all changes that were done regarding the following topic:

http://groups.google.com/group/boost-list/browse_thread/thread/845fc694f99b9c32?pli=1

I don't have a clue which revisions belong to it.

Lately I asked Bogdan if the issue was finally solved and he confirmed release 1.44 brought the fix.

status changed; resolution set

Robert Ramey — Tue, 14 Dec 2010 17:53:12 GMT

status new → closed
resolution → fixed

ruediger.berlich@… — Tue, 21 Dec 2010 16:17:58 GMT

Replying to ramey:Has this fix already been committed ? I am still getting the same error with trunk release 67387.

status changed; resolution deleted

jan.boehme@… — Wed, 12 Jan 2011 16:41:09 GMT

status closed → reopened
resolution fixed

Could you explain why this ticket was closed. Was it classified to be a problem not related to the library itself?

Please remember, a bug similar to this was fixed for Windows (comment 5), the Unix problem obviously still exists.

Robert Ramey — Wed, 12 Jan 2011 17:14:11 GMT

I couldn't reproduce it and I just assumed it was fixed as indicated in the comment. Not having any other information, I marked it as fixed.

Robert Ramey

status changed; resolution set

Robert Ramey — Sun, 23 Jan 2011 02:40:17 GMT

status reopened → closed
resolution → fixed

status changed; resolution deleted

ruediger.berlich@… — Tue, 25 Jan 2011 21:36:14 GMT

status closed → reopened
resolution fixed

Hi there, the problem still exists for me with the current trunk version. I have seen a similar message on the Boost.Build mailing list, which appears to relate to the same problem (although few details are given). So I assume that it is not just me. See the message "Pure virtual method called" on Boost.Build, posted on January 13, 2011 . Best Regards, Ruediger

ruediger.berlich@… — Thu, 27 Jan 2011 14:23:24 GMT

Hi again, an additional piece of information: I'm observing the same problem ("pure virtual method called") with CentOS 5.5 64 bit, g++ 4.1.2 (using Boost 1.45). So I'd wager that this is a general problem on Linux systems. Best Regards, Ruediger

Robert Ramey — Thu, 27 Jan 2011 19:03:42 GMT

OK - just to recap -

On which environments (os, compiler, shared/static library, debug vs release etc.

So far I see linux, g++ 4.1.2 - 64 bit - no information regarding shared/static, debug/release, etc.

Feel free to add to the above list. It would also be helpful to attempt to trim down the test case. Often times that can give information about the source of the problem.

I am interested in addressing this and I do believe it's a real problem. It's just that I can't do it with the little information I have (and no reasonable test case)

Robert Ramey

ruediger.berlich@… — Thu, 27 Jan 2011 22:21:34 GMT

O.k., I'll try to be more specific:

I'm seeing this behaviour on three systems:

Kubuntu Linux 10.10, 64 bit, g++ 4.4.5, Boost 1.44, 1.45 and current trunk version/pre-beta of 1.46
CentOS 5.5, 64 bit, g++ 4.1.2, tested only with Boost 1.45
I have in the past seen similar behaviour on Scientific Linux 5, which is however close to CentOS

The problem does not happen with Boost 1.40 - 1.43 .

I'd be happy to try other (Linux-)distributions you suggest. My code is so far "Linux only", so I cannot try other Operating Systems.

I experience the problem in conjunction with our Geneva library collection (or its demo applications, to be more exact). The library is available on http://www.launchpad.net/geneva. I'm using the current trunk version of Geneva, which can be downloaded via the Bazaar version control system, e.g. with the command "bzr co lp:geneva ." .

I'm always linking dynamically, and the problem happens both in Debug and Release mode.

There are two libraries which involve serialization (libgemfony-geneva.so and libgemfony-geneva-individuals.so) and which get linked into all of the demo applications. I am not loading any libraries manually, but rather rely on the dynamic linker of the operating system, which loads the libraries upon program startup, likely in the order in which they were specified during the linking stage. I have tried reversing the order of the libraries in my CMake build scripts, but to no avail (it is possible that CMake has some built-in magic which always enforces a certain order). Not all of my demo applications are affected. I am currently trying to find out, which change of conditions make the problem go away. Will say so here, if I find a solution.

Trimming down Jan Boehme's test case: I have tried to reproduce the problem with his code. As I am not a "bjam person", I have however written a CMake build script for his code. Using this, however, the problem cannot be reproduced in my main environment (Kubuntu 10.10, the same, where Geneva fails with the aforementioned "Pure virtual method called").

And here is an offer: If you have a means of receiving a file of maybe 2 GB, I'd be happy to set up a VMWare virtual machine (or any other VM that is free and runs both on Linux and Windows), with Kubuntu and my library installed, and send it to you. This way you could see for yourself what is happening.

I intend to come to BoostCon so that, if the problem still exists then, I'd be happy to work with you on a solution (assuming you will be there).

Hope that helps. I'll try to respond on short notice, if you have further questions. In the meantime, I will try to reproduce the problem with Jan's code.

Robert Ramey — Sun, 30 Jan 2011 18:25:34 GMT

Here's an idea:

recompile void_cast.cpp with the compile time defined BOOST_SERIALIZATION_LOG

This will emit output to standard error as different static object are created and destroyed. It is one way I use to track down these types of problems

Robert Ramey

ruediger.berlich@… — Mon, 31 Jan 2011 20:31:56 GMT

Hi Robert, here are the last few lines of the output of one of my examples ("G01Optimizer"), at the end of the program run. The total output has approx. 2000 lines, but doesn't look much different from what you see below:

begin

[...]

N3Gem6Geneva6GSwarmE<-N3Gem6Geneva12GMutableSetTINS0_13GParameterSetEEE recursive_unregister N3Gem6Geneva6GSwarmE<-N3Gem6Geneva23GOptimizationAlgorithmTINS0_13GParameterSetEEE recursive_unregister N3Gem6Geneva6GSwarm25GSwarmOptimizationMonitorE<-N3Gem6Geneva23GOptimizationAlgorithmTINS0_13GParameterSetEE21GOptimizationMonitorTE recursive_unregister N3Gem6Geneva18GPersonalityTraitsE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva23GSwarmPersonalityTraitsE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva23GSwarmPersonalityTraitsE<-N3Gem6Geneva18GPersonalityTraitsE recursive_unregister N9boost_1326detail20sp_counted_base_implIPN3Gem6Geneva14GParameterBaseEN5boost13serialization12null_deleterEEE<-N9boost_1326detail15sp_counted_baseE recursive_unregister N9boost_1326detail20sp_counted_base_implIPN3Gem6Geneva18GPersonalityTraitsEN5boost13serialization12null_deleterEEE<-N9boost_1326detail15sp_counted_baseE recursive_unregister N3Gem6Geneva11GIndividualE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem5Tests16GTestIndividual1E<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva13GParameterSetE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva12GMutableSetTINS0_14GParameterBaseEEE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem5Tests16GDelayIndividualE<-pure virtual method called terminate called without an active exception Abgebrochen

end

"Abgebrochen" means "Terminated". The example uses GFunctionIndividual. "GDelayIndividual", which appears to be the point during whose unregistration the crash happens, is never explicitly instantiated by my program. GDelayIndividual's code is located in a separate library, together with "GFunctionIndividual". It is linked to the application, and there is another dynamic library using Serialization services, which is also linked to the code.

I have used Beta 1 of Boost 1.46, the platform is Ubuntu Linux 64 bit, g++ 4.4.5. Boost and my own code have been compiled in Debug mode.

ruediger.berlich@… — Tue, 01 Feb 2011 14:13:58 GMT

Here is more information: Just to be sure that the problem is not related to GDelayIndividual, I have removed that class from the library. The error still happens, albeit when unregistering another class. Thus I am quite sure it is unrelated to my code.

cc changed

crispin.boylan@… — Wed, 23 Feb 2011 08:26:22 GMT

cc crispin.boylan@… added

crispin.boylan@… — Wed, 23 Feb 2011 08:40:16 GMT

i too see this problem on mandriva with 64 bit gcc 4.5.2. only when my program is compiled in release mode though. The boost libs are in release mode in both cases.

anonymous — Wed, 16 Mar 2011 13:20:48 GMT

Hi,

Looks that I'm also experiencing this problem, my system is:

2.6.36-gentoo-r5 SMP x86_64 Intel(R) Core(TM)2 Quad CPU Q8200 @ 2.33GHz,
g++ (Gentoo 4.4.5 p1.2, pie-0.4.5) 4.4.5,
boost_1_46_0

Problem did not happen on boost_1_41, and started right after upgrading the boost (without any other code changes).

Backtrace is:

Program received signal SIGABRT, Aborted. 0x00007fffe4d80165 in raise () from /lib/libc.so.6 (gdb) bt #0 0x00007fffe4d80165 in raise () from /lib/libc.so.6 #1 0x00007fffe4d81580 in abort () from /lib/libc.so.6 #2 0x00007fffe538b695 in gnu_cxx::verbose_terminate_handler() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #3 0x00007fffe5389ac6 in cxxabiv1::terminate(void (*)()) () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #4 0x00007fffe5389af3 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #5 0x00007fffe538a3cf in cxa_pure_virtual () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #6 0x00007ffff3f93e66 in boost::serialization::void_cast_detail::void_caster::operator<(boost::serialization::void_cast_detail::void_caster const&) const ()

from /Data/See3D/oncomorph/2.6.36-gentoo-r5/3rdParty/boost/lib/libboost_serialization.so.1.46.0

#7 0x00007ffff3f9477d in boost::serialization::void_cast_detail::void_caster::recursive_unregister() const ()

from /Data/See3D/oncomorph/2.6.36-gentoo-r5/3rdParty/boost/lib/libboost_serialization.so.1.46.0

#8 0x00007ffff7930cc9 in boost::serialization::void_cast_detail::void_caster_virtual_base<Oncomorph::Data::SliceId, Oncomorph::Persistence::Serializable>::~void_caster_virtual_base() () from /Data/See3D/oncomorph/2.6.36-gentoo-r5/lib/liboncomorph_base.so #9 0x00007ffff7930e38 in boost::serialization::detail::singleton_wrapper<boost::serialization::void_cast_detail::void_caster_virtual_base<Oncomorph::Data::SliceId, Oncomorph::Persistence::Serializable> >::~singleton_wrapper() () from /Data/See3D/oncomorph/2.6.36-gentoo-r5/lib/liboncomorph_base.so #10 0x00007fffe4d82f55 in cxa_finalize () from /lib/libc.so.6 #11 0x00007ffff7912dc6 in ?? () from /Data/See3D/oncomorph/2.6.36-gentoo-r5/lib/liboncomorph_base.so #12 0x0000000000000092 in ?? () #13 0x0000000000000000 in ?? ()

thanks Julle Juntunen

attachment set

Sat, 19 Mar 2011 00:09:47 GMT

attachment → void_cast.cpp.patch

Patch to fix the issue.

Sat, 19 Mar 2011 00:14:22 GMT

This issue affects not only *nix shared libraries, but also Windows DLLs. I have attached a patch that fixes this issue. However, there are still situations where this can break, which I will discuss.

The change that caused this regression was in void_caster::recursive_unregister(). It does a loop over the entire set of registered void casters to remove all related void casters to the one being unregistered. In boost 1.43 and prior, it used to compare the pointer in the iterator to this within the loop to remove itself. However, in the boost 1.44 release that was changed to doing a find and erase after the loop. Since there are void_caster objects in the map with invalid extended type infos, that find caused a pure virtual function call.

I think the reason why it has invalid extended type infos in the map is due to how it registers the void caster shortcuts. In void_caster::recursive_register(), it will grab the extended type info from related void_casters, but those objects could reside in another library. Since only the current void caster being registered is marked as the parent, and not the other one it gets the other extended type info from, the other extended type info can be destructed before all the void_casters that use it are unregistered.

Though my patch, which restores the previous behavior, works around the issue during normal program startup and shutdown, if you dynamically load two libraries then later unload the first one, leaving the second library loaded, you can potentially have invalid void casters left behind to cause a crash the next lookup. Unfortunately, getting this to work properly will be very tricky, since if you simply unregister the void caster in this case, the void cast itself may still be valid, you just need a different extended type info instance to handle the comparison. I see two potential paths you can take to fix this: either set up some sort of system to replace extended type infos being destructed with valid equivalents or remove the shortcuts and use recursive lookups instead.

anonymous — Sat, 19 Mar 2011 06:40:45 GMT

I spent a little time looking at this.

I looked at the stack traces (and log output) from a couple of different reports. The traces I see look like:

~void_caster

recurrsive_unregister

operator<

Or some variation - all ending with operator<

operator< compares the dereferenced pointers to extended_type_info.

I couldn't see any reason why this should occur with void_caster. I'm thinking the issue would be with extended_type_info. recurrsive_unregister invokes operator< on dereferenced pointers to extended_type_info objects.

I'm concerned that there may be multilple such objects in the map. This would occur if one included code in both the DLL and the mainline. There is a trap to detect this because I was concerned about just this problem. I had to disable this trap because it required more discipline in code organization than many could implement. I would be curious to know if those who have experienced this problem were to re-enable this trap if it would catch this issue.

This hypothesis would also explain why the situation isn't as common as I would expect it to be if were something else.

This trap is found and explained at basic_serializer_map.cpp line 46. Try these test cases with the following:

a) uncomment the code at line 63 in basic_serializer_map.cp b) rebuild the library c) rerun the problem program d) let me know the results.

I realize that this is a lot of work. But without this kind of effort, it's very difficult to make progress here. I thank all those who have invested efforts here and appreciate your patience and help.

Robert Ramey

Sat, 19 Mar 2011 20:27:47 GMT

Though I can't enable that trap right now, I can say without a doubt that it would be hit in my case. I have encountered other issues due to serializers being instantiated in multiple libraries, which I provided a patch for in ticket #5341. Unfortunately, in my use case it would be nearly impossible to avoid serializing across shared library boundaries, and I'm sure that's the case for a number of other users as well.

After looking at void_cast.cpp for a while and thinking about different situations where there are multiple instances of void_casters, I think that I have found a situation that would cause a crash. This situation can also cause a crash if everything is statically linked.

First, I will point out an issue with the current deregistration method. The first void caster that is constructed for a particular type relation will be inserted into the set, and any duplicates will be ignored. However, since it's doing a find and remove in recursive_unregister, the frist void caster to be destructed will remove whichever void caster was registered for that type relation, since it's comparing the type infos. For example, if you have void casters A and B, which both represent the same base and derived types, A will be registered on construction, while B will be ignored. On destruction, B will be destructed first and remove A. With the previous behavior, it was comparing pointers, so B wouldn't find itself in the set, and A wouldn't be unregistered until A was destructed.

I will now lay out a particular case where this unregistration behavior can cause a crash. Lets say you have a class hierarchy of A, B, C, and D, where A is the base class and D is the most derived class. If void casters are registered for A->B, B->C, and C->D, shortcuts will be made along the way. Depending on the order of initialization, such as if C->D is registered before B->C, those shortcuts can then then recursively create shortcuts of their own. In this example, let's say the shortcut B->D recursively creates the shortcut A->D, so the parent of A->D is the shortcut for B->D.

Now we have our hierarchy of void casters set up. However, let's say somewhere else in code the void caster B->D is registered directly, and is constructed after all the void casters and resulting shortcuts above. That void caster will see the B->D shortcut registered already, and won't insert it into the set. When the program shuts down, that explicit B->D void caster will be destructed first, since it was constructed last. When that void caster is destructed, recursive_unregister will do a find for the B->D type relation, and end up removing the shortcut registered earlier. That shortcut is merely removed from the set, and is never actually deleted, so it is leaked. As a result of that, the A->D shortcut created by the B->D shortcut is never removed from the set, since its parent's destructor is never called and is never recursively unregistered. At some point, the type infos for A and D are destroyed, but that A->D shortcut that references them is still left in the set. If there are still void casters registered at that point, the next time a find is performed, it will crash with the pure virtual function call.

Since order of construction at startup isn't guaranteed, this specific ordering can occur regardless of if shared libraries are used or not. This will generally occur if you are serializing polymorphic type hierarchies by base pointer, as that serialization will register an explicit void caster that can skip over multiple classes in the hierarchy and conflict with an existing shortcut. I think there has been a correlation with the use of shared libraries and experiencing this bug since projects with large and complex type hierarchies are more likely to use shared libraries.

ruediger.berlich@… — Sun, 20 Mar 2011 09:54:42 GMT

Hi Robert,

Replying to anonymous:

a) uncomment the code at line 63 in basic_serializer_map.cp b) rebuild the library c) rerun the problem program d) let me know the results.

I did this and there is no change at all (i.e. the trap doesn't trigger, program runs through as normal and crashes at the end with message "pure virtual method called").

Best Regards, Ruediger

ruediger.berlich@… — Sun, 20 Mar 2011 10:11:59 GMT

Replying to ruediger.berlich@…:

I did this and there is no change at all (i.e. the trap doesn't trigger, program runs through as normal and crashes at the end with message "pure virtual method called").

I have now also tried Aaron's patch (with Robert's trap still active).

For all programs I have tried it fixes the problem ...

This was done on Ubuntu Linux 10.10 / 64 bit, g++ 4.4.5 .

Note that the Geneva library is a toolkit, so I have to run further tests to see whether this cures the problem in all our usage scenarios. I am travelling at the moment, so I can only run further tests when I am back.

Best Regards, Ruediger

anonymous — Sun, 20 Mar 2011 10:16:03 GMT

Replying to ruediger.berlich@…:

This was done on Ubuntu Linux 10.10 / 64 bit, g++ 4.4.5 .

Apologies for repeatedly replying to myself. But I guess it is important to note that I have used Boost 1.46.1 for the tests above. I am not linking statically, and serialized classes come from different dynamic libraries in my test programs.

Best Regards, Ruediger

anonymous — Mon, 21 Mar 2011 12:09:02 GMT

Hi Robert, Aaron

Replying to anonymous:

a) uncomment the code at line 63 in basic_serializer_map.cp b) rebuild the library c) rerun the problem program d) let me know the results.

I tried this but the problem occurs just as before.

The patch by Aaron, however, fixes the problem.

br.
Julle Juntunen

status changed; resolution set

Robert Ramey — Sun, 10 Apr 2011 00:00:46 GMT

status reopened → closed
resolution → fixed

I've tested Aaron's patch and committed it to the trunk. I'm taking everyone's word for it that this is the fix as I can't reproduce it myself.

Robert Ramey

Bryce Adelstein Lelbach — Wed, 11 May 2011 02:41:44 GMT

Ramey, for whatever peace of mind it's worth, Aaron's patch fixed a serious issue in the codebase which Hartmut and I work on. Thanks for applying, this resolves a big headache for me.