"pure virtual method called; terminate called without an active exception" on shutdown

[Eric Niebler]

On 11/11/2010 6:32 AM, Ruediger Berlich wrote:

Hi again,

here is an update: I have now also tested the behaviour below ("pure virtual method called; terminate called without an active exception" -> related to Boost.Serialization) with Boost 1.44 and see the same problem. This does not happen with 1.43.

Anyway, I thought it might be important to know that whatever causes this wasn't introduced with 1.45 but 1.44 (and of course I might be doing something wrong which only triggers now due to 1.44's changes in the serialization library).

Thanks and Best Regards, Ruediger

Ruediger Berlich wrote:

Dear all, here are a few experiences with the beta:

Beman Dawes wrote: [...]

Please download the beta, give it a try, and report any problems you encounter.

Platform: ========= Kubuntu 10.10 64 bit, g++ 4.4.5

Compilation: ============ Runs mostly smoothely, albeit with many warnings of the type:

[...] libs/program_options/src/parsers.cpp:233: instantiated from here ./boost/function/function_base.hpp:321: warning: dereferencing type-punned pointer will break strict-aliasing rules ./boost/function/function_base.hpp:325: warning: dereferencing type-punned pointer will break strict-aliasing rules [...]

This is not a new situation and has been there with many prior versions. It happens in different components.

Compiling a complex application =============================== (The Geneva library's trunk version, close to version 0.85, see ​http://launchpad.net/geneva ; depends on probably a dozen different boost libraries)

In order to get it to compile I hat to make a single change: I had to add "#include <boost/serialization/nvp.hpp>" prior to the inclusion of the date_time libraries in a single file (out of some 65 headers, some of which also include date_time), or else I would get messages of the type

/opt/boost145/include/boost/date_time/gregorian/greg_serialize.hpp: In function ‘void boost::serialization::save(Archive&, const boost::gregorian::date&, unsigned int)’: /opt/boost145/include/boost/date_time/gregorian/greg_serialize.hpp:58: error: there are no arguments to ‘make_nvp’ that depend on a template parameter, so a declaration of ‘make_nvp’ must be available

I _do_ serialize date_time objects in my application. This is new with Boost 1.45 (compared with 1.43).

Running the application =======================

When running the application in serial. multithreaded or networked mode (Geneva does distributed parametric optimization, but also allows multithreaded optimization and has a serial mode for debugging), I get the error:

[...] 1000: 4.99999977648258e-05 End of optimization reached Done ... pure virtual method called terminate called without an active exception Aborted

"Done" is printed at the end of main(), so there are only some singletons left to clean up, one of which stems from the Boost.Serialization library, AFAIK.

Running the application in gdb yields:

// (gdb) up #1 0x00007ffff40726b0 in abort () at abort.c:92 92 abort.c: No such file or directory.

in abort.c

(gdb) up #2 0x00007ffff49126bd in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6 (gdb) up #3 0x00007ffff4910906 in ?? () from /usr/lib/libstdc++.so.6 (gdb) up #4 0x00007ffff4910933 in std::terminate() () from /usr/lib/libstdc++.so.6 (gdb) up #5 0x00007ffff491128f in __cxa_pure_virtual () from #/usr/lib/libstdc++.so.6 (gdb) up #6 0x00007ffff4fc1f16 in boost::serialization::void_cast_detail::void_caster::operator<(boost::serialization::void_cast_detail::void_caster const&) const () from /opt/boost145/lib/libboost_serialization.so.1.45.0 (gdb) up #7 0x00007ffff4fc282d in boost::serialization::void_cast_detail::void_caster::recursive_unregister() const

() from /opt/boost145/lib/libboost_serialization.so.1.45.0

(gdb) up #8 0x00007ffff4fc2f27 in boost::serialization::void_cast_detail::void_caster_shortcut::~void_caster_shortcut()

() from /opt/boost145/lib/libboost_serialization.so.1.45.0

(gdb) up #9 0x00007ffff4fc289f in boost::serialization::void_cast_detail::void_caster::recursive_unregister() const

() from /opt/boost145/lib/libboost_serialization.so.1.45.0

(gdb) up #10 0x00007ffff72ad64f in boost::serialization::void_cast_detail::void_caster_primitive<Gem::Geneva::GPersonalityTraits, Gem::Geneva::GObject>::~void_caster_primitive (this=0x7ffff7bb7e80,

__in_chrg=<value optimized out>) at

/opt/boost145/include/boost/serialization/void_cast.hpp:198 198 recursive_unregister(); (gdb) up #11 0x00007ffff72ad7be in boost::serialization::detail::singleton_wrapper<boost::serialization::void_cast_detail::void_caster_primitive<Gem::Geneva::GPersonalityTraits, Gem::Geneva::GObject>::~singleton_wrapper (

this=0x7ffff7bb7e80, __in_chrg=<value optimized out>) at /opt/boost145/include/boost/serialization/singleton.hpp:111

111 m_is_destroyed = true; (gdb) up #12 0x00007ffff40748c0 in __cxa_finalize (d=0x7ffff7baa2d0) at cxa_finalize.c:56 56 cxa_finalize.c: No such file or directory.

in cxa_finalize.c

(gdb) up #13 0x00007ffff7016366 in __do_global_dtors_aux ()

from /home/rberlich/geneva-build/src/geneva/libgemfony-geneva.so.0.8.5rc0

(gdb) up #14 0x0000000000000000 in ?? () (gdb) up Initial frame selected; you cannot go up. //

Gem::Geneva::GPersonalityTraits is a purely virtual class, marked as such with the following code: "BOOST_SERIALIZATION_ASSUME_ABSTRACT(Gem::Geneva::GPersonalityTraits)" . GObject is the base class of all optimization-related classes.

The exact same code compiles and runs fine with Boost 1.43, the last version I have been using excessively. I cannot really comment on Boost 1.44 , but would be happy to try if it helps.

Note that the above problem does not seem to have any influence on the results of the program. The problems only appear at the very end of the execution, when various singletons get destroyed.

Best Regards, Ruediger

[jan.boehme@…]

sample project to reproduce the problem

[jan.boehme@…]

I attached a packed project containing a Jamroot defining two rules, one to show the crash at the mentioned location, another which works around the problem by changing the libraries link order. Of course a workaround like this isn't appropriate for real projects. This sample should help to identify the problems background. I think it's because a dll was already unloaded which brings a later delete of a belonging extended_type_info into trouble.

[Robert Ramey]

Can you verify that DLLS are unloaded in the reverse sequence of that which they are loaded?

Are they loaded/unloaded explicitly or automatically via import libraries?

Robert Ramey

[jan,boehme@…]

Sorry, but I have no chance to verify the DLL load/unload sequence as I can't place debug code inside this automatic mechanism.

As you can see in the sample they are loaded automatically.

BTW this is an only UNIX issue, which occurs since the so called "Bogdan fix" which in turn solved similar problems under Windows.

Jan

[Robert Ramey]

What is the "Bogdan fix"?

[jan.boehme@…]

Sorry, it contains all changes that were done regarding the following topic:

​http://groups.google.com/group/boost-list/browse_thread/thread/845fc694f99b9c32?pli=1

I don't have a clue which revisions belong to it.

Lately I asked Bogdan if the issue was finally solved and he confirmed release 1.44 brought the fix.

[ruediger.berlich@…]

Replying to ramey:Has this fix already been committed ? I am still getting the same error with trunk release 67387.

[jan.boehme@…]

Could you explain why this ticket was closed. Was it classified to be a problem not related to the library itself?

Please remember, a bug similar to this was fixed for Windows (comment 5), the Unix problem obviously still exists.

[Robert Ramey]

I couldn't reproduce it and I just assumed it was fixed as indicated in the comment. Not having any other information, I marked it as fixed.

Robert Ramey

[ruediger.berlich@…]

Hi there, the problem still exists for me with the current trunk version. I have seen a similar message on the Boost.Build mailing list, which appears to relate to the same problem (although few details are given). So I assume that it is not just me. See the message "Pure virtual method called" on Boost.Build, posted on January 13, 2011 . Best Regards, Ruediger

[ruediger.berlich@…]

Hi again, an additional piece of information: I'm observing the same problem ("pure virtual method called") with CentOS 5.5 64 bit, g++ 4.1.2 (using Boost 1.45). So I'd wager that this is a general problem on Linux systems. Best Regards, Ruediger

[Robert Ramey]

OK - just to recap -

On which environments (os, compiler, shared/static library, debug vs release etc.

So far I see linux, g++ 4.1.2 - 64 bit - no information regarding shared/static, debug/release, etc.

Feel free to add to the above list. It would also be helpful to attempt to trim down the test case. Often times that can give information about the source of the problem.

I am interested in addressing this and I do believe it's a real problem. It's just that I can't do it with the little information I have (and no reasonable test case)

Robert Ramey

[ruediger.berlich@…]

O.k., I'll try to be more specific:

I'm seeing this behaviour on three systems:

• Kubuntu Linux 10.10, 64 bit, g++ 4.4.5, Boost 1.44, 1.45 and current trunk version/pre-beta of 1.46
• CentOS 5.5, 64 bit, g++ 4.1.2, tested only with Boost 1.45
• I have in the past seen similar behaviour on Scientific Linux 5, which is however close to CentOS

The problem does not happen with Boost 1.40 - 1.43 .

I'd be happy to try other (Linux-)distributions you suggest. My code is so far "Linux only", so I cannot try other Operating Systems.

I experience the problem in conjunction with our Geneva library collection (or its demo applications, to be more exact). The library is available on ​http://www.launchpad.net/geneva. I'm using the current trunk version of Geneva, which can be downloaded via the Bazaar version control system, e.g. with the command "bzr co lp:geneva ." .

I'm always linking dynamically, and the problem happens both in Debug and Release mode.

There are two libraries which involve serialization (libgemfony-geneva.so and libgemfony-geneva-individuals.so) and which get linked into all of the demo applications. I am not loading any libraries manually, but rather rely on the dynamic linker of the operating system, which loads the libraries upon program startup, likely in the order in which they were specified during the linking stage. I have tried reversing the order of the libraries in my CMake build scripts, but to no avail (it is possible that CMake has some built-in magic which always enforces a certain order). Not all of my demo applications are affected. I am currently trying to find out, which change of conditions make the problem go away. Will say so here, if I find a solution.

Trimming down Jan Boehme's test case: I have tried to reproduce the problem with his code. As I am not a "bjam person", I have however written a CMake build script for his code. Using this, however, the problem cannot be reproduced in my main environment (Kubuntu 10.10, the same, where Geneva fails with the aforementioned "Pure virtual method called").

And here is an offer: If you have a means of receiving a file of maybe 2 GB, I'd be happy to set up a VMWare virtual machine (or any other VM that is free and runs both on Linux and Windows), with Kubuntu and my library installed, and send it to you. This way you could see for yourself what is happening.

I intend to come to BoostCon so that, if the problem still exists then, I'd be happy to work with you on a solution (assuming you will be there).

Hope that helps. I'll try to respond on short notice, if you have further questions. In the meantime, I will try to reproduce the problem with Jan's code.

[Robert Ramey]

Here's an idea:

recompile void_cast.cpp with the compile time defined BOOST_SERIALIZATION_LOG

This will emit output to standard error as different static object are created and destroyed. It is one way I use to track down these types of problems

Robert Ramey

[ruediger.berlich@…]

Hi Robert, here are the last few lines of the output of one of my examples ("G01Optimizer"), at the end of the program run. The total output has approx. 2000 lines, but doesn't look much different from what you see below:

begin

[...]

N3Gem6Geneva6GSwarmE<-N3Gem6Geneva12GMutableSetTINS0_13GParameterSetEEE recursive_unregister N3Gem6Geneva6GSwarmE<-N3Gem6Geneva23GOptimizationAlgorithmTINS0_13GParameterSetEEE recursive_unregister N3Gem6Geneva6GSwarm25GSwarmOptimizationMonitorE<-N3Gem6Geneva23GOptimizationAlgorithmTINS0_13GParameterSetEE21GOptimizationMonitorTE recursive_unregister N3Gem6Geneva18GPersonalityTraitsE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva23GSwarmPersonalityTraitsE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva23GSwarmPersonalityTraitsE<-N3Gem6Geneva18GPersonalityTraitsE recursive_unregister N9boost_1326detail20sp_counted_base_implIPN3Gem6Geneva14GParameterBaseEN5boost13serialization12null_deleterEEE<-N9boost_1326detail15sp_counted_baseE recursive_unregister N9boost_1326detail20sp_counted_base_implIPN3Gem6Geneva18GPersonalityTraitsEN5boost13serialization12null_deleterEEE<-N9boost_1326detail15sp_counted_baseE recursive_unregister N3Gem6Geneva11GIndividualE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem5Tests16GTestIndividual1E<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva13GParameterSetE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem6Geneva12GMutableSetTINS0_14GParameterBaseEEE<-N3Gem6Geneva7GObjectE recursive_unregister N3Gem5Tests16GDelayIndividualE<-pure virtual method called terminate called without an active exception Abgebrochen

end

"Abgebrochen" means "Terminated". The example uses GFunctionIndividual. "GDelayIndividual", which appears to be the point during whose unregistration the crash happens, is never explicitly instantiated by my program. GDelayIndividual's code is located in a separate library, together with "GFunctionIndividual". It is linked to the application, and there is another dynamic library using Serialization services, which is also linked to the code.

I have used Beta 1 of Boost 1.46, the platform is Ubuntu Linux 64 bit, g++ 4.4.5. Boost and my own code have been compiled in Debug mode.

[ruediger.berlich@…]

Here is more information: Just to be sure that the problem is not related to GDelayIndividual, I have removed that class from the library. The error still happens, albeit when unregistering another class. Thus I am quite sure it is unrelated to my code.

[crispin.boylan@…]

i too see this problem on mandriva with 64 bit gcc 4.5.2. only when my program is compiled in release mode though. The boost libs are in release mode in both cases.

[anonymous]

Hi,

Looks that I'm also experiencing this problem, my system is:

• 2.6.36-gentoo-r5 SMP x86_64 Intel(R) Core(TM)2 Quad CPU Q8200 @ 2.33GHz,
• g++ (Gentoo 4.4.5 p1.2, pie-0.4.5) 4.4.5,
• boost_1_46_0

Problem did not happen on boost_1_41, and started right after upgrading the boost (without any other code changes).

Backtrace is:

Program received signal SIGABRT, Aborted. 0x00007fffe4d80165 in raise () from /lib/libc.so.6 (gdb) bt #0 0x00007fffe4d80165 in raise () from /lib/libc.so.6 #1 0x00007fffe4d81580 in abort () from /lib/libc.so.6 #2 0x00007fffe538b695 in gnu_cxx::verbose_terminate_handler() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #3 0x00007fffe5389ac6 in cxxabiv1::terminate(void (*)()) () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #4 0x00007fffe5389af3 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #5 0x00007fffe538a3cf in cxa_pure_virtual () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/libstdc++.so.6 #6 0x00007ffff3f93e66 in boost::serialization::void_cast_detail::void_caster::operator<(boost::serialization::void_cast_detail::void_caster const&) const ()

from /Data/See3D/oncomorph/2.6.36-gentoo-r5/3rdParty/boost/lib/libboost_serialization.so.1.46.0

#7 0x00007ffff3f9477d in boost::serialization::void_cast_detail::void_caster::recursive_unregister() const ()

from /Data/See3D/oncomorph/2.6.36-gentoo-r5/3rdParty/boost/lib/libboost_serialization.so.1.46.0

#8 0x00007ffff7930cc9 in boost::serialization::void_cast_detail::void_caster_virtual_base<Oncomorph::Data::SliceId, Oncomorph::Persistence::Serializable>::~void_caster_virtual_base() () from /Data/See3D/oncomorph/2.6.36-gentoo-r5/lib/liboncomorph_base.so #9 0x00007ffff7930e38 in boost::serialization::detail::singleton_wrapper<boost::serialization::void_cast_detail::void_caster_virtual_base<Oncomorph::Data::SliceId, Oncomorph::Persistence::Serializable> >::~singleton_wrapper() () from /Data/See3D/oncomorph/2.6.36-gentoo-r5/lib/liboncomorph_base.so #10 0x00007fffe4d82f55 in cxa_finalize () from /lib/libc.so.6 #11 0x00007ffff7912dc6 in ?? () from /Data/See3D/oncomorph/2.6.36-gentoo-r5/lib/liboncomorph_base.so #12 0x0000000000000092 in ?? () #13 0x0000000000000000 in ?? ()

thanks Julle Juntunen

[Aaron Barany <akb825@…>]

Patch to fix the issue.

[Aaron Barany <akb825@…>]

This issue affects not only *nix shared libraries, but also Windows DLLs. I have attached a patch that fixes this issue. However, there are still situations where this can break, which I will discuss.

The change that caused this regression was in void_caster::recursive_unregister(). It does a loop over the entire set of registered void casters to remove all related void casters to the one being unregistered. In boost 1.43 and prior, it used to compare the pointer in the iterator to this within the loop to remove itself. However, in the boost 1.44 release that was changed to doing a find and erase after the loop. Since there are void_caster objects in the map with invalid extended type infos, that find caused a pure virtual function call.

I think the reason why it has invalid extended type infos in the map is due to how it registers the void caster shortcuts. In void_caster::recursive_register(), it will grab the extended type info from related void_casters, but those objects could reside in another library. Since only the current void caster being registered is marked as the parent, and not the other one it gets the other extended type info from, the other extended type info can be destructed before all the void_casters that use it are unregistered.

Though my patch, which restores the previous behavior, works around the issue during normal program startup and shutdown, if you dynamically load two libraries then later unload the first one, leaving the second library loaded, you can potentially have invalid void casters left behind to cause a crash the next lookup. Unfortunately, getting this to work properly will be very tricky, since if you simply unregister the void caster in this case, the void cast itself may still be valid, you just need a different extended type info instance to handle the comparison. I see two potential paths you can take to fix this: either set up some sort of system to replace extended type infos being destructed with valid equivalents or remove the shortcuts and use recursive lookups instead.

[anonymous]

I spent a little time looking at this.

I looked at the stack traces (and log output) from a couple of different reports. The traces I see look like:

~void_caster

recurrsive_unregister

operator<

Or some variation - all ending with operator<

operator< compares the dereferenced pointers to extended_type_info.

I couldn't see any reason why this should occur with void_caster. I'm thinking the issue would be with extended_type_info. recurrsive_unregister invokes operator< on dereferenced pointers to extended_type_info objects.

I'm concerned that there may be multilple such objects in the map. This would occur if one included code in both the DLL and the mainline. There is a trap to detect this because I was concerned about just this problem. I had to disable this trap because it required more discipline in code organization than many could implement. I would be curious to know if those who have experienced this problem were to re-enable this trap if it would catch this issue.

This hypothesis would also explain why the situation isn't as common as I would expect it to be if were something else.

This trap is found and explained at basic_serializer_map.cpp line 46. Try these test cases with the following:

a) uncomment the code at line 63 in basic_serializer_map.cp b) rebuild the library c) rerun the problem program d) let me know the results.

I realize that this is a lot of work. But without this kind of effort, it's very difficult to make progress here. I thank all those who have invested efforts here and appreciate your patience and help.

Robert Ramey

[Aaron Barany <akb825@…>]

Though I can't enable that trap right now, I can say without a doubt that it would be hit in my case. I have encountered other issues due to serializers being instantiated in multiple libraries, which I provided a patch for in ticket #5341. Unfortunately, in my use case it would be nearly impossible to avoid serializing across shared library boundaries, and I'm sure that's the case for a number of other users as well.

After looking at void_cast.cpp for a while and thinking about different situations where there are multiple instances of void_casters, I think that I have found a situation that would cause a crash. This situation can also cause a crash if everything is statically linked.

First, I will point out an issue with the current deregistration method. The first void caster that is constructed for a particular type relation will be inserted into the set, and any duplicates will be ignored. However, since it's doing a find and remove in recursive_unregister, the frist void caster to be destructed will remove whichever void caster was registered for that type relation, since it's comparing the type infos. For example, if you have void casters A and B, which both represent the same base and derived types, A will be registered on construction, while B will be ignored. On destruction, B will be destructed first and remove A. With the previous behavior, it was comparing pointers, so B wouldn't find itself in the set, and A wouldn't be unregistered until A was destructed.

I will now lay out a particular case where this unregistration behavior can cause a crash. Lets say you have a class hierarchy of A, B, C, and D, where A is the base class and D is the most derived class. If void casters are registered for A->B, B->C, and C->D, shortcuts will be made along the way. Depending on the order of initialization, such as if C->D is registered before B->C, those shortcuts can then then recursively create shortcuts of their own. In this example, let's say the shortcut B->D recursively creates the shortcut A->D, so the parent of A->D is the shortcut for B->D.

Now we have our hierarchy of void casters set up. However, let's say somewhere else in code the void caster B->D is registered directly, and is constructed after all the void casters and resulting shortcuts above. That void caster will see the B->D shortcut registered already, and won't insert it into the set. When the program shuts down, that explicit B->D void caster will be destructed first, since it was constructed last. When that void caster is destructed, recursive_unregister will do a find for the B->D type relation, and end up removing the shortcut registered earlier. That shortcut is merely removed from the set, and is never actually deleted, so it is leaked. As a result of that, the A->D shortcut created by the B->D shortcut is never removed from the set, since its parent's destructor is never called and is never recursively unregistered. At some point, the type infos for A and D are destroyed, but that A->D shortcut that references them is still left in the set. If there are still void casters registered at that point, the next time a find is performed, it will crash with the pure virtual function call.

Since order of construction at startup isn't guaranteed, this specific ordering can occur regardless of if shared libraries are used or not. This will generally occur if you are serializing polymorphic type hierarchies by base pointer, as that serialization will register an explicit void caster that can skip over multiple classes in the hierarchy and conflict with an existing shortcut. I think there has been a correlation with the use of shared libraries and experiencing this bug since projects with large and complex type hierarchies are more likely to use shared libraries.

[ruediger.berlich@…]

Hi Robert,

Replying to anonymous:

a) uncomment the code at line 63 in basic_serializer_map.cp b) rebuild the library c) rerun the problem program d) let me know the results.

I did this and there is no change at all (i.e. the trap doesn't trigger, program runs through as normal and crashes at the end with message "pure virtual method called").

Best Regards, Ruediger

[ruediger.berlich@…]

Replying to ruediger.berlich@…:

I did this and there is no change at all (i.e. the trap doesn't trigger, program runs through as normal and crashes at the end with message "pure virtual method called").

I have now also tried Aaron's patch (with Robert's trap still active).

For all programs I have tried it fixes the problem ...

This was done on Ubuntu Linux 10.10 / 64 bit, g++ 4.4.5 .

Note that the Geneva library is a toolkit, so I have to run further tests to see whether this cures the problem in all our usage scenarios. I am travelling at the moment, so I can only run further tests when I am back.

Best Regards, Ruediger

[anonymous]

Replying to ruediger.berlich@…:

This was done on Ubuntu Linux 10.10 / 64 bit, g++ 4.4.5 .

Apologies for repeatedly replying to myself. But I guess it is important to note that I have used Boost 1.46.1 for the tests above. I am not linking statically, and serialized classes come from different dynamic libraries in my test programs.

Best Regards, Ruediger

[anonymous]

Hi Robert, Aaron

Replying to anonymous:

a) uncomment the code at line 63 in basic_serializer_map.cp b) rebuild the library c) rerun the problem program d) let me know the results.

I tried this but the problem occurs just as before.

The patch by Aaron, however, fixes the problem.

br.
Julle Juntunen

[Robert Ramey]

I've tested Aaron's patch and committed it to the trunk. I'm taking everyone's word for it that this is the fix as I can't reproduce it myself.

Robert Ramey

[Bryce Adelstein Lelbach]

Ramey, for whatever peace of mind it's worth, Aaron's patch fixed a serious issue in the codebase which Hartmut and I work on. Thanks for applying, this resolves a big headache for me.