Boost C++ Libraries: Ticket #9612: Memory Access Violation when recursively serializing classes https://svn.boost.org/trac10/ticket/9612 <p> See attached example. I have a class which contains one member variable: a shared pointer to its base class. If the actual object type is the same as the object being serialized, the serialization libraries crash. It also only occurs if they are in separate modules. </p> en-us Boost C++ Libraries /htdocs/site/boost.png https://svn.boost.org/trac10/ticket/9612 Trac 1.4.3 Chris Rusby <chris.rusby@…> Tue, 28 Jan 2014 13:46:07 GMT attachment set https://svn.boost.org/trac10/ticket/9612 https://svn.boost.org/trac10/ticket/9612 <ul> <li><strong>attachment</strong> → <span class="trac-field-new">CrashTest.zip</span> </li> </ul> Ticket Brandon Kohn Fri, 31 Jan 2014 15:16:22 GMT <link>https://svn.boost.org/trac10/ticket/9612#comment:1 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:1</guid> <description> <p> It looks like there are two instances of the basic_iserializer&lt;xml_iarchive, <a class="missing wiki">TestObject</a>&gt; which live in the CrashTestDLL and then in main application. The one in the dll has a bpis member where the one in the main app has a null member. To figure this out I set a breakpoint on baseic_iserializer.hpp: basic_iserializer::set_bpis (around line 64 in boost 1.53). When it loads the dll you'll see these singletons being setup with the pointer iserializer. </p> <p> Then in the test case code add: </p> <div class="wiki-code"><div class="code"><pre> <span class="n">BOOST_AUTO</span><span class="p">(</span> <span class="k">const</span> <span class="o">&amp;</span><span class="n">cinst</span><span class="p">,</span> <span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">serialization</span><span class="o">::</span><span class="n">singleton</span><span class="o">&lt;</span><span class="n">boost</span><span class="o">::</span><span class="n">archive</span><span class="o">::</span><span class="n">detail</span><span class="o">::</span><span class="n">iserializer</span><span class="o">&lt;</span><span class="n">boost</span><span class="o">::</span><span class="n">archive</span><span class="o">::</span><span class="n">xml_iarchive</span><span class="p">,</span> <span class="n">TestObject</span><span class="o">&gt;</span> <span class="o">&gt;::</span><span class="n">get_const_instance</span><span class="p">())</span> <span class="p">);</span> <span class="n">BOOST_CHECK</span><span class="p">(</span><span class="n">cinst</span><span class="p">.</span><span class="n">get_bpis_ptr</span><span class="p">()</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span><span class="c1">//! This is null!</span> </pre></div></div><p> I used the debugger to copy the value of m_bpis from the singleton in the DLL into the one in the main application and it then loads. </p> </description> <category>Ticket</category> </item> <item> <author>Chris Rusby <chris.rusby@…></author> <pubDate>Fri, 31 Jan 2014 15:21:57 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:2 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:2</guid> <description> <p> Yes - I agree 100% with your analysis: I should probably have added in the original ticket that it doesn't crash on Linux or when I'm not serializing across a DLL boundary. </p> <p> Am I doing something wrong (I don't think I am), or is this a boost serialization bug? It strikes me that the definition of the basic_iserialization class might be missing a BOOST_DLLEXPORT macro to ensure it's exported from CrashTestDLL and hence not duplicated? </p> <p> Thanks, </p> <p> Chris </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Brandon Kohn</dc:creator> <pubDate>Fri, 31 Jan 2014 18:04:12 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:3 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:3</guid> <description> <p> The singleton&lt;iserializer&lt;Archive,<a class="missing wiki">TestObject</a>&gt; &gt; is the thing being duplicated, and it is exported. BOOST_DLL_EXPORT is (in msvc) always defined as _declspec(dllexport), and is used on the static public interface of the singleton type. Near as I can tell, this means it will be exported from every DLL or exe in which it is included. I don't know enough about what could be done differently to be able to call it a bug. I'm surprised we haven't run into an issue like this example in our codebase as we use a lot of DLLs. I suppose we tend to concentrate our use of serialization in the same DLLs as where the types are exported. Perhaps that could be a workaround? </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Robert Ramey</dc:creator> <pubDate>Fri, 31 Jan 2014 18:36:48 GMT</pubDate> <title>status changed; resolution set https://svn.boost.org/trac10/ticket/9612#comment:4 https://svn.boost.org/trac10/ticket/9612#comment:4 <ul> <li><strong>status</strong> <span class="trac-field-old">new</span> → <span class="trac-field-new">closed</span> </li> <li><strong>resolution</strong> → <span class="trac-field-new">invalid</span> </li> </ul> <p> I've seen problems occur with DLLS when the "same" code is included in two different DLLS. </p> <p> This can occur when an inline serialization function is in a header which is used by several DLLS. Each inline code generation ripples back to include stuff in the library which ends up being duplicated. example </p> <p> Instead of </p> <pre class="wiki">// X.hpp struct X { template&lt;class Archive&gt; serialize(Archive &amp; ar, unsigned int file_version){ ar &amp; ... }; </pre><p> Use </p> <pre class="wiki">// X.hpp struct X { template&lt;class Archive&gt; serialize(Archive &amp; ar, unsigned int file_version); }; // X.cpp // use explicit instantiation template X::serialization&lt;boost::serialization::text_iarchive&gt;( boost::serialization::text_iarchive &amp; ar, unsigned int file_version){ ar &gt;&gt; .. } template X::serialization&lt;boost::serialization::text_iarchive&gt;( boost::serialization::text_oarchive &amp; ar, unsigned int file_version){ ar &lt;&lt; .. } </pre><p> This will leave all the serialization code for X in one and only one place. Thereby diminishing DLL hell and making DLLS smaller. </p> <p> If someone want's to write an article on tis subject - organizing code in DLLS, plugins using polymorphic versions of archives, etc. I'll consider including it in the documentation. </p> <p> Robert Ramey </p> Ticket Brandon Kohn Sun, 02 Feb 2014 21:32:23 GMT attachment set https://svn.boost.org/trac10/ticket/9612 https://svn.boost.org/trac10/ticket/9612 <ul> <li><strong>attachment</strong> → <span class="trac-field-new">0001-Changes-to-singleton-to-allow-extern-templates-on-si.patch</span> </li> </ul> <p> Patch to workaround issue </p> Ticket Brandon Kohn Sun, 02 Feb 2014 21:33:29 GMT attachment set https://svn.boost.org/trac10/ticket/9612 https://svn.boost.org/trac10/ticket/9612 <ul> <li><strong>attachment</strong> → <span class="trac-field-new">CrashTestDLLFileFixes.zip</span> </li> </ul> <p> Changes to files in test to work with patch </p> Ticket Brandon Kohn Sun, 02 Feb 2014 21:37:20 GMT <link>https://svn.boost.org/trac10/ticket/9612#comment:5 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:5</guid> <description> <p> I've looked into this a bit more and I think there is a way to support this case for compilers which support extern templates. That includes msvc back to 8.0. </p> <p> The trick is to combine the dllexport/import with the extern for the explicit instantiations of the singleton's for the i/oserialzer instances. This requires changes to singleton.hpp (moved the exports to the class level.) Was there a particular reason the exports were put on the members? </p> <p> I put together a patch. Check it out Chris. </p> <p> Note: There is a warning by the vc12 compiler c4910 : <a class="ext-link" href="http://msdn.microsoft.com/en-us/library/bb531392%28v=vs.90%29.aspx"><span class="icon">​</span>http://msdn.microsoft.com/en-us/library/bb531392%28v=vs.90%29.aspx</a> </p> <p> I agree with the comment on that page that this shouldn't be a warning. Export and extern are not mutually exclusive and should be able to modify the same thing. In spite of the warning, the compiled code does the right thing in all the versions I tested. (vc11, vc12). </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Robert Ramey</dc:creator> <pubDate>Mon, 03 Feb 2014 02:10:57 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:6 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:6</guid> <description> <p> I've taken a cursory look at your patches. </p> <p> I've spent quite a bit of time over the years looking at this problem and concluded there is no robust, maintainable solution other than to follow the advice I gave above. Following the practice outlined above will avoid problems with the same code appearing in different modules having the possibility of conflicting. It also supports plug-in DLLS in an effective way. And it eliminates unnecessary code bloat. Trying to support that which is really an application design mistake is a fools errand that never ends. </p> <p> So I wouldn't expect such a patch to appear in the library. </p> <p> Robert Ramey </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Brandon Kohn</dc:creator> <pubDate>Mon, 03 Feb 2014 12:40:04 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:7 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:7</guid> <description> <p> Hi Robert, </p> <p> Did you look at his example? It follows your suggestions and still fails. I don't see where there is a design mistake. </p> <p> As for the patch, I think you should consider changing the way the singleton is exported so doing externs like this is possible. I believe the risks are minimal. </p> <p> Cheers, </p> <p> Brandon </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Robert Ramey</dc:creator> <pubDate>Mon, 03 Feb 2014 16:20:46 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:8 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:8</guid> <description> <p> I took another look at your example. It is not a simple example or test. it has serialization of cycles, boost shared pointers, implementation across DLLs and serialization through a base class pointer and xml. Each of these is sort a "corner case" in that it required separate coding and attention to these aspects so that the features can be combined in an orthogonal way. </p> <p> I also took another look at your patch. This involves Export facility which is another very murky area which touches upon C++ instatiation, visibility across compilers and other stuff. </p> <p> So introducing such a patch under these circumstances has a strong possibility of turning into a very major investment of time. </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Brandon Kohn</dc:creator> <pubDate>Mon, 03 Feb 2014 16:35:45 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:9 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:9</guid> <description> <p> Strictly speaking, all serialization library needs to do to support this fix is to modify the way the singleton is exported. Currently the export specifier is on the static members of singleton. Moving the specifier to the class level allows users to do their own exports of the instantiations. The change works with all supported msvc compilers (i.e. since vc8) and on gcc 4.8.2. Perhaps it would be worthwhile to make the change on develop and see how the test runners perform? If anything I would say the mixing of extern/dllexport in the extern.hpp macros is the more risky proposition. However, it's easy enough for a user to implement those if needed. </p> <p> Why was the export implemented on the members instead of at the class level initially? Was there some compiler which did not handle this? I know clang doesn't now, but it's logged as a bug. </p> </description> <category>Ticket</category> </item> <item> <author>Chris Rusby <chris.rusby@…></author> <pubDate>Mon, 03 Feb 2014 17:20:28 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:10 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:10</guid> <description> <p> Based on the above, I am of the opinion that I (the user) need to tell the boost serialization libraries whether to export the implementation of the instantiated template from the DLL. I envisage adding a new macro that I'd use something like this: </p> <p> BOOST_CLASS_EXPORT_KEY3(<a class="missing wiki">TestObject</a>, "<a class="missing wiki">TestObject</a>", CRASHTESTDLL_EXPORTS) </p> <p> This would <span class="underline">declspec(dllexport) the instantiated <a class="missing wiki">TestObject</a> serialization templates from the DLL (because CRASHTESTDLL_EXPORTS would be defined), and from the main program would </span>declspec(dllimport) the serialization templates and hence import from the DLL. Hence there would be one and only one implementation of iserializer&lt;Archive,<a class="missing wiki">TestObject</a>?&gt; (which seems to be the root cause of my problem). </p> <p> I've not verified that this works, I'm afraid, but I hope to soon. If it does work, though, the advantage is that it's not going to modify existing code so is much lower risk. </p> <p> Thanks, </p> <p> Chris </p> <p> PS In answer to your (Robert) original question, when I understand it fully i.e. when this is fixed and I understand why/how it's fixed, I'd be more than happy to do some documentation for you. </p> </description> <category>Ticket</category> </item> <item> <author>Chris Rusby <chris.rusby@…></author> <pubDate>Wed, 05 Feb 2014 22:41:34 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:11 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:11</guid> <description> <p> Hi again, </p> <p> I've just made some small serialization changes in our code-base in a much less edge-casey way. </p> <p> If I understand the cause of the problem, it arises because I have a class defined in DLL A, and I'm serializing a shared pointer to this within classes in DLL B and DLL C - and this isn't happy (windows _and_ linux). </p> <p> I've not yet managed to simplify the test case sufficiently to be able to upload it, but I hope to do this in the next few days. </p> <p> On the one hand, I think Brandon's fix looks good (I will try it out fully tomorrow) - but on the other hand I can see why Robert's nervous about putting this live as it's fairly fundamental! </p> <p> However, maybe as a compromise we could take Brandon's changes to singleton.hpp? These look fairly innocuous, but they would allow me (or anybody else who comes across this problem) to make changes as per extern.hpp in our own code-base. Would this work? </p> <p> Thanks, </p> <p> Chris </p> </description> <category>Ticket</category> </item> <item> <author>Chris Rusby <chris.rusby@…></author> <pubDate>Fri, 07 Feb 2014 19:12:46 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:12 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:12</guid> <description> <p> I have finally figured out the cause of the 2nd crash. Here's what was going on. </p> <p> I was serializing in/out (correctly) shared_ptr&lt;const Object&gt;. In a completely different part of the codebase, somebody else was serializing out a shared_ptr&lt;const Object&gt; but serializing it back in as a shared_ptr&lt;Object&gt;. </p> <p> If I tested my code, it worked fine. However, if both the other code and mine were used in the same serialized file I got a null pointer exception (same place as above). </p> <p> I think the erroneous code was messing up the type registration in some way. It would be really nice if this could be caught at compile time - although I must admit, I can't think of any way to do this. Maybe the library could throw a nicer runtime error though? I guess the downside is performance? </p> <p> Thanks, </p> <p> Chris </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Robert Ramey</dc:creator> <pubDate>Fri, 07 Feb 2014 20:04:00 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:13 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:13</guid> <description> <p> If you look into basic_serializer_map.cpp you'll find a passage of commented out code which I implemented just to trap this type of situation. I believe that uncommenting this code would be effective in your situation. </p> <p> I had to comment it out because so many people just ignored this issue and instantiated code all over the place and this prevented their code from compiling. They complained a lot that the serialization library was "buggy" because of this. So I had to comment it out. </p> <p> Probably this should be re-enabled but permit the check to be suppressed with a run-time switch. But then this is also a hassle to implement so I just left it commented out. </p> <p> You might want to experiment with enabling this code and see what impact it would have on your application and perhaps think about how to conditionally enable it. - assuming you've got nothing else to do. </p> <p> Robert Ramey </p> </description> <category>Ticket</category> </item> <item> <dc:creator>anonymous</dc:creator> <pubDate>Fri, 07 Feb 2014 20:40:31 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:14 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:14</guid> <description> <p> This looks interesting, I'll try it next week. </p> <p> Thanks for your help, </p> <p> Chris </p> </description> <category>Ticket</category> </item> <item> <dc:creator>anonymous</dc:creator> <pubDate>Tue, 24 Jun 2014 22:02:30 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:15 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:15</guid> <description> <p> Hi again, </p> <p> We've come up against a situation which cannot easily be worked around, and hence I'm having to revisit this. I do believe that Brandon's fix is good. In fact, my current plan is to patch in his change to just singleton.hpp in our boost install; the rest can be managed within our codebase. </p> <p> I don't see an reason why the BOOST_DLLEXPORT needs to be on each function, rather than on the class itself. If it could be moved, then it would resolve my issue. </p> <p> Note that, with C++ 11's comes extern templates, and this is the "proper" fix for this issue as then there will only be one singleton implementation. </p> <p> Thanks, </p> <p> Chris </p> </description> <category>Ticket</category> </item> <item> <author>vincent.legoll@…</author> <pubDate>Sat, 27 Sep 2014 13:24:34 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:16 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:16</guid> <description> <p> Hello, I don't know if this is the same bug I'm getting, but it looks at least related. </p> <p> I'm getting: /usr/include/boost/serialization/singleton.hpp:132:13: runtime error: reference binding to null pointer of type 'const boost::archive::detail::extra_detail::guid_initializer&lt;<a class="missing wiki">CombatEvent</a>&gt;' </p> <p> I'm compiling freeorion (www.freeorion.org) with clang++-3.6 and boost version 1.55.0.2 from debian </p> <p> Is there anything else or new on the subject that I can try ? </p> </description> <category>Ticket</category> </item> <item> <dc:creator>Chris</dc:creator> <pubDate>Sat, 27 Sep 2014 13:26:17 GMT</pubDate> <title/> <link>https://svn.boost.org/trac10/ticket/9612#comment:17 </link> <guid isPermaLink="false">https://svn.boost.org/trac10/ticket/9612#comment:17</guid> <description> <p> Sorry - not the same bug; this one impacts windows only. </p> </description> <category>Ticket</category> </item> </channel> </rss>