Boost C++ Libraries: Ticket #4660: Error read binary archive, created by old boost version

attachment set

Mon, 20 Sep 2010 09:12:30 GMT

attachment → demofile.3.dat

binary archive 1.43 version

attachment set

Mon, 20 Sep 2010 09:12:50 GMT

attachment → demofile_1_44.dat

attachment set

Mon, 20 Sep 2010 09:15:25 GMT

attachment → demo.cpp

demo.cpp with my modification

Robert Ramey — Mon, 20 Sep 2010 15:17:26 GMT

Indeed this is a serious problem.

It is addressed in the release notes documentation 1.44 . Try the recommendation described there and let me know how that goes. If that does it for you, please close this ticket

Robert Ramey

anonymous — Tue, 21 Sep 2010 02:01:24 GMT

Replying to ramey:

Indeed this is a serious problem.

It is addressed in the release notes documentation 1.44 . Try the recommendation described there and let me know how that goes. If that does it for you, please close this ticket

Robert Ramey

I had searche documentation before open bug and I did not find any information. I have search http://www.boost.org/doc/libs/1_44_0/libs/serialization/doc/index.html : "As of this writing, there are no known bugs. However, due to compiler/library quirks and or bugs, some tests fail with some combinations of compilers and libraries. Differences from Boost 1.43 •fixed bug in the serialization of virtual base classes. Due to heroic efforts by Takatoshi Kondo"

There are not anathing in http://www.boost.org/users/download/version_1_44_0 too.

Can you give me link to your recomendation?

Robert Ramey — Tue, 21 Sep 2010 04:54:23 GMT

ouch - my mistake. This last documentation change didn't make it into the release branch. I've checked it in though I don't think it will show up until the next release. So I'm including the relevant text right here

Differences from Boost 1.43

fixed bug in the serialization of virtual base classes. Due to heroic efforts by Takatoshi Kondo.
Native binary archives created under versions 1.42 and 1.43 suffer from a serious problem. It's likely they won't be readable by this latest version. This due to the fact that 1.42 made some changes in the binary format of some types. Normally this could be addressed by detecting the library version number written into the archive header. Unfortunately, this library version number was not incremented at 1.42 as it should have been. So now we have two different binary archive versions with the same library version number.

This has been addressed by including a small utility in the example directory named fix_six.cpp. This should be run with the command line

fix_six <file name>

This will assign 7 to the library version number of the archive. This fix will need to be applied to native binary archives created with boost versions 1.42 and 1.43.

Let me know if this addresses the situation.

Robert Ramey

Tue, 21 Sep 2010 05:25:01 GMT

Replying to ramey:

ouch - my mistake. This last documentation change didn't make it into the release branch.
.........
Let me know if this addresses the situation.

Robert Ramey

I did not find any fix*.cpp in boost_1_44_0\libs\serialization\

But I think it is not problem with "6" version. I have native binary archives 1.34.1 and 1.43.0 And all was good. But version 1.44 change binary format of version information. Format before 1.44.0 was unsigned char. Format of 1.44 is unsigned short. So 1.44.0 read invalid value. For example it read 0x0404 as library version from file, which was created by 1.34.1, but should read 0x04. Please see files, which I sent as attachements, in any hex editor.

You can see library version information below as bold value
1.34.1 first 32 bytes of file:
16 00 00 00 73 65 72 69 61 6c 69 7a 61 74 69 6f 6e 3a 3a 61 72 63 68 69 76 65 04 04 04 04 08 01 1.43.0 first 32 bytes of file:
16 00 00 00 73 65 72 69 61 6c 69 7a 61 74 69 6f 6e 3a 3a 61 72 63 68 69 76 65 07 04 04 04 08 01 1.44.0 first 32 bytes of file:
16 00 00 00 73 65 72 69 61 6c 69 7a 61 74 69 6f 6e 3a 3a 61 72 63 68 69 76 65 07 00 04 04 04 08

attachment set

Robert Ramey — Tue, 21 Sep 2010 06:32:57 GMT

attachment → fix_six.cpp

Robert Ramey — Tue, 21 Sep 2010 06:33:30 GMT

ouch - again. I merging the missing file into the release branch. In the mean time her eit is.

Robert Ramey

Tue, 21 Sep 2010 06:57:38 GMT

Replying to ramey:

ouch - again. I merging the missing file into the release branch. In the mean time her eit is.

Robert Ramey

I download fix_six.cpp from trunk and build it with 1.44.0 release. The program does not work with archive, created by boost 1.34.1 and 1.43.0. The code " boost::archive::binary_iarchive ia(fb); " throw exception from line in basic_binary_iarchive.ipp:

    if(BOOST_ARCHIVE_VERSION() < input_library_version)
        boost::serialization::throw_exception(
            archive_exception(archive_exception::unsupported_version)
        );

Value of variable input_library_version is 0x0407 Please read my comments above about change format of library version information from "unsigned char" -> "unsigned short"

Wed, 22 Sep 2010 06:50:23 GMT

I prepare some simple test program cser and tser. See attachments. First program (cser) creates simple native binary archive. Second program (tser) test these archive. I build these programs with 3 different version of BOOST – 1.34.1, 1.43 and 1.44. Then I run all version of cser and create 3 archive – arch_1_34_1.dat, arch_1_43.dat and arch_1_43.dat (see attachments). Then I run each version of tser with each version of archive. The results are:

tser + boost 1.34.1
arch_1_34_1.dat – “Test – OK”
arch_1_43.dat – “archive serializations error - unsupported version”
arch_1_44.dat – “archive serializations error - unsupported version”

tser + boost 1.43
arch_1_34_1.dat – “Test – OK”
arch_1_43.dat – “Test – OK”
arch_1_44.dat – “archive serializations error - incompatible native format - size of int”

tser + boost 1.44
arch_1_34_1.dat – “archive serializations error - unsupported version”
arch_1_43.dat – “archive serializations error - unsupported version”
arch_1_44.dat – “Test – OK”

My summary
1) serialization 1.44 can not work with old version of binary archives. So backward compatibility was ruined.
2) format of binary archive in version 1.44 was changed, so version 1.43 does not work with new archive. And now we have 2 different incompatible version of native binary archive (before 1.44 and 1.44). Text archive does not have this problem

So implementation of serialization in 1.44 is broken. I think this problem must be resolved ASAP. I think users must be informed about these problem with serialization and users must do not use this version in production code. May be we need 1.44.1 version?

attachment set

Wed, 22 Sep 2010 06:51:34 GMT

attachment → cser.cpp

Create simple native archive

attachment set

Wed, 22 Sep 2010 07:11:30 GMT

attachment → tser.3.cpp

test native binary archive

attachment set

Wed, 22 Sep 2010 07:12:28 GMT

attachment → arch_1_34_1.dat

arcive 1.34.1 version

attachment set

Wed, 22 Sep 2010 07:13:20 GMT

attachment → arch_1_43.dat

archive 1.43 version

attachment set

Wed, 22 Sep 2010 07:14:00 GMT

attachment → arch_1_44.dat

archive 1.44 version

Robert Ramey — Sat, 25 Sep 2010 18:41:48 GMT

OK, I'm now going to look at this more carefully.

I do think there is a little more to this that meets the eye. The problem started when I made a simple tweak in 1.42 to eliminate a warning. It wasn't really correct and the issue was very muddled because the concept of "version" was applied to library version and data type version which are quite different things. There were a few other problems besides. Your observations and effort ARE appreciated so don't give up hope. I'm looking at this

Robert Ramey

hajokirchhoff — Mon, 18 Oct 2010 09:45:54 GMT

Hi there, I am bitten by this as well.

Problem: Prior to 1.44 the archive version was written as "one byte". Starting with 1.44 archive version type changed from 'version_type' to 'library_version_type' and is expected to be read as two bytes.

There is a comment in basic_archive.cpp:

// 7 - separated version_type into library_version_type and class_version_type
//     changed version_type to be stored as 8 bits.

this definitely points to the problem.

This is a really serious problem, a potential showstopper. The longer this remains open, the more binary archives of version <1.44 and 1.44 will both exist. And if this problem cannot be resolved in a backwards compatible way, either all binary <1.44 or all binary 1.44 archives will break.

Also I think it is imperative to add backwards compatibility to the test suite! A new version of boost::serialization should not be released unless tests reading previous versions of the archives pass.

Breaking changes to the archives are really, really bad! And I mean really bad!

(Otherwise, thanks a lot for your work. It's a great library :-)

version changed

Vladimir Prus — Wed, 20 Oct 2010 05:50:31 GMT

version Boost 1.44.0 → Boost 1.45.0

Bryce Adelstein Lelbach — Wed, 20 Oct 2010 06:09:08 GMT

Ramey has committed (r66106 and r66107); this should be fixed. Can the original submitter of this ticket please check if r66106 and r66107 fix this error?

milestone changed

Vladimir Prus — Wed, 20 Oct 2010 06:11:56 GMT

milestone To Be Determined → Boost-1.45.0

Wed, 20 Oct 2010 08:04:31 GMT

Replying to wash:

Ramey has committed (r66106 and r66107); this should be fixed. Can the original submitter of this ticket please check if r66106 and r66107 fix this error?

I have updated my version of serializatopn library to revision 66114 and built my small test programs cser and tser (see above). New version (1.44+) work well. (VC 7.1 + STLPort)

I think we need detailed description of possible problem of using 1.44 version of serialization and some recomendation (not using 1.44 for binary archive?)

I think hajokirchhoff is is absolutely right :

Also I think it is imperative to add backwards compatibility to the test suite! A new version of boost::serialization should not be released unless tests reading previous versions of the archives pass.

It would be very nice if someone checked backward compatibility with other types of archives (text, XML, portability ...)

Robert Ramey — Wed, 20 Oct 2010 15:59:47 GMT

I have checked in changes to the trunk to address this. Please feel free to verify that they have addressed the problem.

I would love to have tests to verify compatibility with previous archives - especially binary ones. There are a lot of problems with this though.

a) It's a heck of a lot of work. b) The serialization library already stretches testing facilities to the breaking point. It has a fairly large number of tests which are run with all the different archive implementations. It has been referred to as "carpet bombing".

If someone want's to do this I would support the effort though.

Robert Ramey

status changed; cc, resolution set

Bryce Adelstein Lelbach — Sat, 30 Oct 2010 00:25:16 GMT

cc Bryce Adelstein Lelbach added
status new → closed
resolution → fixed

This has since been resolved.

mark.stijnman@… — Wed, 24 Nov 2010 10:45:51 GMT

I was bitten by this bug too. Did this make it into the 1.45 release? I didn't see serialization listed in the updated libraries, and it's not in the "changes since version..." section of the serialization documentation either.

thomas.bretgeld@… — Wed, 08 Dec 2010 20:30:58 GMT

Replying to mark.stijnman@…:

I was bitten by this bug too. Did this make it into the 1.45 release? I didn't see serialization listed in the updated libraries, and it's not in the "changes since version..." section of the serialization documentation either.

I second this question. Has this problem been resolved or does it also affect v1.45? I ask, because I have updated to boost v1.45 yesterday and have problems to open older binary archives, which have been created with boost v1.37 and v1.42. To sort out where the error comes from, I tried to recompile my software and the boost libraries with some intermediate versions of boost. It works ok with v1.42. On v1.44 I get the "unsupported version" error, but on v1.45 I am getting a lot of exceptions "basic_string::resize" when I try to open older binary archives. I am using Debian lenny with gcc 4.3.2.

Kind regards, Thomas

status, severity changed; resolution deleted

Thu, 09 Dec 2010 03:05:28 GMT

status closed → reopened
resolution fixed
severity Showstopper → Problem

Replying to mark.stijnman@…: and thomas.bretgeld@…

I wrote letter about problem with documentation of this bug - http://thread.gmane.org/gmane.comp.lib.boost.devel/210992 I see the problem is concerned not only me. As I do not undestand situation with 1.45 version, I reopen this bug.

Thu, 09 Dec 2010 19:54:10 GMT

Replying to thomas.bretgeld@…:

Replying to mark.stijnman@…: I second this question. Has this problem been resolved or does it also affect v1.45? I ask, because I have updated to boost v1.45 yesterday and have problems to open older binary archives, which have been created with boost v1.37 and v1.42. To sort out where the error comes from, I tried to recompile my software and the boost libraries with some intermediate versions of boost. It works ok with v1.42. On v1.44 I get the "unsupported version" error, but on v1.45 I am getting a lot of exceptions "basic_string::resize" when I try to open older binary archives. I am using Debian lenny with gcc 4.3.2.

It looks to me like errors are expected when reading archives created with v1.42 or v1.43.

http://www.boost.org/doc/libs/1_45_0/libs/serialization/doc/release.html#differences_1_43

However, I think archives created with v1.37 are supposed to work fine as is, so that might be a separate issue.

Robert Ramey — Thu, 23 Dec 2010 22:42:33 GMT

I've updated the trunk in an attempt to fix this problem. That is, I believe that now it should be able to read binary archives created prior to 1.43. Please check this out and let me know if it addresses the situation.

Robert Ramey

Wed, 05 Jan 2011 08:42:29 GMT

Replying to ramey:Please check this out and let me know if it addresses the situation.

I tried to read binary archives created with 1.42, it failed in the exact same way as with using 1.45 (std::string allocator throwing bad_alloc).

Wed, 05 Jan 2011 10:47:57 GMT

The most simple test case to reproduce the error I was able to find is:

std::map<std::string, size_t> m; maaa = 111; mbbb = 222;

std::ofstream fb("archive.ser"); std::ifstream fb("archive.ser"); boost::archive::binary_oarchive oa(fb); boost::archive::binary_iarchive ia(fb); oa << m; ia >> m;

I compiled it using ms vc8 and boost 1.42 and created the test archive
If I read it using the same setup the map gets restored flawlessly
If I read it using the code compiled with ms VC10 and boost svn trunk "m" is empty (my real world rather dies with bad_alloc deep inside the call stack )

Wed, 05 Jan 2011 10:50:41 GMT

Sorry the code formation got lost, I pasted it here http://codepad.org/PK2fwo4c

Robert Ramey — Wed, 05 Jan 2011 23:12:21 GMT

Hmmm....

I built tser_3.cpp on my system with the trunk version of the library.

I used it read arch_34_1.dat arch_43.dat and arch_44.dat without problem. So things look OK to me now.

Robert Ramey

Thu, 06 Jan 2011 08:19:15 GMT

Hi Robert, I see the following possible differences between us:

I failed to read an archive created with 1.42, you succeeded reading 1.34,1.43 and 1.44

I used VS 2010 to build the reader and VC 2005 to build the writer, so maybe it's the compiler

I used a map<std::std::string, size_t>, you rather integral and float numbers. Since my production code fails with throwing bad_alloc inside the basic_string allocator, that may also be an hint that the problem is specific to serialization of std::string

Regards

Robert Ramey — Tue, 11 Jan 2011 19:11:43 GMT

"I failed to read an archive created with 1.42, you succeeded reading 1.34,1.43 and 1.44"

Hmmm - does you're trunk version succeed or fail with 1.34,1.43 and 1.44? I don't your 1.42 test to try. Basically we're not running the same tests. I'm willing to try any test which fails for you.

Robert Ramey

attachment set

Wed, 12 Jan 2011 09:52:48 GMT

attachment → ser_test.cpp

read or write test archive

attachment set

Wed, 12 Jan 2011 09:53:31 GMT

attachment → archive-1_43.ser

archive created with 1_43

attachment set

Wed, 12 Jan 2011 09:53:47 GMT

attachment → archive-1_46.ser

archive created with trunk

Wed, 12 Jan 2011 10:01:38 GMT

Robert, I attached:

ser_test.cpp: simple code to read/write archive with std::map<std::string, size_t> in it
archive-1_43.ser: output of ser_test.cpp build using boost 1_43
archive-1_46.ser: output of ser_test.cpp build using boost trunk

I run the following tests:

read archive-1_43.ser using ser_test.exe compiled using boost 1_43 -> success
read archive-1_43.ser using ser_test.exe compiled using boost trunk -> fail (map is empty after read)
read archive-1_46.ser using ser_test.exe compiled using boost trunk -> success
read archive-1_46.ser using ser_test.exe compiled using boost 1_43 -> throws unsupported_version

All compile output was created using msvc 10.0. Should be easy to reproduce. Regards

attachment set

Robert Ramey — Wed, 12 Jan 2011 17:26:14 GMT

attachment → tser_3.cpp

Robert Ramey — Wed, 12 Jan 2011 17:31:47 GMT

"read archive-1_46.ser using ser_test.exe compiled using boost 1_43 -> throws unsupported_version"

This is correct behavior. Older versions of the library cannot necessarily read archives created with subsequent versions of the library.

"# read archive-1_43.ser using ser_test.exe compiled using boost trunk -> fail (map is empty after read)"

I downloaded ser_test.cpp but it failed to compile on my vc 9.0system

Robert Ramey

Robert Ramey — Wed, 12 Jan 2011 17:34:43 GMT

Looking at ser_test.cpp (in spite of it's failure to compile) I see at least one other problem: A binary archive is being used with a stream which has not been opened with ios::binary. This is doomed to fail in any event.

Robert Ramey

attachment set

Thu, 13 Jan 2011 11:40:45 GMT

attachment → ser_test_2.cpp

create / read test archive including std::map<std::string, size_t>, corrections: open file in binary mode, adapted to legacy c++ standard

Thu, 13 Jan 2011 11:43:10 GMT

Sorry for the inconvenience. I corrected the test program, it now opens the file streams in binary mode and it should compile on vc9.0, too.

All test results stays the same, the problematic one beeing:

read archive-1_43.ser using ser_test.exe compiled using boost trunk -> fail (map is empty after read)

Robert Ramey — Fri, 14 Jan 2011 22:41:40 GMT

OK, I've checked in a change into the trunk which I believe fixes this latest problem. That is, the loading a collection. Please update from the trunk and verify that I've fixed this. Thanks for giving me this test case.

Robert Ramey

Mon, 17 Jan 2011 12:49:55 GMT

Hello Robert, Thank you, the latest trunk fixes all my use cases so far. Since the problem will live in 1.45 it may be a good idea to add a note to it's docs. Regards, Joerg

status changed; resolution set

Robert Ramey — Tue, 18 Jan 2011 22:03:56 GMT

status reopened → closed
resolution → fixed

attachment set

d s m a l l @… — Sat, 22 Jan 2011 20:25:23 GMT

attachment → basic_binary_iarchive-modified.zip

File modifed to read version 3 archives

d s m a l l @… — Sat, 22 Jan 2011 20:35:04 GMT

We are still seeing a problem when using the latest ( Jan 21 2010 ) code. Seems there is still an issue with the assumption regarding the size used to store the file version.

Fwiw we are just in the process of upgrading our product to use boost 1.45. It has previously used 1.33.1 and 1.37. We have unit tests that test the ability to read old config archives and one of them is failing. ( Actually others may fail as well but we are stuck on the first failure ).

We have worked around the problem by producing a modified version of basic_binary_iarchive.hpp which we include before including any other boost archiving headers. I have attached this modified file in basic_binary_iarchive-modified.zip. Note that the basis for file was 1.45 and not the current code.

Please let me know if you need additional information.

status changed; resolution deleted

d s m a l l @… — Sat, 22 Jan 2011 20:40:17 GMT

status closed → reopened
resolution fixed

Oops, looks like I forgot to select re-open.

Robert Ramey — Sun, 23 Jan 2011 03:23:04 GMT

OK - I've got your patch and have rolled in your changes. Should be in the trunk by monday. Thanks for tracking this down.

Robert Ramey

d s m a l l @… — Wed, 26 Jan 2011 16:09:37 GMT

Thanks for putting the fix in.

We have made further our upgrade and have found that the problem also occurs when the library version is 5. ( Believe version 3 --> boost 1.33.1 and version 5 --> boost 1.37 ). In other words in our work around header we have changed the condition from

if(boost::archive::library_version_type(3) == lvt )

if(boost::archive::library_version_type(3) == lvt

boost::archive::library_version_type(5) == lvt )

Btw we hope to complete our upgrade within the next couple of weeks. When we are done, which means we will have made it through all our unit tests, I'll post a comment to let you know.

Robert Ramey — Wed, 26 Jan 2011 16:32:34 GMT

note that the current code in trunk and release looks like:

    void load_override(version_type & t, int version){
        library_version_type lvt = this->get_library_version();
        if(boost::archive::library_version_type(7) < lvt){
            this->detail_common_iarchive::load_override(t, version);
        }
        else
        if(boost::archive::library_version_type(6) < lvt){
            uint_least8_t x=0;
            * this->This() >> x;
            t = boost::archive::version_type(x);
        }
        else
        if(boost::archive::library_version_type(5) < lvt){
            uint_least16_t x=0;
            * this->This() >> x;
            t = boost::archive::version_type(x);
        }
        else
        if(boost::archive::library_version_type(2) < lvt){
            // upto 255 versions
            unsigned char x=0;
            * this->This() >> x;
            t = version_type(x);
        }
        else{
            unsigned int x=0;
            * this->This() >> x;
            t = boost::archive::version_type(x);
        }
    }

Doesn't this do the job for you?

Robert Ramey

fperbet — Thu, 27 Jan 2011 11:05:41 GMT

Hi,

I ran into the same (serious) problem: exception "unsupported version" (see callstack)

I understand there's a potential fix in the trunk but this is not an option for me.

I tried the "fix_six.cpp" but it didn't help. Is there another way to simply fix the binary archive files so they can be read by 1.44 ?

I am using 1.44 and the archive were generated using 1.39 (if I remember correctly).

Any idea would help.

Frank Perbet

CALLSTACK: boost::serialization::throw_exception<boost::archive::archive_exception>(const boost::archive::archive_exception & e={...}) Line 37 C++ boost::archive::basic_binary_iarchive<boost::archive::binary_iarchive>::init() Line 93 + 0x35 bytes C++ boost::archive::binary_iarchive_impl<boost::archive::binary_iarchive,char,std::char_traits<char> >::init(unsigned int flags=0) Line 57 C++ boost::archive::binary_iarchive_impl<boost::archive::binary_iarchive,char,std::char_traits<char> >::binary_iarchive_impl<boost::archive::binary_iarchive,char,std::char_traits<char> >(std::basic_istream<char,std::char_traits<char> > & is={...}, unsigned int flags=0) Line 86 C++ bodyanalyzer.dll!boost::archive::binary_iarchive::binary_iarchive(std::basic_istream<char,std::char_traits<char> > & is={...}, unsigned int flags=0) Line 84 + 0x3d bytes C++

d s m a l l @… — Tue, 15 Feb 2011 01:37:47 GMT

Replying to ramey:

Sorry for the delay in reply. The code you posted won't work for us. Fwiw, below is the code that we have in our over-ride header. With it all of our unit tests are passing

    void load_override(version_type & t, int version){
        library_version_type lvt = this->get_library_version();
        if(boost::archive::library_version_type(7) < lvt){
            this->detail_common_iarchive::load_override(t, version);
        }
        else
        if(boost::archive::library_version_type(6) < lvt){
            uint_least16_t x=0;
            * this->This() >> x;
            t = boost::archive::version_type(x);
        }
        else
        if(boost::archive::library_version_type(3) == lvt || boost::archive::library_version_type(5) == lvt ){
#pragma message( "CTMS fix for serialization bug ( lack of backwards compatability ) introduced by boost 1.45." )
            // upto 255 versions
            unsigned char x=0;
            * this->This() >> x;
            t = version_type(x);
        }
        else{
            unsigned int x=0;
            * this->This() >> x;
            t = boost::archive::version_type(x);
        }
    }

Robert Ramey — Sat, 19 Feb 2011 17:12:24 GMT

Hmmm - this is quite different than what we've checked into the trunk.

version    mine          yours
7*         uint_least8   uint_least16
6*         uint_least16  unsigned int
5          unsigned char unsigned char
4*         unsigned char unsigned int
3          unsigned char unsigned char
2          unsigned int  unsigned int

So we have differences in 7, 6 and 4. I notice you're not explicitly checking for 4 so maybe you're don't have archives of that version.

I can't explain the differences between 7 and 6. As far as I know, your case is the last pending reported issue with this. Of course that doesn't mean much. I'll leave this open some time longer

Robert Ramey

monkeyiq — Sat, 09 Apr 2011 08:40:04 GMT

Fedora 14 just happens to include boost 1_44. I've submitted a report to get the boost in Fedora 14 patched to allow binary archives from older fedora to be read again... the bug id is 694448 on redhats bugzilla.

thomas.bretgeld@… — Mon, 30 May 2011 18:01:12 GMT

Hi again,

today I have tested my old archives again against a newly compile boost 1.46.1 filesystem library. The "Unsupported Version" error is gone, but now it fails to open the archive with the following error:

/usr/include/boost/archive/basic_archive.hpp:116: boost::archive::class_id_type::class_id_type(int): Assertion `t_ <= boost::integer_traits<base_type>::const_max' failed.

This archive opens fine with V1.42. Is this a known error? If not, what can I do track it down?

Thomas Bretgeld

attachment set

anonymous — Thu, 23 Jun 2011 17:53:08 GMT

attachment → boost-1.46.0-binary-archive-version.patch

Additional fix for reading version 6 archives

anonymous — Thu, 23 Jun 2011 17:57:14 GMT

Hi there. Here where I'm sitting, version 6 archives have only one byte of version information, which means we should handle it the same way as version <6. That's what this patch does. I tested this with boost 1.46 and it fixes the issue that I'm seeing (archive produced by boost 1.41 being read by boost 1.46-consumer).

I'm not sure if one of the interim versions of boost happened to have two-byte version-6 archives. If yes, then handling version 6 should be merged with version 7. If not, then it can be merged with handling version <6 as the patch does. In any case, separate branch for version 6 doesn't seem to be necessary.

anonymous — Thu, 29 Mar 2012 13:52:20 GMT

Can anyone tell more about the status of this bug?

Unfortunately the problem seems to persist in boost 1.49.0. An archive created with boost 1.32 can be read by 1.43 but not by 1.49.

binhvanpham@… — Thu, 29 Mar 2012 23:53:08 GMT

Hi, I encounter this issue while linking

Can't load '/home/scratch.bipham_ctg100/generic_dev/scripts/x86_64-linux/auto/nDB/nDB.so' for module nDB: /home/scratch.bipham_ctg100/generic_dev/scripts/x86_64-linux/auto/nDB/nDB.so: undefined symbol: _ZTIN5boost7archive17archive_exceptionE at /home/utils/perl-5.8.6/lib/5.8.6/x86_64-linux/XSLoader.pm line 70.

at /home/scratch.bipham_ctg100/generic_dev/scripts/x86_64-linux/nDB.pm line 56

Compilation failed in require at /home/scratch.bipham_ctg100/generic_dev/scripts/test2.pl line 11. BEGIN failed--compilation aborted at /home/scratch.bipham_ctg100/generic_dev/scripts/test2.pl line 11.

Do you know what is happening?

markus.henschel@… — Thu, 20 Sep 2012 09:33:58 GMT

Compatibility ist still broken. We have binary archives from 1.34.1 that fail to load in 1.51 now. I stepped through the code and it seems like the class_id is 2 bytes in 1.34.1 but 1.51 tries to load it as 4 bytes. See basic_binary_iarchive.hpp:

This is what the code looks like in 1.51

void load_override(class_id_type & t, int version){
    library_version_type lvt = this->get_library_version();
    if(boost::archive::library_version_type(7) < lvt){
        this->detail_common_iarchive::load_override(t, version);
    }
    else
    if(boost::archive::library_version_type(6) < lvt){
        int_least16_t x=0;
        * this->This() >> x;
        t = boost::archive::class_id_type(x);
    }
    else{
        int x=0;
        * this->This() >> x;
        t = boost::archive::class_id_type(x);
    }
}

1.34.1 has library version 4 and had this code

void load_override(class_id_type & t, int){
    // upto 32K classes
    int_least16_t x;
    * this->This() >> x;
    t = class_id_type(x);
}

So it seems to me that this is the problem. I checked the source code of releases from 1.34 to 1.40 (library version 6) and it looks like the class id has always been 16 bits. So this is was the code should look like:

void load_override(class_id_type & t, int version){
    library_version_type lvt = this->get_library_version();
    if(boost::archive::library_version_type(7) < lvt){
        this->detail_common_iarchive::load_override(t, version);
    }
    else
	{
        int_least16_t x=0;
        * this->This() >> x;
        t = boost::archive::class_id_type(x);
    }
}

serge-voropaev@… — Thu, 20 Sep 2012 09:53:25 GMT

Replying to markus.henschel@…:

Compatibility ist still broken.

.....

My advise - do not use boost.serialization. This library does not have good maintenance. You can see some discussion in boost developer mail-list. We ended use of this library in our projects.

Sergey Voropaev (Reporter of this bug)

Thu, 20 Sep 2012 15:30:49 GMT

Replying to serge-voropaev@…:

Replying to markus.henschel@…:

Compatibility ist still broken.

.....

My advise - do not use boost.serialization. This library does not have good maintenance. You can see some discussion in boost developer mail-list. We ended use of this library in our projects.

Sergey Voropaev (Reporter of this bug)

If you're going to transition, a less disruptive option is to use text archives -- which are less bug prone than binary -- that are passed through a gzip filter. That's worked very well for us.

anonymous — Thu, 29 Nov 2012 07:11:57 GMT

Unfortunately, new users won't see this advice - I doubt they'd look for something like this ticket until they stumble upon the problem themselves... In the meanwhile, it seems like nothing has changed in this library since boost 1.46.

See also this bug: https://svn.boost.org/trac/boost/ticket/5567

markus.henschel@… — Fri, 22 Feb 2013 19:54:40 GMT

I made a last effort to get this stuff fixed and did a lot of research. I have tested library versions from 1.34.1 to 1.46.1 with regards to how they store different types in binary archives. Because the main problem is that sizes for some types differ between library versions I tested how these types were serialized in different versions of the library: -version_type
-class_id_type
-class_id_reference_type
-object_id_type
-object_reference_type
-tracking_type
-collection_size_type

These files that are relevant:

boost/archive/binary_oarchive.hpp

This file contains overrides for storing the mentioned types that changed in size.

boost/archive/basic_archive.hpp

This file defined the the types mentioned above.

boost/archive/binary_iarchive.hpp

This file contains several load_override member functions that handle backwards compatibility to older archive versions.

With different versions of the library the save overrides changed as well as the definitions of the types themselves. But for every version there is one definite type that has it's memory written to the output stream buffer. So I created an excel table that lists this information and attached it.

The essence of this research is the following: a) The source of the problem started in 1.44. The documentation and the maintainers comments indicate that 1.42 changed the size of several types. This is true but it wasn't problematic for binary archives because they did explicitly override the serialization of these types. For instance class_id_type changed from int to int_least16_t but this didn't matter for the serialization of this type because the binary output archive stored it as a 16 bit type anyway. (To be precise the type used in save_override also changed but only from uint_least16_t to int_least16_t and this shouldn't usually cause any trouble unless a lot of classes are being used.)

b) 1.44.0 removed the custom save_override functions for most types but the only type that really got a different binary representation is version_type. It changed from char to uint_least32_t. At the same time a load_override function was added but it used the wrong type for older archives. Instead of unsigned char is used unsigned int. Because 1.44.0 didn't increase the archive number it cannot be distinguished from archives of 1.42.0 and 1.43.0. All have version 7.

c) Later changes fixed some things but introduced new bugs. It seems like the code tries to read all types like they are defined in basic_archive.hpp in the respective version of the library. The results are hit or miss. The problem is that the fixes seem to ignore the fact that these types had save_override functions in prior library versions that did alter the type actually stored in the data stream. So the current situation is much worse than it has to be.

d) The only "unfixable" problem is that 1.42, 1.43 and 1.44 have the same version number. Either 1.42+1.43 or 1.44 cannot be loaded.

I contacted the maintainer Robert Ramey via the boost developer mailing list and he agreed to at least take a look at the issue. I took a long time to extract all this information. I will provide a proper fix next week. Hopefully it will make it into boost so I can work with an unpatched boost version again.

attachment set

markus.henschel@… — Fri, 22 Feb 2013 19:59:11 GMT

attachment → Boost_binary_output_archive_storage_data_types.xlsx

data types stored by binary archives for various internal serialization library types

attachment set

markus.henschel@… — Mon, 25 Feb 2013 18:58:45 GMT

attachment → boost serialization binary archive compatibility fix.patch

Fix for making binary archives created by ALL versions work

markus.henschel@… — Mon, 25 Feb 2013 19:26:43 GMT

After digging even deeper into the matter and creating the promised patch I have good news. If I'm not mistaken the patch fixes compatibility of binary archives created by all versions. Even 1.42, 1.43 and 1.44 should work. There already was code that could detect if the archive version was 1 or 2 bytes. This made it possible to differentiate between 1.43 (1 byte archive id) and 1.44 (2 byte archive id). I added some comments to make clear how this works.

Because I didn't want to introduce another variable to tell if an archive with version 7 is from 1.43 or from 1.44 I increased the archive version to 8 if 1.44 was detected while loading the archive header. This should work because when 1.45 finally did increase the archive version to 8 it didn't introduce any other changes in the way it writes binary archives. I also searched the rest of the serialization code for some other places where there are checks for an archive version of 8 in some way but didn't find anything. (Even if I really overlooked something it would only lead to version 1.44 binary archives not working and a later patch could fix this by adding another variable to differentiate between 1.43 and 1.44.)

The patch is against the current trunk. I did some tests with various archives from 1.34-1.46.1 and ran the regression tests. Please test and review this patch. If it is applied the documentation also has to be updated.

Robert Ramey — Tue, 26 Feb 2013 18:20:00 GMT

a) in the course of this, did you come up with any tests which can add to the test suite to test backward capability? I would like to consider adding such tasks to the test suite.

b) How would the documentation have to be changed?

Robert Ramey

anonymous — Tue, 26 Feb 2013 19:20:27 GMT

Replying to ramey:

a) in the course of this, did you come up with any tests which can add to the test suite to test backward capability? I would like to consider adding such tasks to the test suite.

First I created a simple test app that would write a binary archive. It contained just one class for now. Then I compiled it with every boost version from 1.34 - 1-46.1 to get some test data and to debug what exactly gets written. When I had the patch ready I created another small test app that tries to read the archives back. I think this could be turned into a simple test that checks compatibility with a defined set of binary archives from different versions. I don't have access to the code right now but I can provide it. I'm not familiar with the boost regression tests yet (well, I can run them). If you wan't me to I can try to add some tests myself and add them in another patch.

b) How would the documentation have to be changed?

As far as I can see the release notes are wrong about several things. The version that started to break something was 1.44. There is also some text about a fix-six utility that changes archive version to from 7 to 6. The reference to this utility should be removed. It shouldn't be required anymore.

My fix detects archives written by version 1.44 that have an archive version of 7 and increases it to 8. I think it is possible that binary archives are being used without a header. In this mode the user has to provide the right archive number by calling set_library_version, right? So the documentation should mention that when reading back archives from 1.44 the user should manually set archive version 8.

I also found a problem report that mentions another compatibility issue with binary archives that is independent from the one my patch addresses. https://svn.boost.org/trac/boost/ticket/5579

This only affects people that split serialization across multiple dlls. I don't think this problem should be fixed in the way the user suggests there though. The solution adds more complexity and cannot work 100% without the user knowing which boost serialization version was really used. But IMHO it should be mentioned in the release notes that there is an issue with a link to the track item. This way users that have this problem can go there and find the patch.

markus.henschel@… — Wed, 27 Feb 2013 08:48:44 GMT

The anonymous post above was mine. Forgot to login.

joao@… — Fri, 01 Mar 2013 16:20:44 GMT

This patch does not fix all the binary compatibility problems. I have a file serialized with 1.35.1 that cannot be read by 1.53 when it is in binary format. Converting that file to text (compressed or not) allows the serialization to correctly read it. The file is too large to attach here, but you can contact me if a copy would be helpful.

markus.henschel@… — Fri, 01 Mar 2013 16:30:46 GMT

Replying to joao@…:

This patch does not fix all the binary compatibility problems. I have a file serialized with 1.35.1 that cannot be read by 1.53 when it is in binary format. Converting that file to text (compressed or not) allows the serialization to correctly read it. The file is too large to attach here, but you can contact me if a copy would be helpful.

Just to be sure: This patch is not in 1.53 yet. Did you apply it yourself and build boost from source?

I'm not aware of a 1.35.1 release. Was that a typo?

Otherwise I would appreciate getting the data as text and binary. i think it would also help to get the sources if possible. Can you upload to some share hoster like rapidshare?

joao@… — Fri, 01 Mar 2013 16:46:18 GMT

Sorry, typo; the earlier release was 1.34.1. And yes, both version of Boost built from source and I applied the proposed patch manually to our 1.53 source tree. The binary file is 33Mb, compressed text is 59Mb, compressed XML is 82Mb. Note also that the reading problem applies to both binary & xml files. Only the text version reads correctly in the 1.53 release (with or without the patch). Currently working through the clearance procedures at my company to see if I can release this data and/or spend the time to create a simplified version that duplicates the same issue.

anonymous — Fri, 01 Mar 2013 17:05:58 GMT

The last patch just handles issues of binary archives. If your xml archives also have a problem this might be a different bug. Do you split serialization across multiple libraries? In that case this might be this bug: https://svn.boost.org/trac/boost/ticket/5579

It would be preferable if you could make the repro case as small as possible.

Robert Ramey — Fri, 01 Mar 2013 18:32:38 GMT

Note also that the reading problem applies to both binary & xml files. Only the text version reads correctly in the 1.53 release (with or without the patch). Currently working through the clearance procedures at my company to see if I can release this data and/or spend the time to create a simplified version that duplicates the same issue.

Note that this whole issue is related to native binary archives.

If you've got problem which shows up in other archive types as well, that it's got to be a different problem. Upside, any xml problem should be easy to track down since it's easy to read the archive file. Also xml archives check that the beginning/end tags match and throw an exception when they don't.

Robert Ramey

attachment set

Fri, 01 Mar 2013 18:59:13 GMT

attachment → problem_1_34_1_serialized_file.tar.gz

Serialized files (txt, binary, xml) that can be read by 1.34.1 but only the txt file can be read in by 1.53

Fri, 01 Mar 2013 19:02:33 GMT

OK, got permission to share our data. As per attachment above, the tar.gz file contains the same data serialized by 1_34.1 in several different ways: compressed binary, text, xml. Only the txt version of the file is readable by 1.53; If the txt file is read into 1.53 and the re-generated as binary from that version, 1.53 can read its own binary version correctly. However, 1.53 can't read back in its own XML format.

anonymous — Mon, 04 Mar 2013 09:21:42 GMT

Hi,

could you please file a separate bug report? This really doesn't belong here and confuses people that search for solutions to problems with binary archives only. And if you do so I would suggest you add some more information. It would be best to have sources that read back these archives. Otherwise the least thing you could do is exactly say where the deserialization process fails. IMHO you should do some debugging first. Like Robert Ramey said before it shouldn't be that hard to find the place where the loading process starts to fail with xml archives.

Please note that I'm not the maintainer - Robert Ramey is. I just replied because I initially thought you found a problem with the patch I submitted.

markus.henschel@… — Mon, 04 Mar 2013 09:24:15 GMT

Post above has been written by me. Forgot to login (again).

Mon, 04 Mar 2013 16:24:39 GMT

I'm stunned incredulous by the attitude displayed here Markus.

I've contributed a testcase that shows that there is still a problem with the current version of boost serializer to read binary archives produced by an older version. This testcase was requested by Ramey, which I've spent time minimizing and clearing with my company to be allowed to share it with this team. In virtually every other open source project I'd be thanked for this and other people more expert with the code would dive in and diagnose what the real problem is etc etc.

Instead here you complain for my having provided you a testcase???

And once again, let me be clear as to what exactly this testcase shows:

1) binary serialization written by 1.34.1 cannot be read by 1.53

2) binary serialization written by 1.34.1 can be read by 1.34.1; likewise binary written by 1.53 can be read back by 1.53. I'd have thought it obvious that this should work, but as you'll see later, such an assumption is not legitimate in all cases.

3) text serialization written by 1.34.1 can be read by 1.34.1 and 1.53. Text archive written by 1.53 can also be read back by 1.53.

4) xml serialization written by 1.34.1 cannot be read by 1.53, but can be read back by 1.34.1.

5) xml serialization written 1.53 cannot be read back by 1.53!!! There goes that assumption that read/write within the same version and same backend format works.

6) the serialization code all lives in a single piece of code, runs in Linux 64bit; the problem here does not appear in any way related to bug 5579. But I'm not deeply familiar with the boost serializer code so I could be wrong about that.

Note that I work for a closed source company and cannot under any circumstances reveal any of the our code.

Echoing similar sentiments elsewhere, I'm seriously starting to wonder if my team has made a major mistake in using and relying on this component.

markus.henschel@… — Mon, 04 Mar 2013 17:12:41 GMT

I'm not complaining about you providing a test case. I was just asking you to create a separate bug report because your bug actually seems to differ from the one discussed here (although there is some overlap). I also think that it's quite hard to find a problem if you just have the data without the code. But this is just my personal opinion. I'm just a user of boost serialization as you are. Please do not make your opinion about boost serialization depend on my posts. As I said: I'm not the maintainer and I do not represent boost serialization. Simply wait for Robert Ramey's response.

PS: I'm not a native speaker. So if it's some wording in my post that made you upset I apologize. Sometimes it's hard to hit the right tone.

anonymous — Sat, 09 Nov 2013 08:33:12 GMT

Replying to Joao Geada <joao@…>:

Echoing similar sentiments elsewhere, I'm seriously starting to wonder if my team has made a major mistake in using and relying on this component.

YES.

My team has the same thoughts. Some time ago, we started our project with Boost 1.41, and we still can't even move to any newer version of Boost because of this bug.

To everyone planning to use Boost.Serialization: don't trust this library. Choose something else, or just invent your own local data format. Save your time and nerves.

Three years have passed and the bug is still not solved. It's a crying shame all this happens in such well-known and respected set of libraries like Boost is.

Robert Ramey — Sat, 09 Nov 2013 20:43:23 GMT

I think I've answered these questions previously in the thread. Here is a summary

a) This is a tricky problem to solve - it's it's solvable at all. I know this for a fact as I spent significant time trying to solve it and actually made things worse. Juast pasting in someone's patch might fix their problem but I can't know that ti will not break something else.

b) What the serialization library needs is a comprehensive set of tests to verify loading of archives produced by previous versions. This is a large effort which no one has resources to undertake.

c) If you want to move to a more recent of boost and preserve the data in one's previously written archives, I would recommend the following:

i) Under your current version of boost, create a small program which reads your binary archive and write a text (or portable binary archive). ii) upgrade to a more recent version of boost iii) leave the archives as text or portable binary archive or make small program as in i) above which reads the portable and writes the (non-portable)binary ones.

In the future, I will be much, much more careful about making changes to the binary_archive. If someone want's to commit resources to making a test suite to guarantee future binary portability, I would be willing to review it. That would likely be much less effort than making your own binary archive.

Note that this problem could occur even if I had never made a mistake. An upgrade to 64 bit compiler from a 32 bit could present the exact same situation. The problem really has it's root in usage of the binary archive. This is indeed useful for MPI and applications like that, as it's the fastest - but it is the fastest at the expense of coupling the format to a specific machine architecture - not a great idea for permanent data storage.

I do regret any inconvenience that my previous mistakes have caused, but this only makes less reluctant to commit changes without a lot of thought. Probably the most time I spend is finding problems cause by checkins by other people who don't really appreciate all that is involved in a library like this. Sometimes these changes get checked in even without being tested!!!

Robert Ramey

Mon, 10 Mar 2014 10:58:36 GMT

Hi all,

I understand the situation with the binary archives is tricky. However, we devices out in the wild which have configuration data written in a binary boost 1.43 archive on their eeprom. We changed our build environment from Visual Studio 2008 to 2012, which has also required a change from boost 1.43 to a newer version (1.54 in our case). Of course, we still need to support the old devices with our new software.

So, I need to solve the following problems:

The new software (VS 2012) should be able to write binary boots archives which can still be read by our old software (VS 2008/boost 1.43).
The new software (VS 2012) should be able to read old binary archives written by boost 1.43.

Is there any way I can achieve that? Unfortunately with Visual Studio 2012 I cannot link to boost 1.43 anymore. Is there sth. like a compatibility flag, so that boost 1.54 can write binary archives in the old boost 1.43 fashion? I happy for any useful hints or comments.

Thanks in advance

Alex

Robert Ramey — Mon, 10 Mar 2014 21:09:13 GMT

First of all, this thread deals with a different subject. That is it tries to address a problem whereby I introduced an incompatibility in binary_archives and we've had trouble getting over it. If you want to pursue this, you should create a new trac item.

"So, I need to solve the following problems:"

"The new software (VS 2012) should be able to write binary boots archives which can still be read by our old software (VS 2008/boost 1.43)."

I don't see why you can't using the 1.43 version of serialization with vc 2012

"The new software (VS 2012) should be able to read old binary archives written by boost 1.43."

This should be handled using the standard versioning available in all versions of the library.

Robert Ramey