Boost C++ Libraries: Ticket #4903: Serialization library in Boost 1.45 is unable to read archive created with Boost 1.39

attachment set

Mon, 29 Nov 2010 09:39:04 GMT

attachment → TestSerial.7z

The attached VS 2010 project demonstrates the problem

Mon, 29 Nov 2010 09:48:45 GMT

Be sure to adjust the path to the boost library properly in the project settings before you build the solution.

The project contains two archives, Test139.bin and Test145.bin. Run the programme and type "l" to load both archives. Test139.bin will fail to load.

Maybe, the new version of the serialization library requires some setting for the class in question. Otherwise, I am stuck to version 1.39.

Any help is appreciated.

Best regards

keckmatd@… — Mon, 29 Nov 2010 14:29:27 GMT

I'm suffering from the same issue, with the same results, i.e. simple objects are not read correctly. I'm coming from 1.38, but error manifests the same way.

component changed; owner set

viboes — Mon, 29 Nov 2010 15:51:51 GMT

owner set to Robert Ramey
component None → serialization

Robert Ramey — Mon, 29 Nov 2010 20:44:08 GMT

Is this the 1.45 beta or official release. I think I got a change into late for the beta but that it made it into the release?

Robert Ramey

keckmatd@… — Mon, 29 Nov 2010 20:45:44 GMT

I can verify this as of today, with 1.45 release also downloaded from the site today.

keckmatd@… — Mon, 29 Nov 2010 21:34:35 GMT

Replying to keckmatd@…:

I can verify this as of today, with 1.45 release also downloaded from the site today.

I can verify this is still a problem, rather, with 1.45 official release.

Robert Ramey — Mon, 29 Nov 2010 22:47:29 GMT

how about sending a small file which you're having trouble with. I would like to know if the problem is when one reads the header, or some other part further down in the file

Robert Ramey

Tue, 30 Nov 2010 06:52:37 GMT

Please find the attached project TestSerial.7z. It contains two binary files, Test139.bin which was created using version 1.39, and Test145.bin created using 1.45. It is currently configured to use the 1.45 version. When the Test139.bin file is loaded, the preamble is read correctly. The version number 5 is determined and the magic values for sizeof(int) etc. are loaded correctly. However, the new version reads too much bytes for the object header. The first item to load is a string. Its string length (32 bit) is composed of the last byte of the actual string length and the first three bytes of the string.

attachment set

Tue, 30 Nov 2010 06:56:55 GMT

attachment → Serialisation #4903.pdf

Demonstration of binary layout differences causing the problem

keckmatd@… — Mon, 13 Dec 2010 18:05:54 GMT

Is there any status update on this? I realize everyone is probably busy, but this is a pretty big showstopper for those of us stuck at earlier versions.

If I can be of any help, although I'm seeing the same thing Rüdiger is seeing, please let me know.

Robert Ramey — Sat, 18 Dec 2010 23:20:23 GMT

I've been struggling with this. Here is the output from test_simple in xml format

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE boost_serialization>
<boost_serialization signature="serialization::archive" version="8">
<a class_id="0" tracking_level="0" version="0">
	<b>1</b>
	<f>757147</f>
	<g>167851000</g>
	<l>2</l>
	<m>15724</m>
	<n>11478</n>
	<o>29358</o>
	<p>26962</p>
	<q>24464</q>
	<r>5705</r>
	<c>-15</c>
	<s>-15</s>
	<t>187</t>
	<u>9961</u>
	<v>491</v>
	<w>2995</w>
	<x>11942</x>
	<y>uirxnrvpfehwtrbifmqaylbxblotupkjewsfcgbupvdkwfravemvnrnfpnuxlk</y>
	<z>n</z>
</a>
</boost_serialization>

I've looked carefully at Serialisation #4903.pdf and see you're comments - this is very helpful. BUT there are several possible alternative explanations:

Either class_id, tracking_level or version expanded from on byte to 4 bytes. My suspicion is that the culprit is tracking_level. The whole problem came about when I tweaked some types to eliminate some warnings. This changed the size of these types which then changed the archive format. This looks easy now, but the whole issue was obscured by a number of things like implicit conversions. Turns out that I never touched tracking_type. And there is confusing point on this as well. The type trait is an enum of 3 values - the usage in the file is bool - indicated whether that particular type has been tracked in that file. So I believe that something is mixed up here - unfortunately it's very, very hard to pin down since i can't build archives in older formats without re-installing and building old library versions.

So at this moment I'm stuck - I don't know if this makes anyone feel better, but there it is.

Robert Ramey

attachment set

Tue, 21 Dec 2010 08:20:58 GMT

attachment → TestSerial versioned.zip

Updated with a version number

Tue, 21 Dec 2010 08:34:31 GMT

Replying to ramey:

I've been struggling with this. Here is the output from test_simple in xml format

Thanks for looking into the issue. However, I believe that it is the version number which has got expanded to 4 bytes. I have attached two new binary files with the same structure, both in 1.39 and 1.45. The object has now the version number 37 (0x25). In the binary file, it can be clearly seen that the version number is stored in 4 bytes in version 1.45. When reading an archive created with 1.39 the library, although it read the archive version 5 correctly, wrongly assumes to be able to read a 4-byte version number instead of a 1-byte number (as stored by 1.39). I think it is necessary to split the version number reading function based on the archive version number. For archives up to 1.39 (or 1.40 or so) it must read only one byte, newer archives must read 4 bytes.

Rüdiger

Robert Ramey — Thu, 23 Dec 2010 20:55:11 GMT

looking your PDF file

In BOTH versions the version # is in position 0x1a. In the 1.39 version it's equal to 0x05 and in the 1.45 version it's equal to 0x08. This is what I expect. In 1.39 the version # is just one byte while in 1.45 it's two bytes (0x0800). Then come the sizes of the types. In both cases it's 0x04 0x04 0x04 0x08 0x01. Then on 1.39 it's followed by 5 x 0x00 while in 1.4 it's followed by 8 x 0x00. This is after the header has been processed. On my test file, this number is the boolean flag. In 1.39 this is one byte while on 1.45 this is one byte. I believe this is the error.

Robert Ramey

Robert Ramey — Thu, 23 Dec 2010 22:40:13 GMT

I've updated the trunk in an attempt to fix this problem. Please check this out and let me know if it addresses the situation.

Robert Ramey

keckmatd@… — Tue, 11 Jan 2011 14:47:03 GMT

After rebuilding from the trunk with your latest patch, Mr. Ramey, I am still seeing the problem with archives before 1.42 not being able to be consumed properly. I am most definitely still seeing the problem in the header. I will try to see if I'm still seeing the problem in the same way as before when I have a moment.

But, regardless, for me at least, this has not remedied the issue for older archives. If it makes a difference, I'm running on MSVC 9.0.

Robert Ramey — Sun, 23 Jan 2011 02:50:04 GMT

OK - looks like you're on to something here. I'm testing the following now:

    void load_override(class_id_type & t, int version){
        library_version_type lvt = this->get_library_version();
        if(boost::archive::library_version_type(7) < lvt){
            this->detail_common_iarchive::load_override(t, version);
        }
        else
        if(boost::archive::library_version_type(6) < lvt){
            int_least16_t x=0;
            * this->This() >> x;
            t = boost::archive::class_id_type(x);
        }
        else
        if(boost::archive::library_version_type(3) < lvt){
        {
            int x=0;
            * this->This() >> x;
            t = boost::archive::class_id_type(x);
        }
        else{
            // upto 255 versions
            unsigned char x=0;
            * this->This() >> x;
            t = version_type(x);
        }
    }

This should be a more general solution.

Robert Ramey

Robert Ramey — Sun, 23 Jan 2011 03:22:00 GMT

Whoops - ignore the above - I meant to paste that into a different track item.

I'm testing another change. It should be in the trunk by monday.

Robert Ramey

version changed

Fri, 25 Feb 2011 15:43:12 GMT

version Boost 1.45.0 → Boost 1.46.0

Fri, 25 Feb 2011 15:44:50 GMT

Hello Robert,

thank you for looking into the issue. Now, it seems that serialising objects is working. However, serialisation through pointers crashes the app.

I am going to attach a VS 2010 project that shows the issue with Boost 1.46.

Rüdiger

attachment set

Fri, 25 Feb 2011 15:45:59 GMT

attachment → TestSerial.2.7z

Sample project showing the issue

Fri, 25 Feb 2011 15:49:53 GMT

Hello Robert,

the attached project can be compiled using different versions of boost by changing the Boost.props file.

When the app is built with ver 1.39 it can successfully read the included file 0.ref which was created using 1.39. However, building with 1.46 crashes the app. The serialisation of the plain object works now, but serialising through a pointer aborts the app.

Any suggestions?

Rüdiger

keckmatd@… — Fri, 04 Mar 2011 14:39:32 GMT

Robert,

I can again confirm that I am seeing the same problem as Rüdiger. If we can be of any help to solving this, let us know. This is most definitely a showstopper for us moving forward to a different boost version without resorting to something like converting massive amounts of saved archives in a tool that reads the information as 1.38 archives and writes them back out as 1.46 archives.

Thanks, Matthew

keckmatd@… — Wed, 28 Sep 2011 14:23:29 GMT

Robert,

Any update on this? We are literally stuck almost ten versions behind now in boost. We'd like to get away from the old archives and move forward, but it's just something at this point we can't migrate away from.

Thanks, Matthew

Thu, 17 Nov 2011 08:43:24 GMT

Hello all,

just FYI, I downloaded version 1.48 today and built the TestSerial project with this version. Unfortunately, the behaviour is the same as in 1.46. So, the issue is still unresolved.

Rüdiger

attachment set

Thu, 17 Nov 2011 12:25:03 GMT

attachment → AltSerial.7z

Ripped stand-alone serialization library

Thu, 17 Nov 2011 12:38:58 GMT

Hello,

please find the attached AltSerial.7z file. It contains an MSVC library project of the 1.39 serialization library. I have renamed the namespaces to boost::alt_serialization and boost::alt_archive, respectively. In a preliminary test, using this library, I was able to load my old archives in a test project which is otherwise using boost 1.48. It is important to a) replace all occurrences of the old namespaces in the client code with alt_serialization and alt_archive, respectively. b) pay attention that the headers of the AltSerial library must go BEFORE including any of the other boost includes. This will effectively hide the 1.48 headers from client code. The names of the header files (as well as their include guards) are the same as in boost, so it is sufficient to e. g. change the statement

#include <boost/serialization/export.hpp>

#include "serialization/export.hpp"

(note the dropping of the boost folder name and the usage of quotation marks).

Maybe, this information is of help for anyone having trouble with the binary serialization.

Please note that I did not fully test the library yet. You may use it as a starting point for your own evaluations.

Best regards Rüdiger