Serialization library in Boost 1.45 is unable to read archive created with Boost 1.39

[Rüdiger Brünner <rbruenner@…>]

The way the serialization library is handling object version and property information has obviously changed so that an attempt to read an archive with a simple object fails with version 1.45.

The archive header is correctly read in (as opposed to 1.44) but an attempt to read a simple object fails because the version 1.45 is reading too much data.

[Rüdiger Brünner <rbruenner@…>]

The attached VS 2010 project demonstrates the problem

[Rüdiger Brünner <rbruenner@…>]

Be sure to adjust the path to the boost library properly in the project settings before you build the solution.

The project contains two archives, Test139.bin and Test145.bin. Run the programme and type "l" to load both archives. Test139.bin will fail to load.

Maybe, the new version of the serialization library requires some setting for the class in question. Otherwise, I am stuck to version 1.39.

Any help is appreciated.

Best regards

[keckmatd@…]

I'm suffering from the same issue, with the same results, i.e. simple objects are not read correctly. I'm coming from 1.38, but error manifests the same way.

[Robert Ramey]

Is this the 1.45 beta or official release. I think I got a change into late for the beta but that it made it into the release?

Robert Ramey

[keckmatd@…]

I can verify this as of today, with 1.45 release also downloaded from the site today.

[keckmatd@…]

Replying to keckmatd@…:

I can verify this as of today, with 1.45 release also downloaded from the site today.

I can verify this is still a problem, rather, with 1.45 official release.

[Robert Ramey]

how about sending a small file which you're having trouble with. I would like to know if the problem is when one reads the header, or some other part further down in the file

Robert Ramey

[Rüdiger Brünner <rbruenner@…>]

Please find the attached project TestSerial.7z. It contains two binary files, Test139.bin which was created using version 1.39, and Test145.bin created using 1.45. It is currently configured to use the 1.45 version. When the Test139.bin file is loaded, the preamble is read correctly. The version number 5 is determined and the magic values for sizeof(int) etc. are loaded correctly. However, the new version reads too much bytes for the object header. The first item to load is a string. Its string length (32 bit) is composed of the last byte of the actual string length and the first three bytes of the string.

[Rüdiger Brünner <rbruenner@…>]

Demonstration of binary layout differences causing the problem

[keckmatd@…]

Is there any status update on this? I realize everyone is probably busy, but this is a pretty big showstopper for those of us stuck at earlier versions.

If I can be of any help, although I'm seeing the same thing Rüdiger is seeing, please let me know.

[Robert Ramey]

I've been struggling with this. Here is the output from test_simple in xml format

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE boost_serialization>
<boost_serialization signature="serialization::archive" version="8">
<a class_id="0" tracking_level="0" version="0">
	<b>1</b>
	<f>757147</f>
	<g>167851000</g>
	<l>2</l>
	<m>15724</m>
	<n>11478</n>
	<o>29358</o>
	<p>26962</p>
	<q>24464</q>
	<r>5705</r>
	<c>-15</c>
	<s>-15</s>
	<t>187</t>
	<u>9961</u>
	<v>491</v>
	<w>2995</w>
	<x>11942</x>
	<y>uirxnrvpfehwtrbifmqaylbxblotupkjewsfcgbupvdkwfravemvnrnfpnuxlk</y>
	<z>n</z>
</a>
</boost_serialization>

I've looked carefully at Serialisation #4903.pdf and see you're comments - this is very helpful. BUT there are several possible alternative explanations:

Either class_id, tracking_level or version expanded from on byte to 4 bytes. My suspicion is that the culprit is tracking_level. The whole problem came about when I tweaked some types to eliminate some warnings. This changed the size of these types which then changed the archive format. This looks easy now, but the whole issue was obscured by a number of things like implicit conversions. Turns out that I never touched tracking_type. And there is confusing point on this as well. The type trait is an enum of 3 values - the usage in the file is bool - indicated whether that particular type has been tracked in that file. So I believe that something is mixed up here - unfortunately it's very, very hard to pin down since i can't build archives in older formats without re-installing and building old library versions.

So at this moment I'm stuck - I don't know if this makes anyone feel better, but there it is.

Robert Ramey

[Rüdiger Brünner <rbruenner@…>]

Updated with a version number

[Rüdiger Brünner <rbruenner@…>]

Replying to ramey:

I've been struggling with this. Here is the output from test_simple in xml format

Thanks for looking into the issue. However, I believe that it is the version number which has got expanded to 4 bytes. I have attached two new binary files with the same structure, both in 1.39 and 1.45. The object has now the version number 37 (0x25). In the binary file, it can be clearly seen that the version number is stored in 4 bytes in version 1.45. When reading an archive created with 1.39 the library, although it read the archive version 5 correctly, wrongly assumes to be able to read a 4-byte version number instead of a 1-byte number (as stored by 1.39). I think it is necessary to split the version number reading function based on the archive version number. For archives up to 1.39 (or 1.40 or so) it must read only one byte, newer archives must read 4 bytes.

Rüdiger

[Robert Ramey]

looking your PDF file

In BOTH versions the version # is in position 0x1a. In the 1.39 version it's equal to 0x05 and in the 1.45 version it's equal to 0x08. This is what I expect. In 1.39 the version # is just one byte while in 1.45 it's two bytes (0x0800). Then come the sizes of the types. In both cases it's 0x04 0x04 0x04 0x08 0x01. Then on 1.39 it's followed by 5 x 0x00 while in 1.4 it's followed by 8 x 0x00. This is after the header has been processed. On my test file, this number is the boolean flag. In 1.39 this is one byte while on 1.45 this is one byte. I believe this is the error.

Robert Ramey

[Robert Ramey]

I've updated the trunk in an attempt to fix this problem. Please check this out and let me know if it addresses the situation.

Robert Ramey

[keckmatd@…]

After rebuilding from the trunk with your latest patch, Mr. Ramey, I am still seeing the problem with archives before 1.42 not being able to be consumed properly. I am most definitely still seeing the problem in the header. I will try to see if I'm still seeing the problem in the same way as before when I have a moment.

But, regardless, for me at least, this has not remedied the issue for older archives. If it makes a difference, I'm running on MSVC 9.0.

[Robert Ramey]

OK - looks like you're on to something here. I'm testing the following now:

    void load_override(class_id_type & t, int version){
        library_version_type lvt = this->get_library_version();
        if(boost::archive::library_version_type(7) < lvt){
            this->detail_common_iarchive::load_override(t, version);
        }
        else
        if(boost::archive::library_version_type(6) < lvt){
            int_least16_t x=0;
            * this->This() >> x;
            t = boost::archive::class_id_type(x);
        }
        else
        if(boost::archive::library_version_type(3) < lvt){
        {
            int x=0;
            * this->This() >> x;
            t = boost::archive::class_id_type(x);
        }
        else{
            // upto 255 versions
            unsigned char x=0;
            * this->This() >> x;
            t = version_type(x);
        }
    }

This should be a more general solution.

Robert Ramey

[Robert Ramey]

Whoops - ignore the above - I meant to paste that into a different track item.

I'm testing another change. It should be in the trunk by monday.

Robert Ramey

[Rüdiger Brünner <rbruenner@…>]

Hello Robert,

thank you for looking into the issue. Now, it seems that serialising objects is working. However, serialisation through pointers crashes the app.

I am going to attach a VS 2010 project that shows the issue with Boost 1.46.

Rüdiger

[Rüdiger Brünner <rbruenner@…>]

Sample project showing the issue

[Rüdiger Brünner <rbruenner@…>]

Hello Robert,

the attached project can be compiled using different versions of boost by changing the Boost.props file.

When the app is built with ver 1.39 it can successfully read the included file 0.ref which was created using 1.39. However, building with 1.46 crashes the app. The serialisation of the plain object works now, but serialising through a pointer aborts the app.

Any suggestions?

Rüdiger

[keckmatd@…]

Robert,

I can again confirm that I am seeing the same problem as Rüdiger. If we can be of any help to solving this, let us know. This is most definitely a showstopper for us moving forward to a different boost version without resorting to something like converting massive amounts of saved archives in a tool that reads the information as 1.38 archives and writes them back out as 1.46 archives.

Thanks, Matthew

[keckmatd@…]

Robert,

Any update on this? We are literally stuck almost ten versions behind now in boost. We'd like to get away from the old archives and move forward, but it's just something at this point we can't migrate away from.

Thanks, Matthew

[Rüdiger Brünner <rbruenner@…>]

Hello all,

just FYI, I downloaded version 1.48 today and built the TestSerial project with this version. Unfortunately, the behaviour is the same as in 1.46. So, the issue is still unresolved.

Rüdiger

[Rüdiger Brünner <rbruenner@…>]

Ripped stand-alone serialization library

[Rüdiger Brünner <rbruenner@…>]

Hello,

please find the attached AltSerial.7z file. It contains an MSVC library project of the 1.39 serialization library. I have renamed the namespaces to boost::alt_serialization and boost::alt_archive, respectively. In a preliminary test, using this library, I was able to load my old archives in a test project which is otherwise using boost 1.48. It is important to a) replace all occurrences of the old namespaces in the client code with alt_serialization and alt_archive, respectively. b) pay attention that the headers of the AltSerial library must go BEFORE including any of the other boost includes. This will effectively hide the 1.48 headers from client code. The names of the header files (as well as their include guards) are the same as in boost, so it is sufficient to e. g. change the statement

#include <boost/serialization/export.hpp>

to

#include "serialization/export.hpp"

(note the dropping of the boost folder name and the usage of quotation marks).

Maybe, this information is of help for anyone having trouble with the binary serialization.

Please note that I did not fully test the library yet. You may use it as a starting point for your own evaluations.

Best regards Rüdiger