Boost C++ Libraries: Ticket #3990: [serialization] data loss in version field

cc set

darose@… — Tue, 09 Mar 2010 18:10:08 GMT

cc darose@… added

status changed

Robert Ramey — Tue, 09 Mar 2010 18:43:20 GMT

status new → assigned

I'm still looking at this.

I may be able to make changes to accomodate older archives which used a version # exceeding the originally expected 8 bits. I expect to enforce the 8 bit expectation in the future to avoid these problems. So you would be left with the capability of reading old archives, but the version number stored would have to be limited to 8 bits. So class using the "old version numbers" would have to be enhanced to make the switch as they are loaded. I don't know if this would be acceptable to you.

Robert Ramey

cc changed

ck@… — Tue, 16 Mar 2010 23:50:49 GMT

cc ck@… added

smr@… — Wed, 26 May 2010 00:03:06 GMT

Robert: any news? I guess the fix didn't make it into 1.43? Is there a way for encfs to cope?

status, type changed; resolution set

Robert Ramey — Sun, 30 May 2010 16:06:10 GMT

status assigned → closed
type Bugs → Feature Requests
resolution → wontfix

I can't fix this. I'm not going to expand the version # definition beyond the originally contemplated 8 bits. I can prevent it from happening in the future and this is what's been done. That is, violations of the 8 bit limitation will be trapped as soon as possible.

So if you want maintain you're system of using dates for version numbers, you'll have to implement it yourself. This is fairly easy if you want to do it. e.g.

template<class Archive>
void serialization(Archive & ar, unsigned int version){
    // don't use the boost serialization number, use ours instead
    ar & my_date;
    ar && old_data_items
    if(ar.is_loading)
       if(my_date > "jan 10, 2010")
           ar & new_data_item;

Of course this begs the question about how to use the old archives. This can be addressed with the function get_library_version() which returns an integer indicating which version of he api is being called. The library version number associated to boost 1.42 (I believe the last one which would work for you) is 5. So the above would be enhanced to be

template<class Archive>
void serialization(Archive & ar, unsigned int version){
    library_version_type library_version = get_library_version
    if(library_version < 6)
        // the library version is a date
        my_date = version;
    else{
        // don't use the boost serialization number, use ours instead
        ar & my_date;
    }
    ar && old_data_items
    if(ar.is_loading)
       if(my_date > "jan 10, 2010")
           ar & new_data_item;

Alternatively if you want to give up on using the date as a version number you can make the transition like this:

template<class Archive>
void serialization(Archive & ar, unsigned int version){
    library_version_type library_version = get_library_version
    if(library_version < 6)
        // the library version is a date
        // look up integer version # from a table
        version = convert_date_to_version_number(version)
    }
    ar & old_data_items
    if(ar.is_loading){
        if(version > 9?)
            ar & new_data_items;
    }

I realize that this is likely not the answer you were interested in, but it DOES address the issue of handling old archives and it will transition you to a method of handling versioning consistent with the design of the boost serialization library.

Robert Ramey

smr@… — Sun, 30 May 2010 18:13:00 GMT

Thanks for the hints, Robert. I'll pass on your suggestions to the Debian users and encfs maintainers.

By the way: this broke for encfs in Boost version 1.42, so your example needs a slight modification. It is known to work for Boost 1.40; not sure about 1.41.

status, version, severity, milestone changed; resolution deleted

vgough@… — Wed, 16 Jun 2010 05:39:35 GMT

status closed → reopened
version Boost 1.42.0 → Boost 1.43.0
resolution wontfix
severity Showstopper → Problem
milestone Boost 1.43.0 → To Be Determined

The internal archive version checking gets in the way of providing backward compatibility.

BOOST_CLASS_VERSION(..., 20080816) fails at compile time as of boost 1.43. However the internal comparison is not done with 8bit values!

I could call BOOST_CLASS_VERSION(..., 255) as the largest 8bit value, but internally when boost 1.43 reads an archive with version 20080816, it truncates that to 26797, decides that is larger then 255 and throws an archive exception.

My work around has been to implement a partial specialization of iserializer for my class which eliminates the version number check in load_object_data, so that I can apply my own version checking in my load function.

Having to specialize functions in boost::archive::detail seems undesirable. Is there a supported way to stop boost from doing the archive version comparison itself?

Robert Ramey — Wed, 16 Jun 2010 06:02:54 GMT

Hmmm - did you look at using library_version as shown above? I don't see why this woudn't give you what you want. That is if library_version < 5? then use you're old system, otherwise, use your new system. I don't think you'd need to specialize anything in the archive. I believe it could be done in the serialization functions for each class.

Robert Ramey

status changed; resolution set

Robert Ramey — Wed, 07 Jul 2010 16:42:24 GMT

status reopened → closed
resolution → wontfix

Guess I have to close this now.