boost::iostreams::gzip_decompressor silently ignores multiple members

[Bruce MacDonald <bruce@…>]

The gzip file format RFC 1952 allows for the concatenation of multiple "members" in one gzip file. I have a provider who, unfortunately, sends me gzipped files which consist of about 8 500M (uncompressed) members. Why they are doing this I don't know, but this seems to be a long standing weirdness with gzip. The gzip command line utility simply decompresses all the members in a single stream and this is the behaviour I would expect from gzip_decompressor.

The gzip_decompressor only processes the first member and silently ignores the others. In fact, the implementation of read_footer attempts to slurp the rest of the compressed file into a string which it then discards.

In order to read the rest of the members all we have to do is read and process the actual trailer (8 bytes) and then recursively process the rest of the input (perhaps after invoking close() on ourself?).

I have attempted to write a fix for this myself but have been defeated by the complexity of the library.

[Bruce MacDonald <bruce@…>]

Toy example gzip file containing two strings.

[Jonathan Turkanis]

Yes, the implementation is tricky because when the end of a deflated sequence is reached, the symmetric filter will usually have some unconsumed characters in the buffer, which need to be fed back through the decompressor as part of the next member.

Now that I've figured out the problem, it shouldn't be too hard to implement.

[Jonathan Turkanis]

(In [46001]) added support for archives with multiple members; added tests for metadata and for multiple members (fixes #1896)

[come.raczy@…]

I doesn't look like the change 46001 ​https://svn.boost.org/trac/boost/changeset/46001 made it into boost_1_36_0 or any other release

Since the change seems to work, would it be possible to push it into 1.41.0?

[Steven Watanabe]

iostreams was fully merged to the release branch in [56830].

[bruce@…]

Yes, but multiple members don't work in 1.43.0. An exception is thrown during footer processing on the second member as the crc's don't match.

I have fixed this locally by adding:

crc_ = 0;

in zlib_base::reset() on line 145 of libs/iostreams/src/zlib.cpp.

[Steven Watanabe]

I fixed the crc problem a few weeks ago.