boost::pool_allocator for vector of vectors exhausts memory

[mattiasg]

glr9940 reported in #386 that using boost::pool_allocator for a vector of vectors caused problems. I have now found a slight extension to the example glr9940 reported which does not crash, but exhausts memory even for a tiny vector of vectors.

I have used the development trunk code and gcc 4.1.2.

#include <boost/pool/pool_alloc.hpp>
#include <vector>
#include <iostream>

typedef std::vector<int, boost::pool_allocator<int> > EventVector;
typedef std::vector<EventVector> IndexVector;

int main() 
{
  IndexVector iv;
  int limit = 100;
  for (int i = 0; i < limit; ++i) 
    iv.push_back(EventVector());

  std::cout << "it works\n";
  return 0;
}

I suspect that the critical value for 'limit' depends on the machine. On my machine a limit of 20 seems to work fine, a limit of 24 uses a few Gb of memory, while a limit of 30 stalls the process, 'top' showing that the memory footprint doubles rapidly until memory is exhausted. I've noticed for other examples of this issue that when I replace the vector push_back loop with a rolled out version just creating a number of pooled vectors, it works fine.

Have I made some noob mistake, or is this a bug?

[Steven Watanabe]

I can't reproduce the problem with gcc 4.4.1, or with MSVC 10. I'd guess that it has something to do with the standard library implementation of vector, though.

Can you try applying the patch to singleton_pool which I'm about to attach, run the following variation and save the output? It should produce code that you can compile which demonstrates the problem. Check that it actually triggers the problem, and attach it.

#include <boost/pool/pool_alloc.hpp>
#include <vector>
#include <iostream>

typedef std::vector<int, boost::pool_allocator<int> > EventVector;
typedef std::vector<EventVector> IndexVector;

int main() 
{
  std::cout << "#include <boost/pool/pool_alloc.hpp>\n";
  std::cout << "int main() {\n";
  std::cout << "void* tmp;\n";
  {
    IndexVector iv;
    int limit = 30;
    for (int i = 0; i < limit; ++i) 
      iv.push_back(EventVector());
  }
  std::cout << "}\n";
  return 0;
}

[Steven Watanabe]

Patch adding instrumentation to singleton_pool

[mattiasg]

Output of the modified test program with the patched singleton_pool.hpp

[mattiasg]

On a Suse 11 machine with gcc 4.3.4 I cannot reproduce the problem, but on a RHEL5 machine with gcc 4.1.2 a run with your patched singleton_pool.hpp produces the output which I've attached to this issue. To avoid crashing the server I ran the test on I had to abort the program around the time it used half (~16Gb) of the available memory.

[Steven Watanabe]

Sorry for the delay. The file you attached is an executable. I don't think it's the output.

[mattiasg]

Proper output file

[mattiasg]

Ahhh, crap. Attached the output file now, sorry.

[John Maddock]

I can reproduce this with gcc-4.0.4 as well. Strange bug!

[John Maddock]

Update:

This is a result of two bugs:

1) std::vector invokes undefined behaviour by calling Allocator::allocate(0). This has been fixed in more recent GCC versions. 2) Boost.Pool, when it tries to allocate 0 chunks thinks allocation has failed (a bug), so allocates a whole new block, *and* ups the size of the next block to allocate by a factor of 2. So just a few allocation requests for 0 chunks blows up the heap.

This will be fixed in the sandbox version of Boost.Pool soon.

[John Maddock]

(In [68920]) Add test case for bug #4960. Add conditional instrumentation code to headers so we can see what's being allocated when. Fix issue #4960 by allowing pool to allocate 0 chunks. Refs #4960.

[John Maddock]

(In [73495]) Merge updated Pool lib from trunk. Fixes #1252. Fixes #2696. Fixes #4960. Fixes #5526. Fixes #5568. Fixes #5700.