windows_shared_memory implicitly commits virtual memory

[Jason Sachs <jsachs@…>]

​http://svn.boost.org/svn/boost/trunk/boost/interprocess/windows_shared_memory.hpp windows_shared_memory::priv_open_or_create() is the function that calls CreateFileMapping() to reserve a segment of virtual memory. It does not specify either SEC_RESERVE or SEC_COMMIT, and the docs for CreateFileMapping() say that SEC_COMMIT is assumed.

Either this is a small bug and SEC_COMMIT should be specified explicitly (w/ no change in behavior or other code needed), or there is a larger problem and SEC_RESERVE should be used instead.

The problem with SEC_COMMIT is as follows. Suppose you have a N processes where each process i=0,1,...N-1 is going to need a pool of related memory with a maximum usage of sz[i] bytes, where sz[i] is not known beforehand but is guaranteed less than some maximum M, and has a mean expected value of m where m is much smaller than M. From a programmer's standpoint, the best way to handle this would be to reserve a single shared memory segment and ask Boost::interprocess to make the segment size equal to M.

But if I use the existing boost libs to do this, then because SEC_COMMIT is used, my total system resource usage for virtual memory is N*M (instead of sum(sz[i]) which is likely to be much smaller).

SEC_RESERVE should be used instead. This is more complicated because then you need to do VirtualAlloc or something later to actually commit the memory on-demand (in fact I'm not quite sure how this would have to work esp. since I'm unfamiliar w/ the inner workings of Boost::interprocess).

as an aside, it would be helpful for me to find out sooner rather than later whether this is considered a valid bug that can be addressed in the coming months; otherwise I would have to change my architecture + use multiple shared memory segments in order to make more efficient use of the limited total system resource of virtual memory.

[Ion Gaztañaga]

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

As mentioned, an alternative would be to use several managed segments created on demand by the user.

[Jason Sachs <jsachs@…>]

Replying to igaztanaga:

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

ok but it would help to explicitly specify SEC_COMMIT in the code + to mention this in the documentation, so that when deciding the memory segment size, one is aware of the resource costs.

[Jason Sachs <jsachs@…>]

Replying to igaztanaga:

I don't consider this a bug, but a feature. Managing pages on-demand is not a feature currently provided by Interprocess and I find difficult this to be managed by Interprocess.

As mentioned, an alternative would be to use several managed segments created on demand by the user.

I hope you won't take offense if I reopen this. I do agree that managing arbitrary pages on demand is something unreasonable. However, the problem I'm facing, and the only thing I'm looking for a resolution, is the implications of (a) having to specify a fixed maximum size for the segment, despite possible/probable usage that is much smaller and (b) the Boost library's only option is to commit all that virtual memory at once in Windows.

I've looked around a little bit at the source files for boost::interprocess and it looks like there some facilities in ​http://svn.boost.org/svn/boost/trunk/boost/interprocess/detail/managed_memory_impl.hpp to grow() file sizes. The memory reserved by CreateFileMapping/MapViewOfFile can't grow, but what if there were a separate "size" and "max_size" in ​http://svn.boost.org/svn/boost/trunk/boost/interprocess/mapped_region.hpp ? (which appears to be the class that calls MapViewOfFile via map_view_of_file_ex()) "max_size" would represent the argument passed to CreateFileMapping (with the SEC_RESERVE flag) / MapViewOfFile, and "size" representing the portion of that memory allocated; calls to grow() would call VirtualAlloc, not in a haphazard random-access per-page way, but merely to increase the size used by the shared memory segment, in a sequential, linear (or exponential if that makes sense) fashion.

[anonymous]

This feature won't be implemented because of complexity and time constraints.