Boost C++ Libraries: Ticket #4827: Windows vs POSIX interface distinction seems unnecessary

Ion Gaztañaga — Tue, 09 Nov 2010 16:57:49 GMT

Sorry for the lack of contest but I would need more details on how to add kernel-lifetime shared memory with Native NT API.

Domagoj Šarić — Thu, 11 Nov 2010 22:18:45 GMT

AFAICT it should be enough to call NtCreateSection with OBJECT_ATTRIBUTES::Attributes

(http://msdn.microsoft.com/en-us/library/ff557749(VS.85).aspx) set to OBJ_PERMANENT...

Also look into: NtMapViewOfSection NtMakeTemporaryObject NtQuerySection NtExtendSection ...

See http://undocumented.ntinternals.net ...

Also, sometimes good/better information can be found in driver oriented documentation that use the Zw-prefixed functions (e.g. http://www.osronline.com/ddkx/kmarch/k111_4oc2.htm) which are equivalent to the Nt ones (but for kernel mode code)...

ps. as mentioned in the original post, I'm willing to help...what I primarily want is a no-bloat/STL-uncluttered shared memory/mapped memory implementation (i.e. that 'looks' and 'acts' 'C++ kosher' but is optimizer-transparent enough so that it compiles to code nearly identical as if the OS calls were manually/hand coded, as was demonstrated in the post linked to in the trac ticket 4234)...so if you want I can try and provide such a collection of lower level layer primitives (that I've, as mentioned, already started developing for GIL.IO) which you can then wrap as needed...

pps. this is a fork of the original thread: http://lists.boost.org/Archives/boost/2010/10/172227.php

Ion Gaztañaga — Fri, 12 Nov 2010 16:29:48 GMT

I think OBJ_PERMANENT requires privileges (SeCreatePermanent or something similar) and if pagefile backed shared memory is created, then shared memory naming must conform global/local rules, and only privileged (SeCreateGlobalPrivilege) processes (e.g. services) can create such global names to be shared across processes from different sessions/users.

For IO in mapped files if you use Interprocess you only need to hold a mapped_region, which only a few pointers and size_t-s.

attachment set

Domagoj Šarić — Sat, 20 Nov 2010 10:34:02 GMT

attachment → native_nt_permanent_section.cpp

Domagoj Šarić — Sat, 20 Nov 2010 10:47:29 GMT

Hi, yes OBJ_PERMANENT does require both SeCreatePermanent and SeCreateGlobalPrivilege. SeCreateGlobalPrivilege is IMO not a problem as:

administrators and services have it by default, which I consider a 'normal'/reasonable requirement (admin/service/daemon level access) in a 'secure' multi-user OS for any sort of global namespace 'pollution'
is required even by a Win32 solution (when not emulated with files ofcourse)...

SeCreatePermanent OTOH is by default held only by (services running under) LocalSystem, however admins can gain it through impersonation. Considering that I can still imagine use cases/need for kernel-lifetime named shared objects only for special service/daemon types of processes and maybe special server-like applications (that one can reasonable expect to run with admin privileges) I'd say that IMO this still does not constitute a real problem.

I've cobbled (and attached) a demonstration of the above and verified that it works on Win7 pro x64, Win7 home x64 and WinXP pro x86... ;)

As argued in the first post on the boost.devel thread, regardless of the chosen underlying implementation, it is IMO wrong to make the majority of users/use cases (that do not require kernel lifetime shared objects capabilities) pay for the requirements of the minority by designing the interface (and then paying for it both in terms of 'fat' in the implementation and 'platform specific clutter' in the interface) in such a way as to always require/offer only kernel lifetime semantics... AFAICT an interface that would offer both scoped lifetime and kernel lifetime semantices would not suffer (or atleast suffer less) from the mentioned issues...

Ad overhead:

sizeof( mapped_region ) is 6 * sizeof( void * ) which is 5 (or 4 if you want to remember the size) sizeof( void * ) more than necessary (check in MSDN, it explicitly says that it is not necessary to hold on to the mapping handle but only the pointer returned by MapViewOfFile()...)...However this is less of a concern when compared to other 'unnecessary' things one has to pay for to get to/construct a mapped_region...i.e. for constructing a file_mapping one has to pay for a std::string, interprocess-to-win32 flags translation, exceptions, granularity adjustment... This sort of 'fat' if not obvious from source can be readily seen by stepping through assembly in release mode...or, as demonstrated in the post linked to in the original trac ticket 4234, comparing the sizes of statically linked binaries, VC 10 /Oxs:

int main( int /*argc*/, char * * /*argv*/ ) { return 0; } ~ 31 kB

int main( int /*argc*/, char * * /*argv*/ ) {

using namespace boost::interprocess; mapped_region region( file_mapping( "test", read_only ), read_only, 0, 0 ); return region.get_size();

} ~ 49 kB

~18 kB difference for what is otherwise a half a dozen OS API calls is IMO a bit too much (alhtough still a lot better than Boost.IOStreams)...

ps. maybe we should move this discussion back to boost.devel (considering the size of the posts)... ;)

Domagoj Šarić — Mon, 22 Nov 2010 17:35:32 GMT

'ps.' If you perhaps have not come across it already http://technet.microsoft.com/en-us/sysinternals/bb896657.aspx is a nice aid for testing memory mapping code on Windows ;)

status changed; resolution set

Ion Gaztañaga — Fri, 01 Apr 2011 17:53:11 GMT

status new → closed
resolution → invalid

Sorry, but this solution is not a general solution due to the need of priviledges.

status changed; resolution deleted

Domagoj Šarić — Sat, 02 Apr 2011 09:49:57 GMT

status closed → reopened
resolution invalid

I'm sorry but this is IMO a too quick/simple dismissal of the presented arguments. The issue identified by the ticked is the (possibly) unnecessary Windows vs POSIX interface distinction. In the linked-to boost.devel thread a proposal is given on how the _interface_ could be redesigned to erase this distinction and in the same thread and the comments for this ticked two proposals for the _implementation_ of this change are given, one of them being the Native NT API solution.

Dismissing one implementation proposal does not invalidate the entire ticket. Further more, the dismissal of the one implementation proposal (the Native NT API one) is not validly argued. IOW why would the need of admin priviledges invalidate this solution? For this limitation to be enough to invalidate the proposal one would have to show that there is actual need/an actuall use case for kernel-life time memory mapped objects for non-admin users on the Windows platform. The very fact that the Windows platform explicitly does not allow this hints that probably this is not the way these things are done on that platform...

Ion Gaztañaga — Sat, 02 Apr 2011 10:55:50 GMT

Sorry, but IMO your reply is too quick. Several users have reported bugs regarding non-administrative user permission issues, so shared memory is definitely needed for non-administrative users.

Maybe we should offer windows-centric new feature for permanent shared memory, but that's another story. Interprocess shared_memory_object is about portability. windows_shared_memory might be another issue.

Domagoj Šarić — Sat, 02 Apr 2011 17:28:25 GMT

Replying to igaztanaga:

Sorry, but IMO your reply is too quick. Several users have reported bugs regarding non-administrative user permission issues, so shared memory is definitely needed for non-administrative users.

Shared memory as such yes, but creation of global, kernel lifetime shared memory..is this also really required for non-admin users?

Maybe we should offer windows-centric new feature for permanent shared memory, but that's another story. Interprocess shared_memory_object is about portability. windows_shared_memory might be another issue.

No, no...the point of the ticket was that there already exists an unnecessary Windows vs POSIX distinction in the interface...adding more windows-centric features would only make the problem worse...and, as you say, ...it is about portability...

As explained in the boost.devel thread, this Win vs POSIX interface complication exists only because 'Boost.Interprocess' insists on the POSIX model that shared memory objects _must_ have kernel lifetime semantics on all platforms...If you give up on that premise, and provide two 'layers' or RAII wrappers for shared memory objects, one with scoped (or implicitly reference counted through the OS handle) life time semantics (like for normal C++ objects) and another with persistent/kernel life time this complication would (probably) vanish...Then you could implement the 'plain'/scoped shm on Windows using the native shared memory mechanisms (as there would no longer be any need for the file-system-persistency emulation) which would in turn remove the need for the separate windows_shared_memory...

Ion Gaztañaga — Sat, 02 Apr 2011 20:15:50 GMT

Interprocess insists on the POSIX model because it is the portable way that can be reliable and easily implemented in all OS and with all users. That's one of the main goals for all Boost libraries. In Interprocess review this was decided and it is not going to change. Windows reference-counted semantics are imposible to achieve reliably in POSIX (a process crash would leave the memory permanently) without kernel help and windows persistent shared memory needs special priviledges. So there is no discussion here.

windows_shared_memory offers referece-counted semantics because they are useful for Windows users. Using OBJ_PERMANENT might be useful to implement shared memory for users accepting priviledge limitations, but not as a general solution.

I'm sorry we don't share views on the library design but there is nothing we can do about it.

Domagoj Šarić — Mon, 04 Apr 2011 09:31:37 GMT

Replying to igaztanaga:

Interprocess insists on the POSIX model because it is the portable way that can be reliable and easily implemented in all OS and with all users. That's one of the main goals for all Boost libraries.

I wouldn't be so eager to call this 'easy' considering the complications in implementation (the registry-temp-path dance, remove_shared_memory_on_destroy helper...) or even quite so portable considering the 'platform specific' 'pollution' of the interface...

In Interprocess review this was decided and it is not going to change. Windows reference-counted semantics are imposible to achieve reliably in POSIX (a process crash would leave the memory permanently) without kernel help and windows persistent shared memory needs special priviledges. So there is no discussion here.

If "a possible process crash" was the only argument put forth in the review (I'm sorry I wasn't around then so I do not know) to justify this design then I beg to differ (on the "no discussion here"). AFAIK Boost is not designed around bugs, UB, abnormal termination or 'catastrophic failures' which, AFAICT, makes this argument bogus. Not to clean up memory mapped objects after a process is a POSIX design decision and Boost should not go out of its way to circumvent this first and foremost for the simple reason that it cannot. Even with the current Boost.Interprocess, if a POSIX-based process crashes (right) before the remove_shared_memory_on_destroy destructor is called the mapped object will also, AFAIK, 'leak'/'dangle'...Users using POSIX already are/have to be aware of this POSIX behaviour/semantics and be prepared to manually handle the cleanup (or decide not to)...

ps. Windows persistent shared memory requires special privileges _for creation only_...is/was there a use case where a limited-user Windows application needs to create persistent shared memory?

pps. no need to discuss reference counted semantics yet (I wasn't asking for those anyway), simple scoped mapped objects would be enough...

windows_shared_memory offers referece-counted semantics because they are useful for Windows users. Using OBJ_PERMANENT might be useful to implement shared memory for users accepting priviledge limitations, but not as a general solution.

Well, at least having the option (to skip the emulation on Windows/use an alternate native implementation) would certainly be welcomed but it would still do nothing about the interface 'platform specificness'...

status changed; resolution set

Ion Gaztañaga — Sat, 24 Dec 2011 18:30:59 GMT

status reopened → closed
resolution → wontfix