Boost C++ Libraries: Ticket #5230: Interprocess shared memory between 32 and 64 bit processes

attachment set

tgermer@… — Fri, 25 Feb 2011 13:49:55 GMT

attachment → boost_interprocess_32_64bit_patch.zip

Ion Gaztañaga — Mon, 28 Feb 2011 10:14:33 GMT

Thanks, I've merged the patch with my latest local changes and I'm reviewing it. Regarding rbtree_best_fit's usage of offset_type, I think I will remove it. That might cause a binary incompatibility with earlier versions, but wrap-around logic is not needed to allocate memory correctly and increases bookeeping overhead for all allocations. I think I can easily remove this need in a day or two.

I hope I'll review the patch and push it into trunk this week. Best.

tgermer@… — Mon, 28 Feb 2011 17:36:43 GMT

I'm sorry, I found a bug in my patch at the selector that gets the offset_type for rbtree_best_fit. (I forgot a template parameter.) This should read:

   template<class S, class T>
   struct ptr_size_type_selector<boost::interprocess::offset_ptr<void, S, T> >
   {
	   typedef typename boost::interprocess::offset_ptr<void, S, T>::offset_type type;
   };

But this will get removed anyway, once we don't rely on wrap-around logic anymore...

Best, Tobias

Ion Gaztañaga — Mon, 21 Mar 2011 15:25:34 GMT

Please check latest SVN trunk code. Download both Move and Interprocess (new Intrusive is optional). There is one thing left, though: I can't make boost::aligned_storage work with offset_ptr and boost 1.47 in msvc (some namespace clashing when compiling). I hope you can find a workaround faster than me. Let me know if this version makes 32-64 bit process communication possible.

tgermer@… — Mon, 21 Mar 2011 16:20:16 GMT

Thank you for integrating the patch.

I also encountered this namespace conflict. It results from the ambiguity between "boost::detail" and "boost::interprocess::detail" when referring to "detail::" at the local namespace. I resolved this by declaring the absolute namespace path in aligned_storage.hpp. So every "detail::" becomes "::boost::detail::", which removes this ambiguity. This was also included in the patch. Is this change not feasible?

We need some time to integrate the new version into our framework and test it. I'll post if it worked.

Ion Gaztañaga — Mon, 21 Mar 2011 21:42:01 GMT

Modifying aligned_storage is not in my hand, I think you should post this patch in the mailing list. If no one replies or has any solid arguments against the patch I will apply it.

tgermer@… — Tue, 22 Mar 2011 10:46:31 GMT

OK, I just posted the patch to the mailing list (waiting for moderation).

I also updated our boost libraries and tested the new version in our 32-64 bit mixed environment, and it works like a charm :) Thanks again for your effort!

Wed, 23 Mar 2011 05:25:06 GMT

Replying to tgermer@…:

OK, I just posted the patch to the mailing list (waiting for moderation).

I also updated our boost libraries and tested the new version in our 32-64 bit mixed environment, and it works like a charm :) Thanks again for your effort!

hey,

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

Cheers

tgermer@… — Wed, 23 Mar 2011 08:08:35 GMT

Replying to xijing dai <dxj19831029@…>:

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

I'm also on 1.46.1. You just have to update boost/interprocess, boost/intrusive and boost/move (which is new) to the trunk version in order to use it. No need to update anything else yet. Currently there is one open issue to make it work: the namespace conflict. If you want to resolve it manually, you need to remove the comment from "union internal_type" in boost/interprocess/offset_ptr.hpp. Then you have to replace every "detail::" with "::boost::detail::" in aligned_storage.hpp (the patch that I suggested in comment 4 and also on the mailing list).

Wed, 23 Mar 2011 14:16:41 GMT

Replying to tgermer@…:

Replying to xijing dai <dxj19831029@…>:

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

I'm also on 1.46.1. You just have to update boost/interprocess, boost/intrusive and boost/move (which is new) to the trunk version in order to use it. No need to update anything else yet. Currently there is one open issue to make it work: the namespace conflict. If you want to resolve it manually, you need to remove the comment from "union internal_type" in boost/interprocess/offset_ptr.hpp. Then you have to replace every "detail::" with "::boost::detail::" in aligned_storage.hpp (the patch that I suggested in comment 4 and also on the mailing list).

First all, thanks to this patch, whoever did it, I really appreciate their work, it helps me a lot.

I tested in my application, it works great. testing includes:

windows_shared_memory
message queue
managed_windows_shared_memory

create named item raw allocate

By the way, I didn't change anything relating to the namespace issue, it passed compiling and worked great.

Mon, 28 Mar 2011 03:27:18 GMT

Replying to xijing dai <dxj19831029@…>:

Replying to tgermer@…:

Replying to xijing dai <dxj19831029@…>:

I am currently on version 1.46.1 I would like to get this patch and have a go, is that possible? where can i get the patch? do i need to upgrade all my boost libs to trunk head in order to use it?

I'm also on 1.46.1. You just have to update boost/interprocess, boost/intrusive and boost/move (which is new) to the trunk version in order to use it. No need to update anything else yet. Currently there is one open issue to make it work: the namespace conflict. If you want to resolve it manually, you need to remove the comment from "union internal_type" in boost/interprocess/offset_ptr.hpp. Then you have to replace every "detail::" with "::boost::detail::" in aligned_storage.hpp (the patch that I suggested in comment 4 and also on the mailing list).

First all, thanks to this patch, whoever did it, I really appreciate their work, it helps me a lot.

I tested in my application, it works great. testing includes:

windows_shared_memory
message queue
managed_windows_shared_memory
create named item raw allocate

By the way, I didn't change anything relating to the namespace issue, it passed compiling and worked great.

Interesting, I am not sure if I had wrong testing last week, or I did something wrong this week.

This week, the program doesn't work properly between 32/64 managed_windows_shared_memory

I do a little trace, when i allocate 1000000 bytes, it said free memory is: 999912, in 32bit. in 64 bit, it is: 999848.

For me, it looks like something wrong in: rbtree_best_fit, since when i trace from get_free_memory method, it gives different values, so I assume there is some setting wrong for that class.

by the way, I do uncomment union internal_type section, and I didn't change "detail::" stuff inside aligned_storage.hpp, is that a problem?

Do you have any idea what's going on here?

Cheers

tgermer@… — Mon, 28 Mar 2011 07:34:46 GMT

Replying to xijing dai <dxj19831029@…>:

by the way, I do uncomment union internal_type section, and I didn't change "detail::" stuff inside aligned_storage.hpp, is that a problem?

No, if your compiler doesn't produce an error, then this should be OK.

Do you have any idea what's going on here?

Not really. Certainly Ion knows the details better and can tell what's going on. You could also look into raw memory and watch for inconsistencies across 32/64-bit builds.

attachment set

tgermer@… — Mon, 28 Mar 2011 07:50:28 GMT

attachment → boost_r70306_aligned_storage.zip

aligned-storage-patch

tgermer@… — Mon, 28 Mar 2011 07:52:06 GMT

Replying to igaztanaga:

Modifying aligned_storage is not in my hand, I think you should post this patch in the mailing list. If no one replies or has any solid arguments against the patch I will apply it.

@Ion: The guys on the mailing list just found an error with a digraph that I mistankingly had in the patch. In the last couple of days, there were no further replies. I think you could apply the patch.

I added the patch as an attachment to this ticket. Interestingly, trac doesn't accept the patch as plain text and thinks it is spam :)

Mon, 28 Mar 2011 08:05:48 GMT

here is the code:

this->memorySegment = new boost::interprocess::managed_windows_shared_memory( boost::interprocess::open_or_create, name, size);
int sizeB = this->memorySegment->get_free_memory();

i allocate 1000000 bytes, it said free memory is: 999912, in 32bit. in 64 bit, it is: 999848.

Just wonder, do I need to use different namespace or class name in order for managed shared memory to be compatible for 32 and 64 system?

emm...by the way, how can i reach Ion?

Cheers

tgermer@… — Mon, 28 Mar 2011 08:11:07 GMT

Replying to xijing dai <dxj19831029@…>:

Just wonder, do I need to use different namespace or class name in order for managed shared memory to be compatible for 32 and 64 system?

No. In our case, the code for the 32 and 64 bit processes is actually identical.

emm...by the way, how can i reach Ion?

Well, I think he follows this conversation :)

Ion Gaztañaga — Mon, 28 Mar 2011 08:42:23 GMT

Replying to xijing dai <dxj19831029@…>:

here is the code:

The patch is not about making all interprocess classes workable between 32 and 64 bit applications. Is about customizing offset_ptr and propagate its attributes (size_type, difference_type, etc..) so that that this *could* be possible, of course depending on the compiler ABI of 32/64 bit applications. The patch is experimental, so it won't be documented and made official until it gets some real-application experience.

Regarding message_queue, it is a bug that it does not work between 32 and 64 bit applications. I should change queue internals to make internal types fixed length. It's in my to-do list but i haven't found time to implement it.

Mon, 28 Mar 2011 10:14:15 GMT

Replying to igaztanaga:

Replying to xijing dai <dxj19831029@…>:

here is the code:

The patch is not about making all interprocess classes workable between 32 and 64 bit applications. Is about customizing offset_ptr and propagate its attributes (size_type, difference_type, etc..) so that that this *could* be possible, of course depending on the compiler ABI of 32/64 bit applications. The patch is experimental, so it won't be documented and made official until it gets some real-application experience.

Regarding message_queue, it is a bug that it does not work between 32 and 64 bit applications. I should change queue internals to make internal types fixed length. It's in my to-do list but i haven't found time to implement it.

got u!

which class have u tested? let me check if I can replace with them.

otherwise, I am not sure if we could wait for next boost release, I need to think some backup plans. :)

Mon, 28 Mar 2011 13:02:17 GMT

Replying to igaztanaga:

Replying to xijing dai <dxj19831029@…>:

here is the code:

The patch is not about making all interprocess classes workable between 32 and 64 bit applications. Is about customizing offset_ptr and propagate its attributes (size_type, difference_type, etc..) so that that this *could* be possible, of course depending on the compiler ABI of 32/64 bit applications. The patch is experimental, so it won't be documented and made official until it gets some real-application experience.

Regarding message_queue, it is a bug that it does not work between 32 and 64 bit applications. I should change queue internals to make internal types fixed length. It's in my to-do list but i haven't found time to implement it.

boost::interprocess::offset_ptr<void> is 4 bytes long, is that normal for 32 program? should it be 8 bytes?

anonymous — Mon, 28 Mar 2011 13:11:46 GMT

offset_ptr uses its template parameter OffsetType for its internal representation. That means it depends on you how large it is. Per default, OffsetType=size_t, which is 4 bytes for 32 bit programs. In 32/64-bit mixed applications, you should indeed take an 8-byte offset_ptr, for example: offset_ptr<void, int, unsigned long long>

status changed; resolution set

Ion Gaztañaga — Fri, 01 Apr 2011 21:14:05 GMT

status new → closed
resolution → fixed

Fixed for Boost 1.47

anonymous — Tue, 07 Feb 2012 01:30:12 GMT

Replying to igaztanaga:

Fixed for Boost 1.47

According to the 1.48.0 release notes, this bug fix went into 1.48.0, not 1.47. (See http://www.boost.org/users/history/version_1_48_0.html )

status changed; resolution deleted

anonymous — Wed, 30 Aug 2017 12:25:51 GMT

status closed → reopened
resolution fixed