boost::posix_time::operator<< results in heap corruption

[nmmreddy11@…]

We are using boost 1.59.0 compiled with Visual Studio 2015. Our application is multi-threaded and is running on a multi-processor server. I am troubleshooting a crash issue related to memory corruption for a while now. Finally able to run a debug build under the load and am getting heap corruption errors from boost::posix_time::operator<<.

HEAP[IPCaptureD.exe]: HEAP: Free Heap block 3A821208 modified at 3A821910 after it was freed (3290.33c0): Break instruction exception - code 80000003 (first chance) eax=fffdd000 ebx=3a821208 ecx=3a821208 edx=00000047 esi=3a82120a edi=00d20000 eip=77df7aa6 esp=1e80ebdc ebp=1e80ed30 iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 ntdllRtlpBreakPointHeap+0x19: 77df7aa6 cc int 3

STACK_TEXT: 1e80ebd8 77d97167 a2f3581f 00000230 00000224 ntdllRtlpBreakPointHeap+0x19 1e80ed30 77d50ef4 00000224 00000240 00000000 ntdllRtlpAllocateHeap+0x46233 1e80edc4 77df628a 00d20000 50000163 00000224 ntdllRtlAllocateHeap+0x14d 1e80ee24 77d96efb 00000224 a2f35aaf 00000230 ntdllRtlDebugAllocateHeap+0xdf 1e80ef80 77d50ef4 00000224 00000230 00000000 ntdllRtlpAllocateHeap+0x45fc7 1e80f014 6be24158 00d20000 40000060 00000224 ntdllRtlAllocateHeap+0x14d 1e80f074 6be23f66 00000200 00000002 694a2dd0 ucrtbased!_toupper+0x248 1e80f098 6be264fb 00000200 00000002 694a2dd0 ucrtbased!_toupper+0x56 1e80f0b8 694ad658 00000100 00000002 00000002 ucrtbased!_calloc_dbg+0x4b 1e80f0e4 694c0622 1e80f0f0 3469be60 00000005 MSVCP140D!_Getctype+0x28 [f:\dd\vctools\crt\crtw32\stdcpp\_tolower.c @ 140] 1e80f114 694c0d41 1e80f128 1e80f434 1e80f374 MSVCP140D!std::_Locinfo::_Getctype+0x12 [f:\dd\vctools\crt\crtw32\stdhpp\xlocinfo @ 117] 1e80f16c 694bc31e 1e80f2c8 bceb0904 3469be60 MSVCP140D!std::ctype<unsigned short>::_Init+0x21 [f:\dd\vctools\crt\crtw32\stdhpp\xlocale @ 2904] 1e80f18c 694e9ac7 1e80f2c8 00000000 bceb0ac0 MSVCP140D!std::ctype<unsigned short>::ctype<unsigned short>+0x4e [f:\dd\vctools\crt\crtw32\stdhpp\xlocale @ 2882] 1e80f248 694cebf1 1e80f2c8 0000003f 3ac32248 MSVCP140D!std::locale::_Locimp::_Makeushloc+0x77 [f:\dd\vctools\crt\crtw32\stdcpp\wlocale.cpp @ 85] 1e80f2ac 694ce5f5 1e80f2c8 0000003f 3ac32248 MSVCP140D!std::locale::_Locimp::_Makeloc+0x391 [f:\dd\vctools\crt\crtw32\stdcpp\locale.cpp @ 80] 1e80f31c 694bc7f6 3ac32248 00d2eb48 bceb0bc8 MSVCP140D!std::locale::_Locimp::_Locimp_ctor+0x55 [f:\dd\vctools\crt\crtw32\stdcpp\locale.cpp @ 92] 1e80f340 694d166b 00d2eb48 bceb0be0 1e80f388 MSVCP140D!std::locale::_Locimp::_Locimp+0x96 [f:\dd\vctools\crt\crtw32\stdhpp\xlocale @ 219] 1e80f368 0074e38e 00d2eb48 1e80f388 1e80f404 MSVCP140D!std::locale::_Locimp::_New_Locimp+0x4b [f:\dd\vctools\crt\crtw32\stdcpp\locale0.cpp @ 195] 1e80f37c 0074e902 1e80f3d4 3ad93a38 d5bbdec2 IPCaptureD!std::locale::locale<boost::date_time::time_facet<boost::posix_time::ptime,char,std::ostreambuf_iterator<char,std::char_traits<char> > > >+0x1e [c:\program files (x86)\microsoft visual studio 14.0\vc\include\xlocale @ 288] 1e80f440 007633ce 1e80f554 1e80f5f4 d5bbdd42 IPCaptureD!boost::posix_time::operator<<<char,std::char_traits<char> >+0x232 [c:\views\develop\recorder_alt\recorder_common\media_common\common\oem\boost\boost\date_time\posix_time\posix_time_io.hpp @ 191] 1e80f7c0 0074c521 1e80fa44 1c9f6ba0 3b3c5e78 IPCaptureD!`anonymous namespace'::TimeZoneDatabase::ConvertUTCToTimeZone+0x3ee [c:\views\develop\recorder_alt\recorder_common\src\shared\timestamp.cpp @ 121]

SYMBOL_NAME: heap_corruption!heap_corruption

[nmmreddy11@…]

I see a similar incident reported 8 years ago. ​https://svn.boost.org/trac/boost/ticket/3369

Issue:

posix_time::operator<< crashes with double free under possible race conditions #13 0x08070aca in boost::posix_time::operator<< <char, std::char_traits<char> > (os=@0x80def94,p=@0x40bb32c8) at posix_time_io.hpp:53

Reply:

Is the streaming expression protected with a mutex lock or a similar synchronization mechanism? If not, that is the most probable source of the problem.

Closure comments:

We haven't seen this issue reappear, so perhaps it was a temporary lapse in synchronization after all. Can be closed.

What should I do?

Am I supposed to synchronize function calls to data time library? posix_time is not thread safe?

[nmmreddy11@…]

I created this as a show stopper as the crash is currently blocking a release.

When the test is run with release build, application crashes with different symptoms as the memory is corrupted by then and different threads get affected on different runs.

When the test is run with debug build, application has seen break exception on heap corruption reported in this ticket.

[nmmreddy11@…]

Running the test on Windows Server 2012 R2 standard

[nmmreddy11@…]

We did find a synchronization issue in our code; using an unprotected global object to support time zone DB, which internally uses boost date time library. This ticket can be closed.