Boost log writes much less records in remote disk than local disk

[xinyan.zhang@…]

Using Boost.log, my application tries to write a large number of records (600 MB) into the file in either a local disk or a remote/network disk. I tested twice, but the remote disk got much less records.

I use text_file_backend as the backend. Both test finished in 8 minutes. Then the throughput is 600MB/8/60sec = 1.25M/sec. The file in local disk has almost 600 MB. But the file in remote disk is only 200 MB. My application is on Windows, so the file path on remote disk is like "
MyFileServer\log\20160817-120329.log". The remote disk is shared by many users, so it may be slow. After the test, I still saw new records written to the remote file. Seems the network is not blocked.

That is strange. Could boost log handle remote disk? How to find out the root cause? Install exception handler? Thank you.

[Andrey Semashev]

Boost.Log has no knowledge about the nature of the filesystem, it works the same regardless. If your sinks are asynchronous then make sure log records are not piling up in the queue or being dropped due to the queue capacity limit. Try switching to synchronous logging or a non-dropping queueing policy and make sure you flush the sink before the application exit. Also check that you're using the same filter in both cases.

[xinyan.zhang@…]

My code uses asynchronous sink without setting any queuing policy. Does that mean "unbounded_fifo_queue" is used by default? And the queue has unlimited depth? If so, it could grow to whatever within the memory size. In my machine the maximum memory usage is less than 70%. The application itself uses about 800 MB memory. I did see any memory over flow.

[xinyan.zhang@…]

Sorry, the last sentence in previous reply should be " I did NOT see any memory over flow. "

Is there any way to check if messages are dropped or not? My way is: I added a counter, every time it increased by +1 when writing a log record. Then after the test, I waited for 20 minutes to ensure log records were written into the file. And then shut down the application. I found last counter is 10,000 larger than the total line number. In other words, almost 10,000 lines of record are missing in the log file.

[Andrey Semashev]

Is there any way to check if messages are dropped or not?

There is not. Unless there is a bug or memory overflow, unbounded_fifo_queue should not drop records. You'll have to debug the problem yourself. I'd start by counting the number of log records that passed the filter (asynchronous_sink::consume) and also the number of records that reached the file stream (text_file_backend::consume). Also try switching to synchronous_sink.

[xinyan.zhang@…]

I found the reason why less log records are in file. It is not related to the boost library, but our test tool which has problems. So, less log records are generated. Thank you very much.