boost.asio: async_copy(in, out, ...) for POSIX sendfile() support

[trapni@…]

hey guys,

i wanted to use boost and boost.asio in order to implement a web service, however, when transmitting static files, i feel not welcome in just memcpy'ing the file into RAM and copying back into kernel space for transmission over TCP/IP. that's where POSIX sendfile() comes into play.

it takes an input file descriptor (local file e.g.), an output file descriptor (socket e.g.), offset and count, and transfers data directly from input to output *without* leaving kernel space, this is highly recommented e.g. when sending local files to sockets in HA software.

just like async_read/async_write, i imagine that it could be possible in providing an

return_type async_copy(input, output, count, finish_callback, ...);

would you mind?

[Christian Parpart <trapni@…>]

I quite don't think that this has anything to do with HA (::= High Availability) but much more with "High Performance" applications.

However, I'm really very interested in a native boost integration of this, too, but still, i think this can be quite challenging as the input and output descriptors need to match certain criteria on different UNIX-alike operating systems (btw: windows has something similar)

for Linux at least, the requirements are:

• input fd must be mmap()-able.
• output fd must be a socket.

Although, when it comes to implementing an HTTP server (in my case), you cannot just pass the filename as a string to async_copy(...) as there are moments where you want to send just parts of a file, or even further: multiple different ranges of a file with some memory "chunks" between.

Although, as in HTTP (and I assume this case for most of the file transfer related network protocols) before sending the plain file, you always send some kind of preamble/header, some memory chunk that introduces the range of bytes located in the file. Solaris helps out here with a special syscall, named sendfilev(), which is a combination of writev() and sendfile().

In the end, i think, that a simple "async_copy()" won't be able to cover all this, although, when just implementing a sendfile() wrapper, it won't be long to the next request of sendfilev().

To solve that, I'd like to propose a generalized idea of composed transmit buffers which is an ordered set of chunks of different types (e.g.: mem, file_descriptor, ...) which can be passed to some write(...) and async_write(...) Asio function.

This is basically what I have written for my http server case, i'd be happy to see something like this in Asio anytime soon :)

The following is a list of links where I implemented the composite transmit buffers with sendfile() support:

• ​http://code.ninchens.net/projects/x0/repository/entry/src/x0/composite_buffer.hpp
• ​http://code.ninchens.net/projects/x0/repository/entry/src/x0/composite_buffer_async_writer.hpp
• ​http://code.ninchens.net/projects/x0/repository/entry/src/x0/composite_buffer_writer.hpp

hope it'll inspire asio development process :)