Boost C++ Libraries: Ticket #9235: Construction from binary data

John Maddock — Fri, 11 Oct 2013 16:16:06 GMT

This is supported.... but only in C++11 compliant compilers (ie not VC++ I'm afraid at present). See: http://www.boost.org/doc/libs/1_54_0/libs/multiprecision/doc/html/boost_multiprecision/tut/lits.html

When supported though, the integer is compile-time constructed from the data with no runtime overhead at all.

Otherwise I'd strongly recommend you use the "construct from hexadecimal string" method, as that runtime code is well optimized (I hope!).

status changed; resolution set

John Maddock — Fri, 11 Oct 2013 16:16:19 GMT

status new → closed
resolution → wontfix

description changed

Domagoj Šarić — Sat, 12 Oct 2013 10:50:01 GMT

description modified (diff)

status changed; resolution deleted

Domagoj Šarić — Sat, 12 Oct 2013 11:09:11 GMT

status closed → reopened
resolution wontfix

1) you somewhat contradict yourself here WRT to https://svn.boost.org/trac/boost/ticket/9234#comment:1 where you refer to contruction from a string as something non trivial

2) regardless, closing this as wontfix (instead of at least postponing until you or someone else finds the time) is lame IMO because this is certainly doable...'of the top of my head': you'd 'just' need to define an underlying POD which could be filled through a macro that'd:

fill the raw array that represents the number
set any aditional/auxiliary members you might have in number<>
define a fully typeed number<> & bound to the helper underlying POD instance...
"I know" it is doable because in my code (which I don't have with me now as I'm not at work) I had to do the opposite: retrieve the raw binary data from a cpp_int in order to compare it to an expected signature which I did by reinterpret_casting your number<> to a boost::array<> and calling std::reverse() on it...so this is one more function missing from the interface (retreiving a numbers raw byte representation as opposed to a slow EH littered dynamically allocated std::string conversion)

3) the ticket isn't (only) about static strings but creation from binary data in general (which one may receive or generate at runtime, for example from a different big-int library) so a function that does this (what I currently do with the mentioned loop in the other ticket) is certainly needed

4 when one uses your lib for cryptography or hash/licence verification storing (private) keys as plain text hex strings is not an option

type changed

John Maddock — Sat, 12 Oct 2013 16:27:01 GMT

type Bugs → Feature Requests

It's not quite as trivial as you think: the internal data layout is highly platform/compiler dependent. For example a 128-bit integer is represented as 4 32 bit words on MSVC, but as a single 128-bit value on GCC-x64 (which has __int128).

Re (1), conversion from a hex string is fast, but decimal string conversion is heavyweight: of course they use the same routine though. Your example(4) is solved by user-defined literals I believe - it's the optimally efficient solution as it's all done at compile-time. For (3) would get_byte/set_byte non-member functions (similar to the existing bit twiddling functions) do the job? Note that these would only be efficient for unsigned types: signed types would require a check for negative values and the associated error handling - and yes that means exceptions.

Domagoj Šarić — Thu, 18 Feb 2016 15:22:39 GMT

Replying to johnmaddock:

I've just noticed you've added import/export_bits() :) ...interface-wise this would be enough to close this ticket however I'm not fully happy with the implementation so let's not yet ;-D

It's not quite as trivial as you think: the internal data layout is highly platform/compiler dependent. For example a 128-bit integer is represented as 4 32 bit words on MSVC, but as a single 128-bit value on GCC-x64 (which has __int128).

This does/should not matter if the endianess of the machine and your internal implementation match, the byte layout will be the same. Except if the 'raw'/binary data is in a different endianess than your internal representation (modeled by the msv_first import/export_bits() parameter) but even then it is a matter of a simple byte reversal. Tried it with both importing and exporting with Boost 1.60 and 'it just works':

    template <class Key>
    void LE_NOTHROW load( Key & key, std::array<std::uint8_t, Constants::rsaKeySizeInBytes> const & keyBytes )
    {
    #if 0
        BOOST_VERIFY( &import_bits( key, keyBytes.begin(), keyBytes.end() ) == &key );
    #else
        key.backend().resize( key.backend().internal_limb_count, 0 );
        std::reverse_copy( keyBytes.begin(), keyBytes.end(), reinterpret_cast<std::uint8_t *>( key.backend().limbs() ) );
    #endif
    }

and looks trivial in the disassembly window ;)

ps. first I tried to force key (a fixed unchecked uint) limb count to max by constructing it as Key key( -1 ) but that didn't quite give a picturesque codegen ;D

status changed; resolution set

John Maddock — Fri, 26 Feb 2016 11:59:17 GMT

status reopened → closed
resolution → fixed

Please see https://github.com/boostorg/multiprecision/commit/53515e9a960e4e7bdfa8f2e6d05ccb174d66bf4d.

Note this is import only, which I think fixes your use case. Export via memcpy to an output iterator is unsafe and not implemented. I could provide a separate interface for that, but I'd like to see a genuine need for high-speed-export first.

In any case, use of user-defined-literals (which is all constexpr) is the better solution for your use case, although I accept that's not supported on msvc yet :(