Construction from binary data

[Domagoj Šarić]

I didn't find a way to directly construct fixed sized cpp_int from existing binary data (a static array, for example an RSA key hardcoded into a .cpp after being generated with OpenSSL)... I had to resort to either using the from-string constructor or inserting the data byte by byte with 8 bit shifts and or-ing in a loop both of which are ugly and inefficient: ​https://svn.boost.org/trac/boost/ticket/9233 ​https://svn.boost.org/trac/boost/ticket/9234 ​https://svn.boost.org/trac/boost/ticket/9243 ...

Ideally it would be possible to create static fixed size numbers as one creates PODs, without a dynamic initializer being created by the compiler..

[John Maddock]

This is supported.... but only in C++11 compliant compilers (ie not VC++ I'm afraid at present). See: http://www.boost.org/doc/libs/1_54_0/libs/multiprecision/doc/html/boost_multiprecision/tut/lits.html

When supported though, the integer is compile-time constructed from the data with no runtime overhead at all.

Otherwise I'd strongly recommend you use the "construct from hexadecimal string" method, as that runtime code is well optimized (I hope!).

[Domagoj Šarić]

1) you somewhat contradict yourself here WRT to ​https://svn.boost.org/trac/boost/ticket/9234#comment:1 where you refer to contruction from a string as something non trivial

2) regardless, closing this as wontfix (instead of at least postponing until you or someone else finds the time) is lame IMO because this is certainly doable...'of the top of my head': you'd 'just' need to define an underlying POD which could be filled through a macro that'd:

• fill the raw array that represents the number
• set any aditional/auxiliary members you might have in number<>
• define a fully typeed number<> & bound to the helper underlying POD instance...
• "I know" it is doable because in my code (which I don't have with me now as I'm not at work) I had to do the opposite: retrieve the raw binary data from a cpp_int in order to compare it to an expected signature which I did by reinterpret_casting your number<> to a boost::array<> and calling std::reverse() on it...so this is one more function missing from the interface (retreiving a numbers raw byte representation as opposed to a slow EH littered dynamically allocated std::string conversion)

3) the ticket isn't (only) about static strings but creation from binary data in general (which one may receive or generate at runtime, for example from a different big-int library) so a function that does this (what I currently do with the mentioned loop in the other ticket) is certainly needed

4 when one uses your lib for cryptography or hash/licence verification storing (private) keys as plain text hex strings is not an option

[John Maddock]

It's not quite as trivial as you think: the internal data layout is highly platform/compiler dependent. For example a 128-bit integer is represented as 4 32 bit words on MSVC, but as a single 128-bit value on GCC-x64 (which has __int128).

Re (1), conversion from a hex string is fast, but decimal string conversion is heavyweight: of course they use the same routine though. Your example(4) is solved by user-defined literals I believe - it's the optimally efficient solution as it's all done at compile-time. For (3) would get_byte/set_byte non-member functions (similar to the existing bit twiddling functions) do the job? Note that these would only be efficient for unsigned types: signed types would require a check for negative values and the associated error handling - and yes that means exceptions.

[Domagoj Šarić]

Replying to johnmaddock:

I've just noticed you've added import/export_bits() :) ...interface-wise this would be enough to close this ticket however I'm not fully happy with the implementation so let's not yet ;-D

It's not quite as trivial as you think: the internal data layout is highly platform/compiler dependent. For example a 128-bit integer is represented as 4 32 bit words on MSVC, but as a single 128-bit value on GCC-x64 (which has __int128).

This does/should not matter if the endianess of the machine and your internal implementation match, the byte layout will be the same. Except if the 'raw'/binary data is in a different endianess than your internal representation (modeled by the msv_first import/export_bits() parameter) but even then it is a matter of a simple byte reversal. Tried it with both importing and exporting with Boost 1.60 and 'it just works':

    template <class Key>
    void LE_NOTHROW load( Key & key, std::array<std::uint8_t, Constants::rsaKeySizeInBytes> const & keyBytes )
    {
    #if 0 
        BOOST_VERIFY( &import_bits( key, keyBytes.begin(), keyBytes.end() ) == &key );
    #else
        key.backend().resize( key.backend().internal_limb_count, 0 );
        std::reverse_copy( keyBytes.begin(), keyBytes.end(), reinterpret_cast<std::uint8_t *>( key.backend().limbs() ) );
    #endif
    }

and looks trivial in the disassembly window ;)

ps. first I tried to force key (a fixed unchecked uint) limb count to max by constructing it as Key key( -1 ) but that didn't quite give a picturesque codegen ;D

[John Maddock]

Please see ​https://github.com/boostorg/multiprecision/commit/53515e9a960e4e7bdfa8f2e6d05ccb174d66bf4d.

Note this is import only, which I think fixes your use case. Export via memcpy to an output iterator is unsafe and not implemented. I could provide a separate interface for that, but I'd like to see a genuine need for high-speed-export first.

In any case, use of user-defined-literals (which is all constexpr) is the better solution for your use case, although I accept that's not supported on msvc yet :(