Calculation overflow for the 4th moment

[m.de.wever@…]

While testing the accumulator library the calculation of the kurtosis goes wrong with a larger dataset. I traced to problem down to the calculation of the 4th moment which seems to overflow. The attached program produces the following output on a 32 bit machine.

2.44141e+08
2.48094e+08
2.52111e+08
2.56192e+08
2.60338e+08
2.6455e+08
2.68829e+08
2.73175e+08
2.77589e+08
2.82072e+08
2.86624e+08
2.91247e+08
2.95942e+08
3.00708e+08
1.92167e+07
4.20257e+07
6.28038e+07
8.19029e+07
9.96007e+07
1.1612e+08
1.3164e+08
1.4631e+08
1.60251e+08
1.73565e+08
1.45388e+07

PS I didn't test whether the 3th moment also can overflow, but I had no problem with the calculation of the skewness.

[m.de.wever@…]

test program to reproduce the problem

[Steven Watanabe]

Of course it overflows if you use unsigned int. Use double instead.

[m.de.wever@…]

I did some more tests and the 4th moment returns a double when I use unsigned as template type. So I still consider it a bug.

This example, from the library documentation, also leads me to believe the result type is a floating point type and it's save to use integer types.

accumulator_set<int, stats<tag::moment<2> > > acc1;

acc1(2); //    4
acc1(4); //   16
acc1(5); // + 25
         // = 45 / 3 = 15

BOOST_CHECK_CLOSE(15., moment<2>(acc1), 1e-5);

[Eric Niebler]

The moment returns a double even for integral sample types because of the division at the end. The additions during the accumulation phase are integral, and so they overflow. Adding overflow checks would slow down the common case, so I'm disinclined to add it unconditionally. Steven is right, the easiest solution is for you to use doubles.

The larger issue is how to handle things like overflow, underflow, divide-by-zero, NaN propagation, etc., throughout the Accumulators library (and beyond to all the math-related libraries in Boost (and beyond to the standard library)). One possibility would be to add checked arithmetic as a policy. Another possibility is to specify the sample type as checked<int> or something. It's probably worth starting such a discussion on the Boost list. I'm sure John Maddock would have some useful insights.

I'm resolving this bug as "wontfix" and will open a separate feature request ticket for better control over error-handling in Accumulators.

[m.de.wever@…]

I tested with the double and it works properly now.

Thanks for opening a feature request. I think it would also be nice to add a warning about the overflow problem in the documentation until that feature has been implemented.