Fixed normalization problem with Boost.Rational's "assign"; add check for negative values too large for normalization (refs #9067)