Skip to content

Fix float precision#470

Closed
rwols wants to merge 2 commits intojbeder:masterfrom
rwols:fix-float-precision
Closed

Fix float precision#470
rwols wants to merge 2 commits intojbeder:masterfrom
rwols:fix-float-precision

Conversation

@rwols
Copy link
Copy Markdown

@rwols rwols commented Feb 22, 2017

Replaced

std::numeric_limits<T>::digits10 + 1

with

std::numeric_limits<T>::max_digits10

everywhere.

This fixes #469

Understandably some tests will start failing now

Replaced

    std::numeric_limits<T>::digits10 + 1
with

    std::numeric_limits<T>::max_digits10

everywhere.
@rwols
Copy link
Copy Markdown
Author

rwols commented Feb 22, 2017

6ebfa37 should fix the failing unit tests. I tried using

Emitter emitter;
emitter.SetFloatPrecision(3);
emitter.SetDoublePrecision(3);

but this only works for the very first float (or double) that gets emitted -- the floats that follow use the default precision. So I had to replace things like

1.01 --> 1.009999999998

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

float is precision is not precise enough

1 participant