Re: Rounding error in gawk 3.1.1 & 3.1.4

[Bob Proulx]

Ratcliffe, Jeffrey (Peters) wrote:
> With:
> 
> gawk '
> BEGIN {
> print "int(log(1000)/log(10))", int(log(1000)/log(10))
> print "log(1000)/log(10)", log(1000)/log(10)
> }
> '

Floating point arithmetic is only an approximation.  Also CPU math
processors have a lot of variance in how they implement their
computations.  So in general you cannot count on an exact result.  You
can only count on getting an approximately close result.

> Both results should, of course, be 3.

No.  This is a common mistake.  In computer science a whole discipline
around numerical methods has been developed to minimize errors made in
computer calculation.  Note I said minimize, not eliminate.  It is
still only an approximation.  If you search the web for numerical
methods you will find a lot of reference material available on the
subject.

> Is there a workaround?

By using int() you are truncating.  This is causing a big jump in
values from 3 to 2 in your example.  But you are probably wanting
rounding instead.  Add 0.5 first and round.

gawk '
BEGIN {
print "int(log(1000)/log(10)+0.5)", int(log(1000)/log(10)+0.5)
print "log(1000)/log(10)", log(1000)/log(10)
}
'

Bob