Re: Rounding error in gawk 3.1.1 & 3.1.4

[Andrew J. Schorr]

On Thu, Oct 06, 2005 at 02:08:35PM +0200, Ratcliffe, Jeffrey (Peters) wrote:
> With:
> 
> gawk '
> BEGIN {
> print "int(log(1000)/log(10))", int(log(1000)/log(10))
> print "log(1000)/log(10)", log(1000)/log(10)
> }
> '
> 
> I get:
> 
> int(log(1000)/log(10)) 2
> log(1000)/log(10) 3
> 
> under both Redhat Linux (2.4.21-27.0.2.ELsmp) and HP-UX (B.11.11)
> 
> Both results should, of course, be 3.

Not really. :-)

This is because of the way floating-point arithmetic works.
Try running the following script:

gawk '
BEGIN {
print "int(log(1000)/log(10)) =", int(log(1000)/log(10))
print "3-log(1000)/log(10) =", 3-log(1000)/log(10)
}
'

and you'll get something like:

   int(log(1000)/log(10)) = 2
   3-log(1000)/log(10) = 4.44089e-16

> Is there a workaround?

If you really want integer truncation, you can add a fuzz factor before calling
int() to take account of the slight imprecision in the floating-point
arithmetic.  So instead of calling int(x), try int(x+epsilon), where epsilon
should be some small fraction of x that may depend on your hardware.
Typically, double precision arithmetic is good to approx. 15 or 16 significant
digits.

Or if you actually want to round to the nearest integer, you can do
something like

   function round_nearest(x) {
     return int((x+0 >= 0) ? (x+0.5) : (x-0.5))
   }

Regards,
Andy