groff
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Groff] About numerical precision


From: Ralph Corderoy
Subject: Re: [Groff] About numerical precision
Date: Wed, 02 Apr 2003 11:50:15 +0100

Hi Thomas,

> But, I can't manage to get \H'36432*72/46800' without any error:
> \H'\\nY*\\n(.s/\\nX' produces a multiplication overflow
> 
> and \H'36432/46800*72' seems to produce some \H'0' for obvious
> precision reasons.
> 
> Any idea ?

Since you know the numbers are up in the thousands, how about avoiding
the overflow by scaling them down first.

    $ bc
    36432*72/46800
    56
    36432/46800*72
    0
    (36432/100*72)/(46800/100)
    56

Cheers,


Ralph.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]