Re: Is this a numerical problem?

[Jaroslav Hajek]

On Thu, Aug 19, 2010 at 12:52 PM, Lukas Reichlin
<address@hidden> wrote:
> Hi Jaroslav
>
> I realized that (in most cases, see below) floats are rounded, not truncated 
> such that -4.00004 and -3.99995 are both displayed as -4.0000. Therefore you 
> can't tell whether -4.0000 is less or greater than -4.
>
> octave:15> 3.9999
> ans =  3.9999
> octave:16> 3.99995
> ans =  4.0000
> octave:17> 0.99995
> ans =  0.99995
> octave:18> 0.999995
> ans =  0.99999  # I'm expecting 1.00000 here
> octave:19> 0.9999995
> ans =  1.00000
> octave:20> 0.999996
> ans =  1.00000  # compare this to command 18
>
> Regards,
> Lukas
>

Exactly. Which brings us back to your observation:

octave:1> format long
octave:2> a = (10 * rem (-12/5, 1))
a = -4.00000000000000
octave:3> a - -4.00000000000000
ans =  8.88178419700125e-16
octave:4> -3.99999999999999 - a
ans =  9.32587340685131e-15

which is an explanation.

-- 
RNDr. Jaroslav Hajek, PhD
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz