RE: Problem with the floor function

[Ted Harding]

On 19-Jan-99 Daniel Tourde wrote:
> Hello,
> 
> I have discovered something strange with the floor function. I suspect
> a bug but where ? John W. Eaton would bet that floor is correct, but
> that the display routines in the C/C++ I/O library are doing
> something slightly different when they display values that are very
> near but just less than a representable integer.
> 
> octave:1> floor(46.000)
> ans = 46
> octave:2> k = 0.03
> k = 0.030000
> octave:3> kmin=0
> kmin = 0
> octave:4> kstep = 0.01
> kstep = 0.010000
> octave:5> modes=15
> modes = 15
> octave:6> (1+(k-kmin)*modes/kstep)
> ans = 46.000
> octave:7> floor(1+(k-kmin)*modes/kstep)
> ans = 45                                     <-  It should be 46, no ?
> octave:8> (k-kmin)*modes
> ans = 0.45000
> octave:9> (k-kmin)*modes/kstep
> ans = 45.000
> octave:10> floor((k-kmin)*modes/kstep)
> ans = 44                                     <-  It should be 45, no ?
> 
> Strange isn't it ?
> Does anyone has an idea of where it comes from and how it could be
> solved ?

The reason is that

  octave:10> 46 - (1+(k-kmin)*modes/kstep)
  ans =  7.10542735760100e-15

i.e. as a consequence of truncation error in the binary representation of
floating-point numbers, the result of the computation [1] of
(1+(k-kmin)*modes/kstep) is slightly less than 46 in the machine
representation. Therefore "floor" is behaving as it should.

The only way to get round this is to add a small positive number to the
result of such a computation, and hope that this doesn't in turn generate
an error in the opposite direction. How small the number should be can in
general only be determined by careful error analysis of the computation.

[1] Note the wording. It is not that a number which is an exact integer
46 (as implied by the mathematical expression) is inaccurately
represented in the machine. In reality, the numbers k=0.03 and kstep=0.01
are both inaccurately represented in the first place (unavoidable), and
-- even assuming that their internal representations were their exact
values -- further inaccuracies are introduced when the machine
computations evoked by "+", "*" and "/" are carried out.

Hope this helps,
Ted.

--------------------------------------------------------------------
E-Mail: (Ted Harding) <address@hidden>
Date: 19-Jan-99                                       Time: 11:31:15
------------------------------ XFMail ------------------------------