Re: precision of floor(a/b)

[Scott Carter]

Yep, that's why I said that if the OP wants exact decimal arithmetic the 
easiest (IMHO) approach is to convert everything to integer and input 
and separately keep track of where the radix point is.

But I still don't know what the original problem is/was:

if 0.6 and 0.1 are meant to be exact decimals, then the convert to 
integer approach works.

if they are really floating point approximations, then the use of the 
floor function is basically ill-defined

since floor((0.6+eps)/(0.1-eps)) != floor ((0.6-eps)/(0.1+eps))



John W. Eaton wrote:
> | Yes, that fixed it, thanks!  Learning something new every day.
> | 
> | --Tim
> | 
> | 
> | > -----Original Message-----
> | > From: Ben Abbott [mailto:address@hidden 
> | > Sent: Monday, April 05, 2010 5:01 PM
> | > To: address@hidden
> | > Cc: address@hidden
> | > Subject: Re: precision of floor(a/b)
> | > 
> | > 
> | > On Apr 5, 2010, at 7:52 PM, Tim Rueth wrote:
> | > 
> | > > If I do floor(6/1), I correctly get 6.  If I do 
> | > floor(0.6/0.1), I get 5.  If I do floor((0.6/0.1 + 
> | > 0.00000001)) I correctly get 6.
> | > >  
> | > > I assume this has to do with the precision of the division? 
> | >  I read in the manual that Octave stores numbers as double 
> | > precision, so this surprises me.  What's the best way to deal 
> | > with this imprecision so I can always get the correct answer, 
> | > or is something else going on here?
> | > 
> | > Will floor(0.6/0.1*(1+eps)) work for you?
>
> I don't think this is a general solution to the problem.  It will
> likely do the "right thing" (i.e., what you expect for decimal
> arithmetic) in some cases but not others.  For example, what about
>
>   floor (0.7999999999999999/0.1*(1+eps))
>
> which on my system returns 8.  Note that in this case,
> 0.7999999999999999 < 0.8 does evaluate to true, so I guess you would
> want the result of the division and floor operation to be 7, not 8 if
> you were doing decimal arithmetic.
>
> The thing is, you aren't doing decimal arithmetic.  So throwing in
> factors of eps to get solutions that look right for decimal arithmetic
> in certain cases is probably not what you want to do.
>
> jwe
> _______________________________________________
> Help-octave mailing list
> address@hidden
> https://www-old.cae.wisc.edu/mailman/listinfo/help-octave
>
>   

--
Scott Carter
Sr. Manager Product R&D
NKUS Lab
Nihon Kohden Corp.
Tel: 949-474-9207, X. 103
FAX: 949-474-8267
address@hidden