RE: precision of floor(a/b)

[Tim Rueth]

Yes, that fixed it, thanks!  Learning something new every day.

--Tim


> -----Original Message-----
> From: Ben Abbott [mailto:address@hidden 
> Sent: Monday, April 05, 2010 5:01 PM
> To: address@hidden
> Cc: address@hidden
> Subject: Re: precision of floor(a/b)
> 
> 
> On Apr 5, 2010, at 7:52 PM, Tim Rueth wrote:
> 
> > If I do floor(6/1), I correctly get 6.  If I do 
> floor(0.6/0.1), I get 5.  If I do floor((0.6/0.1 + 
> 0.00000001)) I correctly get 6.
> >  
> > I assume this has to do with the precision of the division? 
>  I read in the manual that Octave stores numbers as double 
> precision, so this surprises me.  What's the best way to deal 
> with this imprecision so I can always get the correct answer, 
> or is something else going on here?
> 
> Will floor(0.6/0.1*(1+eps)) work for you?
> 
> Ben
> 
>