help-octave
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

precision of floor(a/b)


From: Tim Rueth
Subject: precision of floor(a/b)
Date: Mon, 5 Apr 2010 16:52:44 -0700

If I do floor(6/1), I correctly get 6.  If I do floor(0.6/0.1), I get 5.  If I do floor((0.6/0.1 + 0.00000001)) I correctly get 6.
 
I assume this has to do with the precision of the division?  I read in the manual that Octave stores numbers as double precision, so this surprises me.  What's the best way to deal with this imprecision so I can always get the correct answer, or is something else going on here?
 
I'm on Windows Vista, if that makes a difference.
 
Thanks,
 
--Tim

reply via email to

[Prev in Thread] Current Thread [Next in Thread]