[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
precision of floor(a/b)
From: |
Tim Rueth |
Subject: |
precision of floor(a/b) |
Date: |
Mon, 5 Apr 2010 16:52:44 -0700 |
If I do floor(6/1),
I correctly get 6. If I do floor(0.6/0.1), I get 5.
If I do
floor((0.6/0.1 + 0.00000001)) I correctly get 6.
I assume this has to do with the precision of the
division? I read in the manual that Octave stores numbers as double
precision, so this surprises me. What's the best way to deal with this
imprecision so I can always get the correct answer, or is something else going
on here?
I'm on Windows
Vista, if that makes a difference.
Thanks,
--Tim
- precision of floor(a/b),
Tim Rueth <=
- Re: precision of floor(a/b), Ben Abbott, 2010/04/05
- RE: precision of floor(a/b), Tim Rueth, 2010/04/05
- RE: precision of floor(a/b), John W. Eaton, 2010/04/05
- Re: precision of floor(a/b), Scott Carter, 2010/04/05
- RE: precision of floor(a/b), Tim Rueth, 2010/04/06
- RE: precision of floor(a/b), CdeMills, 2010/04/06
- Re: precision of floor(a/b), Martin Helm, 2010/04/06
- Re: precision of floor(a/b), Dr.-Ing. Torsten Finke, 2010/04/06
- Re: precision of floor(a/b), Jaroslav Hajek, 2010/04/06
- RE: precision of floor(a/b), Tim Rueth, 2010/04/07