Re: trying to understand why I get 2.2204e-16 rather than 0

[James Sherman Jr.]

On Wed, Aug 3, 2016 at 8:41 PM, naryana.shankara
<address@hidden> wrote:
> If I enter cot (pi/4)-1 I get 2.2204e-16
> From my trig days I remember cot (pi/4) is 1 and 1-1 is 0
> Octave also confirms cot (pi/4) is 1 and 1-1 is 0
>
> so why does Octave spit out 2.2204e-16 when cot(pi/4)-1 is entered?
>
>
>
> --
> View this message in context: 
> http://octave.1599824.n4.nabble.com/trying-to-understand-why-I-get-2-2204e-16-rather-than-0-tp4678993.html
> Sent from the Octave - General mailing list archive at Nabble.com.
>
> _______________________________________________
> Help-octave mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/help-octave

Some background reading is probably the best:
https://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding
Note the reason you're getting isn't Octave so much as a property of
floating point numbers.

Really the whole article is pretty good, but it boils down to
essentially that representing most numbers is an approximation because
of finite memory with decimal representation, let alone irrational
ones like pi.  This isn't even getting into things like how trig
functions are calculated.  To show that it is even more fundamental
than that, try

>> (1-1/3)-2/3

and see the result you get.

James Sherman