help-octave
[Top][All Lists]

## Re: trying to understand why I get 2.2204e-16 rather than 0

 From: Maynard Wright Subject: Re: trying to understand why I get 2.2204e-16 rather than 0 Date: Fri, 05 Aug 2016 22:16:43 -0700 User-agent: KMail/4.13.3 (Linux/3.16.0-30-generic; KDE/4.13.3; i686; ; )

```On Wednesday, August 03, 2016 08:59:15 PM Dmitri A. Sergatskov wrote:
> On Wed, Aug 3, 2016 at 7:41 PM, naryana.shankara <address@hidden
> > wrote:
> >
> > If I enter cot (pi/4)-1 I get 2.2204e-16
> > From my trig days I remember cot (pi/4) is 1 and 1-1 is 0
> > Octave also confirms cot (pi/4) is 1 and 1-1 is 0
>
> octave:6> format hex
> octave:7> 1
> ans = 3ff0000000000000
> octave:8> cot(pi/4)
> ​​
> ans = 3ff0000000000001
>
> > so why does Octave spit out 2.2204e-16 when cot(pi/4)-1 is entered?
>
> https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
>
>
> ​Dmitri.

Although cot(pi/4)-1 returns 0, tan(pi/4)-1 returns -1.11022e-16.   One might
therefore suspect that cot() is not obtained by taking the inverse of tan(),
but 1 / (tan(pi/4))-1 returns 0, so the macro cot(x) = 1 / tan(x) may be used,
although I haven't checked.

To see exactly why these anomalies occur, one would have to look into the
floating point representations of the numbers involved and the procedure used
to compute tan().   The references given by others in this thread should be
helpful in pointing out the limitations of representing numbers in floating
point format.

Maynard Wright

```