Precision again

[Vic Norton]

Dmitri Sergatskov suggested that I post the following communication
to the Octave list. I had been a bit confused about the relationship
between Octave's eps and unit roundoff. Now I realize they are the
same.

Dmitri also pointed me to a couple of excellent sources of references.
   <http://docs.sun.com/source/806-3568/ncg_goldberg.html>
   <http://www.cs.berkeley.edu/~wkahan/>

### The original communication follows ###
Our precision discussion was very interesting to me. Golub and Van
Loan, _Matrix Computations_, define "unit roundoff" u as
   u = 0.5 * base^(1 - t)  for rounded arithmetic,
   u = base^(1-t) for chopped arithmetic.
The integer t is the "precision," the number of base-digits in a
floating point number.

IBM main frames used to use base = 16 and chopped arithmetic (in the
70s and 80s). I don't know how they work now. Apparently my Mac uses
base 2 and rounded arithmetic. If my Mac's precision is t = 52, as
suggested by Dmitri, with base 2 and rounded arithmetic, its unit
roundoff is u = eps= 2^(-52) = 2.2204e-16.

Looking back at my ACM Algorithm 631, Transactions on Mathematical
Software, Vol. 11, No. 2, June 1985, pp 120-134, I see that I
specified a parameter eps that "should be at least machine precision,
that is, 1 + eps > 1 in computer arithmetic." Octave's eps satisfies
this condition, but smaller numbers do also---as the following Octave
output shows.

(1 + eps * (1/2)) - 1 = 0.0000e+00
...
(1 + eps * (1/2 + 5/10000000000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/1000000000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/100000000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/10000000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/1000000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/100000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/10000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/1000)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/100)) - 1 = 2.2204e-16
(1 + eps * (1/2 + 5/10)) - 1 = 2.2204e-16
(1 + eps * (2/2)) - 1 = 2.2204e-16
...
(1 + eps * (1 + 5/10000000000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/1000000000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/100000000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/10000000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/1000000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/100000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/10000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/1000)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/100)) - 1 = 2.2204e-16
(1 + eps * (1 + 5/10)) - 1 = 4.4409e-16
(1 + eps * (3/2)) - 1 = 4.4409e-16