[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: precision of calculations

From: A. Scottedward Hodel
Subject: Re: precision of calculations
Date: Fri, 23 Jul 1999 07:12:52 -0500

double precision arithmetic has about 16 significant digits
(depending on hardware).  Hence, if (A-B) should be zero, but
you get
    norm(A-B)/norm(A) approximately = 1e-16
you're o.k.  The relative error can creep up as problem dimension
and problem condition (how sensitive the results are to small
perturbations in data) increases.

For more information, consult Golub and Van Loan's book,
Matrix Computations, Johns Hopkins University Press, available
in paperback (for the frugal) or hardback (for those who use it 
a lot).

A S Hodel Assoc. Prof. Dept Elect Eng, Auburn Univ,AL  36849-5201
On leave at NASA Marshall Space Flight Center (256) 544-1426
Address until 15 Mar 2000:Mail Code TD-55, MSFC, Alabama, 35812

>From: Robert Butora <address@hidden>
>To: address@hidden
>Subject: precision of calculations
>Date: Fri, Jul 23, 1999, 7:27 AM

>Does anybody know how the precision of calculation 
>(not output_precision !) is controlled in Octave ?
>I've calculated some matrix where I expected results
>to be 1 or 0. Instead of 0 I've got values in order
>of 1e-16 ... 1e-18. 
>Should I chase an error in my calculation or does it
>have something to do with precision of calc. ?
>Could anybody give some general thoughts on precision in Octave ?
>Octave is freely available under the terms of the GNU GPL.  To ensure
>that development continues, see
>Instructions for unsubscribing:

Octave is freely available under the terms of the GNU GPL.  To ensure
that development continues, see
Instructions for unsubscribing:

reply via email to

[Prev in Thread] Current Thread [Next in Thread]