help-octave
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Is this a numerical problem?


From: Sergei Steshenko
Subject: Re: Is this a numerical problem?
Date: Thu, 19 Aug 2010 04:16:09 -0700 (PDT)


--- On Thu, 8/19/10, Lukas Reichlin <address@hidden> wrote:

> From: Lukas Reichlin <address@hidden>
> Subject: Re: Is this a numerical problem?
> To: "Jaroslav Hajek" <address@hidden>, address@hidden
> Date: Thursday, August 19, 2010, 3:52 AM
> Hi Jaroslav
> 
> I realized that (in most cases, see below) floats are
> rounded, not truncated such that -4.00004 and -3.99995 are
> both displayed as -4.0000. Therefore you can't tell whether
> -4.0000 is less or greater than -4.
> 
> octave:15> 3.9999
> ans =  3.9999
> octave:16> 3.99995
> ans =  4.0000
> octave:17> 0.99995
> ans =  0.99995
> octave:18> 0.999995
> ans =  0.99999  # I'm expecting 1.00000 here
> octave:19> 0.9999995
> ans =  1.00000
> octave:20> 0.999996
> ans =  1.00000  # compare this to command 18
> 
> Regards,
> Lukas
> 

There is a setting (I don't remember it off the top of my head) which 
controls default display accuracy, i.e. default number of significant
digits.

Set it to meaningful for 'double' maximum in order to avoid such
problems.

...

The setting used to be

output_precision(14);
.

Regards,
  Sergei.


      



reply via email to

[Prev in Thread] Current Thread [Next in Thread]