[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Is this a numerical problem?
From: |
Sergei Steshenko |
Subject: |
Re: Is this a numerical problem? |
Date: |
Thu, 19 Aug 2010 04:16:09 -0700 (PDT) |
--- On Thu, 8/19/10, Lukas Reichlin <address@hidden> wrote:
> From: Lukas Reichlin <address@hidden>
> Subject: Re: Is this a numerical problem?
> To: "Jaroslav Hajek" <address@hidden>, address@hidden
> Date: Thursday, August 19, 2010, 3:52 AM
> Hi Jaroslav
>
> I realized that (in most cases, see below) floats are
> rounded, not truncated such that -4.00004 and -3.99995 are
> both displayed as -4.0000. Therefore you can't tell whether
> -4.0000 is less or greater than -4.
>
> octave:15> 3.9999
> ans = 3.9999
> octave:16> 3.99995
> ans = 4.0000
> octave:17> 0.99995
> ans = 0.99995
> octave:18> 0.999995
> ans = 0.99999 # I'm expecting 1.00000 here
> octave:19> 0.9999995
> ans = 1.00000
> octave:20> 0.999996
> ans = 1.00000 # compare this to command 18
>
> Regards,
> Lukas
>
There is a setting (I don't remember it off the top of my head) which
controls default display accuracy, i.e. default number of significant
digits.
Set it to meaningful for 'double' maximum in order to avoid such
problems.
...
The setting used to be
output_precision(14);
.
Regards,
Sergei.
- Is this a numerical problem?, Lukas Reichlin, 2010/08/19
- Re: Is this a numerical problem?, Jaroslav Hajek, 2010/08/19
- Re: Is this a numerical problem?, Lukas Reichlin, 2010/08/19
- Message not available
- Re: Is this a numerical problem?, Jaroslav Hajek, 2010/08/19
- Re: Is this a numerical problem?, Lukas Reichlin, 2010/08/19
- Re: Is this a numerical problem?,
Sergei Steshenko <=
- Re: Is this a numerical problem?, Jaroslav Hajek, 2010/08/19
- Re: Is this a numerical problem?, Miguel Bazdresch, 2010/08/19
- Re: Is this a numerical problem?, Lukas Reichlin, 2010/08/19
- Re: Is this a numerical problem?, Jaroslav Hajek, 2010/08/19
- Re: Is this a numerical problem?, Lukas Reichlin, 2010/08/19