Re: Strange behaviour of median and std

[Jaroslav Hajek]

On Wed, Feb 17, 2010 at 12:53 PM, Marius Roets <address@hidden> wrote:
> Hi everybody,
>
> It took me an hour to get to the bottom of this. Can somebody please explain
> the output of this code to me.
>
> b = [255 255 0; 255 255 255];
> disp(b);
> median(b(:))
> std(b(:))
> b = uint8(b)
> median(b(:))
> std(b(:))
>
> Output:
>
>    255   255     0
>    255   255   255
> ans =  255
> ans =  104.10
> b =
>
>   255  255    0
>   255  255  255
>
> ans = 128
> ans =  43
>
> Thanks
> Marius
>

The uint8 case is performed using integer arithmetics, which is
saturated and truncated to integers, hence the incorrect results.
An easy remedy is to convert integers to doubles in "center". median
is already computed correctly in the development version.

http://hg.savannah.gnu.org/hgweb/octave/rev/e0767a0965f1

thanks

-- 
RNDr. Jaroslav Hajek, PhD
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz