Kolmogorov-Smirnov test

[Hamish Allan]

Hi,

I've recently been trying to use kolmogorov_smirnov_test_2() to  
compare two distributions. In order to get a feel for it, I thought  
I'd try comparing two normal distributions:

octave:1> a = randn(2000,1);
octave:2> b = randn(2000,1);
octave:3> p = kolmogorov_smirnov_test_2(a,b)
p = 0.90224
octave:4> b = randn(2000,1);
octave:5> p = kolmogorov_smirnov_test_2(a,b)
warning: in /opt/local/share/octave/2.1.71/m/statistics/tests/ 
kolmogorov_smirnov_test_2.m near line 80, column 5:
warning: cannot compute correct p-values with ties
p = 0.98453

[These numbers look okay to me, though the warning I don't understand  
(is the K-S test not used to compare e.g., distributions of school  
grades, A through F, for which there would be many ties?). But that's  
not my biggest problem:]

octave:6> b = randn(2000,1);
octave:7> p = kolmogorov_smirnov_test_2(a,b)
p = 0.50849

[That's a pretty low p-value! Is the K-S test actually going to be  
useful if it can be so wrong about the normally-distributed datasets  
as large as 2,000 points? Let's try bumping it up to 20,000...]

octave:8> a = randn(20000,1);
octave:9> b = randn(20000,1);
octave:10> p = kolmogorov_smirnov_test_2(a,b)
warning: in /opt/local/share/octave/2.1.71/m/statistics/tests/ 
kolmogorov_smirnov_test_2.m near line 80, column 5:
warning: cannot compute correct p-values with ties
p = 0.10441

[Huh?! One more time for luck...]

octave:11> b = randn(20000,1);
octave:12> p = kolmogorov_smirnov_test_2(a,b)
warning: in /opt/local/share/octave/2.1.71/m/statistics/tests/ 
kolmogorov_smirnov_test_2.m near line 80, column 5:
warning: cannot compute correct p-values with ties
p = 0.83675

[Nought point one versus nought point eight? I give up!]

So, is the K-S test really this weak, or is there a bug in the Octave  
implementation?

Thanks,
Hamish



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------