Re: Statistical test for equality ?

[pathematica]

Again, my attempt to explain will be hampered by fading memory and, possibly,
imperfect understanding of the subject from my relatively brief exposure to
it. However, here is an attempt. Please excuse my summarising things that
everyone will know; I do this in an attempt to identify the concept behind
the Likelihood test, to explore whether it might do what you want it to do.
I'm afraid it's not going to be much of a recipe for doing the test. To do
that, it will be necessary to identify the number of degrees of freedom for
the particular test you will undertake, and to calculate the likelihoods so
that the statistic might be compared against a suitable member of the Chi
square distribution. 

As I remember it, the likelihood ratio test is based in Bayesian rather than
frequentist statistics. Roughly speaking, frequentist statistics is
concerned with identifying a notional "average" representative of something
that might be measured, together with a range of values described as a
"confidence interval" (typically derived from the standard error of the
mean, which is often taken as the exemplar of averageness) which attempts to
quantify the probability that the true mean (which is not known) lies
somewhere in the region of the estimated mean. The mathematical models
assume that errors in measurement follow some probability distribution,
typically but not necessarily Gaussian. The parameters of the model are
viewed as fixed and attempts are made to find them (eg mean and standard
deviation). Experimental measurements are viewed as variables and the
pattern of distribution of data points is predicted using the models using
the estimated parameters, with decisions made about the probability of
observing particular values given the parameters. Examples of the flaws
inherent in frequentist methods are highlighted in such jokes as "The
average human being has one boob and one testicle". 

Bayesian statistics treats parameters as variables rather than fixed. In
contrast, observed measurements are treated as fixed (constants after
measurement). Once again mathematical models are required to draw inferences
about the probability of observing something. The probability distributions
that describe the probability of observing some measurement, given some
particular values of parameters are similar (eg mean and variance of a
normal distribution). However, extra probability distributions are required
that describe the distribution of the parameters that have been used in the
probability density distribution that models the errors in measurements.
These are the prior and the posterior distributions. The prior distribution
summarises belief "so far" about the value of the parameters before some
particular set of measurements is taken. This is combined with the
experimental data (which are treated as fixed but which are modelled as
though they have been sampled from some particular probability distribution)
to derive the posterior distribution. The posterior distribution summaries
updated belief about the value of the parameters in the distribution that
models error in measurements given the set of data that have been sampled. 

The use of the coin tossing example simplifies discussion because it is
typically modelled by a Bernoulli distribution, which has only one
parameter, p (the probability of observing one of two possible outcomes, eg
heads). The prior (and the posterior) distribution that describes belief in
the value of p will appear odd to frequentists because p can only take
values between 0 and 1, so it will be defined only on this interval, and the
area beneath the kernel of the distribution will be normalised to 1 by a
suitable normalising factor. Given the nature of the thing being modelled, a
distribution often used as a prior for a Bernoulli trial is the Beta one (eg 
Wikipedia page on Beta distribution
<http://en.wikipedia.org/wiki/Beta_distribution >  ). It is defined on
[0,1]. Like other priors/posteriors for Bayesian modelling, it might be
multimodal given its parameters (the Beta distribution has two parameters). 

In the likelihood ratio test, the two sets of data would be used to
calculate the likelihood for each given the particular distribution with its
particular parameter that has been used to model the process (here,
Bernoulli(0.5) would seem sensible). The posterior would provide a
distribution of the probability that p takes some particular value given the
data (note this is a probability of a particular probability, with the words
used in a subtly different way). For a fair coin, it would be expected that
the posterior would be a Beta distribution with a single mode somewhere near
0.5. The nearest thing to a confidence interval for the value of p (the one
which is the parameter for the Bernoulli distribution) would be a credible
interval; as posteriors may be multimodal, it is often not possible merely
to bracket some mode to find an interval whose area is some proportion of
one, modelling the probability of observing that value for it. It is often
also inappropriate to form a symmetrical interval about a mode, as you might
imagine the shape of the curve in which some mode is not located at 0.5.
Instead, a decision must be made about the way that the credible interval is
found. While others exist, a way that has merit is called the "highest
posterior density" (often abbreviated to HPD) to find some credible interval
(or credible region, which might comprise the union of two separate
intervals for some multimodal distribution). The interval(s) are bounded by
the values of p for which the likelihood takes the same value (ie the bound
form the projection onto the x axis of the intersections with the graph of a
horizontal line drawn across the graph so that the area encompassed by the
bounds provides the desired credible interval; note that this might define
more than one separate interval for some multimodal graph, depending on the
height of the horizontal line). 

Anyway, in the likelihood ratio test, (in the case of tossing two separate
coins), you would be testing the hypothesis that the respective values of p
for each of the coins are the same (that is, you are seeing whether the two
coins are "equally fair" or "equally biased"). This is another way of
quantifying the probability that the two sets of data have been sampled from
the same distribution (or, more strictly, the same distribution with the
same parameter). In the last sentence, the word "distribution" refers to the
one modelling the error in the measurement of the data rather than the ones
(ie the prior and the posterior) modelling the beliefs about the values of
the parameters for the sampling distribution before and after the data have
been sampled.  




-----
However good you think Octave is, it's much, much better.
--
View this message in context: 
http://octave.1599824.n4.nabble.com/Statistical-test-for-equality-tp4660142p4660405.html
Sent from the Octave - General mailing list archive at Nabble.com.