Re: OT: finding the weights used in weighted least squares regression

[Mike Miller]

On Tue, 26 Apr 2011, Kamaraju S Kusumanchi wrote:

> The weighted least squares regression is done by solving
> W A x = W b
> for x, where W is nxn, A is nxm, x is mx1, b is nx1. W is a diagonal matrix.
>
> My question is that, in the above set up, if we are given A, x, b is it 
> possible to solve for the W?


We usually have an equation of this form:

y = X*b + e

The y vector (nx1) and X matrix (nxp) are given and we want to solve for b 
(px1) attempting to minimize the variance of hypothetical e (nx1) -- the 
least squares solution, but we assume that the variance-covariance matrix 
of e is of the form eye(n)*s^2 for some s.  We can estimate s.

But what if the var-covar matrix of e is of a different form?  Let's give 
it the general form V (nxn, not necessarily diagonal) where V is symmetric 
non-negative definite.  It turns out that if V is positive definite (all 
positive eigenvalues), you can produce a matrix U=chol(inv(V)) such that

U*y = U*X*b + U*e

where U*e has var-covar matrix eye(n).  This makes it possible to get a 
proper solution for b like so:

b = (U*X)\(U*y)

I assume that your "A x = b" is what I would call "X b = y".  In other 
words, I think the W you want is the U I've given.

This means that you have to know something about the structure of your 
residuals.  Are there groups of observations that allow you to estimate 
residual variances for the elements of e?  If so, you can use an iterative 
feasible generalized least squares method:

(1) b = X\y ;   [implicitly assumes cov(e) = eye(n)*s^2]

(2) eOLS = y - X*b ;  [compute residuals]

(3) compute var-covar matrix V based on eOLS -- I can't tell you how 
because I don't know the structure of your data.

(4) U = chol(inv(V)) ;

(5) b = (U*X)\(U*y) ;

(6) eGLS = y - X*b ;

(7) compute var-covar matrix V based on eGLS -- again, I can't tell you 
how because I don't know the structure of your data.

(8) repeat steps 4-7 until b stops changing.  It should not require many 
iterations.

Mike

--
Michael B. Miller, Ph.D.
Bioinformatics Specialist
Minnesota Center for Twin and Family Research
Department of Psychology
University of Minnesota