Re: Constrained non linear regression using ML

[Fredrik Lingvall]

On 03/18/10 18:21, Corrado wrote:
> Dear Fredrik, Octavers,
>
> I am firstly trying the maximum likelihood approach. The likelihood
> function, and the log likelihood function, will depend on the pdf of
> the error e in the formula:
>
> y=f(theta*x)+e
>
> Now let's say that e is  Gaussian distributed, then I can use LS which
> is the same as ML in this case as we said.
> The residuals in this case would be distributed Gaussian. Is that right?
The likelihood function, L(theta), would in this case be:

p(y|theta,I) = L(theta) = 1/(2 pi)^(n/2) * 1/sqrt(det(Ce)) * exp(-0.5*
(y-f(theta*x))'*inv(Ce)*(y-f(theta*x)));

where Ce is the covariance matrix for e [e ~ N(0,Ce)]. ML is more
general than LS and in the Gaussian case they are the same only when Ce
is diagonal with the same variance for all elements [Ce =
sigma_e^2*eye(n,n)].

>
> If e is distributed differently (for example: beta, in the continuous
> case,  or binomial, in the discrete case), then I am better off by
> using Maximum Likelihood.
If you have such knowledge then yes.

If would recommend that you try a maximum a posteriori approach (MAP)
instead (of ML) since you have some important prior knowledge about your
parameters, theta - they are positive. For example, a positive
exponential distribution for your parameters seems like a good choice.
Try to maximize,

theta_est = arg max    lambda_theta*exp(-lambda_theta * theta) * L(theta)
                        theta

for some large but finite lambda_theta (use a large value when you don't
know much about the scale of your parameters). The exponential
distribution has the max entropy property when it is know that theta is
positive (which the Gaussian has for parameters that can be both pos and
neg) which makes it a "safe" assumption. 

To be really careful, if you don't have any clue on the scale of your
parameters then you should integrate out (marginalize)  lambda_theta
which gives you more robust estimates. Again I recommend these books:

> http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521841504 but
> also Larry Bretthorst's book (available for download here:
> http://bayes.wustl.edu/glb/bib.html) and papers.
>

/Fredrik