Re: Solving large matrix equations

[Jonathan Stickel]

On 4/7/09 address@hidden wrote:
> Date: Tue, 7 Apr 2009 09:55:57 +0200
> From: "Martin Heller" <address@hidden>
> Subject: Solving large matrix equations
> To: address@hidden
> Message-ID: <address@hidden>
>
> I am new to Octave and am trying to write a function for differentiating 
> some
> data based on this paper:
> <http://math.lanl.gov/Research/Publications/Docs/chartrand-2007-numerical.pdf>
>
> My attempt is shown below and it seems to work but my data sets contain
> 10000-15000 points which is too much for Octave to handle when solving
> the problem using my simple minded approch.
>
> I was wondering if there is a standard way to solve large matrix problems in
> Octave that I could take advantage of? At the moment I just feed smaller
> chunks of data to my function and this gets the job done.


Martin

I think you will need to provide more details to have your general 
question answered:  the appropriate method for solving large matrix 
equations of course depends on the system.  What do you mean when you 
say your system is "too much for Octave to handle"?  Are you running out 
of memory?  Does it take too long?  There are people much more qualified 
to help you with these general issues than I am.

However, I have a particular interest in your application.  I have 
studied data smoothing and differentiation quite a bit in the last year 
and have recently submitted a paper on the subject (I can share off-list 
if you like).  Is your "real data" (not your example test data) 
discontinuous or does it have a discontinuous derivative?  If not, then 
you likely can use a squared term for your roughness rather than the 
absolute value.  This case has an analytical solution that does not 
require an iterative method (Eilers P.H.C, 2003, Anal. Chem. 75, 
3631-3636).  I have implemented this method and provided it in the 
Octave package data-smoothing 
(http://octave.sourceforge.net/data-smoothing/index.html).  However, I 
have found that my implementation is often ill-conditioned for data with 
more than ~1000 points.

At a first glance, it looks like you have made a good start with 
Chartrand's method.  Would you consider submitting your function to 
Octave-Forge?  I think it would make a nice addition to the 
data-smoothing package.

Regards,
Jonathan