Re: Checking GSL for Spectroscopy

[Fritz Sonnichsen]

Mark
  I am converting someone's MATLAB code so I am not sure what he is doing
yet--but several years ago I did spectral analysis in MATLAB and probably
very similar. This is for Raman and LIBS spectra.
1) "Usually" I apply a high pass filter to the spectrum. This gets rid of
the noise I need control over this since as you would expect the signal and
noise can get pretty close! Intuition comes into play here.
2) Next I baseline the spectra. This removes any constant bias.  For LIBS I
was usually able to further filter "spikes" and then take a mean of the
remaining line, subtracting this from the overall spectrum. Raman can get a
bit more difficult-I am, at least,  subtracting the fluorescent line which
can have a lot of features (e.g. spikes). At times, if you know this
background you can subtract it first but you get all types of complications
from normalization. Again--intuition comes into play.
3) The resulting spectrum needs to be compared to a database. For LIBS the
latter is quite small--mostly atomic/elemental data such as NIST. I could
generally do a discrete comparison of the spike locations using a
peak-finder, align them with the known examples and get a pretty high hit
rate. This was for qualitative data.
Raman is, again, much more complex. The data I was using was constrained
and simpler but the case in hand here is much more complex. We are doing
mixed plastics at the moment. My colleague found the best matches by taking
a stats correlation with 44000 entries and pulling out the values closest
to "one". It works remarkably well.

I don't think there is much above that cannot be written in C in a
reasonable amount of time. But we are looking ahead and would like to draw
on the collective experience of the science community. This type of
analysis is quite common and there are enough new wheels out there that we
don't want to re-invent old ones!
    Very important is that "intuition" part. I would think a lot of this
issue has been better solved since I was doing this. There are a lot of
adjustments that could be made-for example iterating trial baselines,
rejecting noise at varied levels etc. Processors are faster now and the AI
movement has brought in PCA and a lot of other techniques that begin to
transcend my current state of knowledge (I work more on the physics end of
things and would prefer to use routines from the communities if possible to
save time).

Thanks for your interest Mark!
Fritz


On Tue, Mar 16, 2021 at 9:25 AM Mike Marchywka <marchywka@hotmail.com>
wrote:

> Can you comment on how you compare spectra? Just for my own
> personal interest, not sure if will further the thread here however..
> Not sure a "dot product" in the conventional sense would help much.
> You could imagine comparing peak positions and relative heights
> or a fit to a continuum for example.  Peaks plus black body in some
> vector comparison?
>
> note new address
>  Mike Marchywka 306 Charles Cox Drive Canton, GA 30115
>  2295 Collinworth  Drive Marietta GA 30062.  formerly 487 Salem Woods
> Drive Marietta GA 30067 404-788-1216 (C)<- leave message 989-348-4796 (P)<-
> emergency
>
>
> ________________________________________
> From: Help-gsl <help-gsl-bounces+marchywka=hotmail.com@gnu.org> on behalf
> of Fritz Sonnichsen <sonnichs@gmail.com>
> Sent: Tuesday, March 16, 2021 9:15 AM
> To: help-gsl@gnu.org
> Subject: Checking GSL for Spectroscopy
>
> I am preparing to convert MATLAB code to something more general. The new
> code will run on LInux and ARM processors.
>    For a lot of reasons I am not going to use Python. We also want to
> keep this project "close" to scientists and do not want to turn it into a
> full time computer programming job. So the final word is that I am looking
> for something that can be called by (and hopefully is written) in C. Worse
> case I will just write the code myself but would prefer to start
> integrating our systems into something with a lot of pre-written and vetted
> routines.
>
> GSL looks like a good choice. Maybe R comes next. We have a mix of needs
> but I will point out a few:
> 1) Baselining a spectrum
> 2) Finding peaks in that spectrum
> 3) using Pearson correlation to compare the spectrum QUICKLY to
> about 50,000 recorded examples.
>
> We also have some uses with basic statistics and we do some image
> processing.
>
> So my question is--does GSL position itself in these areas? MATLAB (with
> packages) does them all.
>      I am not sure how active GSL, if it is keeping up with AI, imaging and
> spectroscopy--or is it fading or giving way to popular languages for
> example. I was surprised that the 600+ page manual did not seem to show
> anything relating to the simple spectral analysis described above for
> example. Certainly I can search the web for others' code but at some point
> if I cannot attach to a well established product I will just write it
> myself.
>
> Any comments appreciated
> thanks
> Fritz
>