Re: Optimal way to handle big data table ?

[Sergei Steshenko]

----- Original Message -----
> From: CdeMills <address@hidden>
> To: address@hidden
> Cc: 
> Sent: Thursday, March 7, 2013 12:07 PM
> Subject: Optimal way to handle big data table ?
> 
> Hello,
> 
> I was doing recently lamp spectrum analysis to extract photometric
> properties. This implies to compute the integral of the spectrum by the
> CIE1931 sensitivity functions; they are tabulated at 400 wavelength, each
> time 4 values. What's the best way to use those data inside a function ?
> 1) encode them inside the function body ? It will be compiled once.
> CIE31Table = [360 0.000130 0.000004 0.000606
>               361 0.000146 0.000004 0.000681
>               362 0.000164 0.000005 0.000765 ... ];
> 2) read them from a text file ?
> 3) read them from a binary file ?
> 4) other ?
> 
> The point is to minimise the computational load of each time refilling this
> matrix with its 1600 entries.
> 
> Regards
> 
> Pascal
> 
> 
> 
> --
> View this message in context: 
> http://octave.1599824.n4.nabble.com/Optimal-way-to-handle-big-data-table-tp4650575.html
> Sent from the Octave - General mailing list archive at Nabble.com.
> _______________________________________________
> Help-octave mailing list
> address@hidden
> https://mailman.cae.wisc.edu/listinfo/help-octave
>


This is not that big - I, for example, deal with loading about 1e6 audio 
samples in ASCII, and it takes, say, half a minute or so.

Anyway, the obvious way is to first create your table in ASCII, load it, then 
save it as binary, and then load it as needed from the binary.

Regards,
  Sergei.