Re: hdf5

[Daniel Heiserer]

>   Daniel> If hdf5 is not a database, is it hard to "combine" some databases
>   Daniel> like BerkeleyDB or GDB with hdf5 to create a scientific database?
> 
> You don't say what you want to do with your data, whether it changes, whether
> you need subqueries etc

I just posted a message. I do not want todo subqueries, I want to use it 
to store all my data, which means, I have load, then change and then
save
the data again. Between two commands, I do not want to have nearly
anything in core. Of course it can be, that the matrix I store again
grows. And of course I want to delete files in the 'database'.
The reason I want the out of core capability is that I 
have to use many large matrices. 
A first step would be to load only the ones, which  I need for an
operation.
A second step should be to load only the part of a matrix I need 
for a function like a column or something like that.

> If you "merely" want to load and save lots of (static) matrices, I would
> conjecture that the builtin formats load and save will be hard to beat in
> terms of both speed and storage efficiency. You can easily build yourself
> small C, C++, Perl, ... programs that read/write in the same format as it
> "really" is just double array which you can store / retrieve with one
> command.
> 
> If however you want to query "relations" between your data, then a relational
> database might be your choice.  I have used Octave with timeseries data on
> Sybase (expensive, proprietary) as well as PostgreSQL (free, BSD-style
> copyright).  I used Perl glue code (called from an .m file) back when I used
> Sybase, and I wrote two C++ .oct functions I posted here a long time ago
> which go directly to PostgreSQL.  Just as a reference, Bill Sharpe at
> Stanford (who is also a winner of an Economics Nobel price) has a little
> timeseries database system which is implemented direcly in Matlab and which
> might be portable.
> 
> That being said, there are limits to what you can do due to design of the
> Octave (Matlab) language.  For full-blown data analysis, the S language is
> better, and I have very good recent experiences with the GNU R implementation
> of it and both the R interface to PostgreSQL as well as the ODBC interface.
> 

Will the desing of Octave allow an efficient implementation 
of out of core handling of large (maybe sparse) matrices?

daniel



-----------------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.che.wisc.edu/octave/octave.html
How to fund new projects:  http://www.che.wisc.edu/octave/funding.html
Subscription information:  http://www.che.wisc.edu/octave/archive.html
-----------------------------------------------------------------------