Re: cannot load large matrix, new datatype/fseek

[daniel heiserer]

Hi,

> I have a large matrix in the following mat file:
> 8153243903 14. Jun 16:49 ftn51.mat
> 
> The matrix is approximately 5,000,000 x 200 and needs around 8GB of
> space.
> Unfortunately octave cannot read the matrix. It either stops immediatley
> on my 32bit machine (of course) or on a 8GB 64bit machine it simply
> crashes.
> 
> I only need a few rows of that matrix, but all columns.
> 
> I have two attesmpts for reading that matrix:
> 
> 1) write an m-file which reads only the elements I want.
> For that I am not sure if I can "naviagte" using fseek in a file larger
> then 2GB.
> If anyone up to know has written an m-file based .mat parser I would be
> glad for
> any help

So this idea is bad. I am even not able to open the file:

[myfile,msg] = fopen("ftn51.mat", "r", "ieee-le")
myfile = -1
msg = File too large

What has to be done to improve this? Must I change the octave related
int4's to int8s?


-- daniel

> 2) Implement a new data type OOC-matrix. This datatype holds only the
> few values
> describing the matrix in memory (imaginary flag, n-columns,n-rows).
> The rest of the matrix is a blob on disk.
> Whenever this datatype is used in a function or whatever it shall load
> the necessary elements from disk.
> As I haver never implemented a new datatype I ask for some documentation
> or examples so I can start realizing it. Also I must somehow be able to
> tell "load" that the  matrix in the file shall be assigned to that
> datatype.
> 
> I am glad for any help,
> 
> daniel

-- 
Mit freundlichen Gruessen
    Daniel Heiserer

--------------------------------------------------------------
Dipl.-Phys. Daniel Heiserer, BMW AG, Knorrstrasse 147, 80788 Muenchen
Abteilung EK-212
Tel.: 089-382-21187, Fax.: 089-382-42820
mailto:address@hidden



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------