Re: reading ascii matrix

[Paul Kienzle]

If the data is too large for your machine, you may want to process it
in blocks.  E.g.,

for i=1:p/1e5+1
   blk=fscanf(myfile,"%f",2e5);
   blk=reshape(blk,length(blk)/2,2);
   a=blk(:,1); b=blk(:,2);
   ...
end

Note that a,b will probably not contain 1e5 data points each on
the final iteration.

On Fri, Feb 09, 2001 at 09:25:40AM -0500, Ben Bauer wrote:
> Info:
> Octave, version 2.1.9 (i686-pc-linux-gnu). 
> (PIII 550 with 256M ram)
> Linux  2.2.16
> egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)  
> 
> 
> 
> I have a plain ascii file of 2 columns and
> approx 2.8 million rows that I need to do some
> statistics on. In my octave script I have 
>   
>  for i=1:p
>   [a(i),b(i)]=fscanf(myfile,"%f %f","C"); 
>   ...
> 
> where p is the number of records in the file.
> 
> 
> This program has been reading in the data for
> 2 days now.  Currently just less than half the
> data are read in according to a print command I
> have in the loop.  There must be a better way!
> The the entire script runs in about 1.5 minutes
> for a shorter file of 20000 data pairs, with the
> majority of the time taken in reading the data.
> 
> 
> 
> Any useful suggestions would be appreciated.  Thanks.
> 
> 
> 
> -------------------------------------------------------------
> Octave is freely available under the terms of the GNU GPL.
> 
> Octave's home on the web:  http://www.octave.org
> How to fund new projects:  http://www.octave.org/funding.html
> Subscription information:  http://www.octave.org/archive.html
> -------------------------------------------------------------
> 
> 



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------