Re: Reading array slow ( and dlmread bug )

[Paul Laub]

Peter, 

If speed of reading and writing data is critical to you, then you
might consider writing an external  C or Perl program converting your
data into Octave's own binary format. Binary format data loads
extremely fast.  Or, more simply, load your ASCII data files once into
Octave and then export each of them in binary.  Use the "save -binary"
command. Then every subsequent loading of your data will be fast.

Paul Laub  

On Mon, 21 Feb 2005 10:54:48 +0000, peter <address@hidden> wrote:
> 
> I tried a new construct this morning,
> however it is still a factor of 10 slower
> than the equivalent perl.
> 
> rdiff_2.m
> --------------------------------
> #!/usr/bin/octave -q
> fid = fopen("new.out","rt") ;
> while( (line = fgetl(fid)) != -1  )
>   ;
> endwhile
> 
> Peter
> 
> peter wrote:
> 
> >
> > I am writing a small utility program to  compare
> > files containing numerical data.
> >
> > I found that reading the files (with octave) was very slow
> > compared to perl.  The format of the data  files is shown
> > below.
> >
> > 38157    10872    33592    26089    47591    14720    45843    16639
> >   .....
> >   ......
> >   55002    29660    45918    52316      36063    56511    31388
> > 36950
> > I have enclosed a small function to generate
> > the data (generate.m)
> >
> > I tried to use "load" and "dlmread". The execution speed
> > is show below (slightly edited output).
> >
> > Sun Feb 20 18:40:50 GMT 2005
> > octave dlmread
> > ans = 100000  11   <=== incorrect size of array returned by "dlmread"
> >
> > Sun Feb 20 18:40:56 GMT 2005
> > Octave load
> > ans = 100000   10
> >
> > Sun Feb 20 18:41:17 GMT 2005
> > perl
> > Sun Feb 20 18:41:18 GMT 2005
> >
> > In this case "dlmread" took 6 seconds and "load" took 21
> > seconds. The perl program took 1 second .....
> >
> > I think there is a bug in  "dlmread"  as empty space followed
> > by "\n" is seen as a zero value. The size for the array  is therefore
> > reported incorrectly.
> >
> > I think the difference is remarkable, given that
> > the octave program is essentialy executing "C"
> > code.
> >
> > Please find enclosed the code snippets I used.
> >
> > rdiff.sh
> > -----------------------
> > #!/bin/bash
> > date
> > echo "octave dlmread"
> > ./rdiff.m
> > date
> > echo "Octave load"
> > ./rdiff_1.m
> > date
> > echo "perl"
> > ./rdiff.pl
> > date
> >
> > rdiff.m
> > ---------------------
> > #!/usr/bin/octave -q
> > a = dlmread("new.out"," ") ;
> > size(a)
> >
> > rdiff_1.m
> > --------------------
> > #!/usr/bin/octave -q
> > a = load("new.out") ;
> > size(a)
> >
> > rdiff.pl.
> > -------------------------
> > #!/usr/bin/perl
> > open(fid,"new.out");
> >
> > while($record = <fid>) {
> >  @number = split(/[ \t]+/,$record) ;
> >  #print("@number\n");
> > }
> > close(fid);
> >
> >
> > generate.m
> > -------------------------------
> > #!/usr/bin/octave -q
> >
> > 1 ;
> >
> > function save_array(name,x)
> >  fid = fopen(name,"wt") ;
> >  for n=1:rows(x)
> >    fprintf(fid,"%8d ",x(n,:)) ;
> >    fprintf(fid,"\n") ;
> >  endfor
> >  fclose(fid) ;
> > endfunction
> >
> > N = 100000 ;
> > a = round(rand(N,10)*2^16) ;
> > save_array("new.out",a) ;
> >
> >
> >
> > -------------------------------------------------------------
> > Octave is freely available under the terms of the GNU GPL.
> >
> > Octave's home on the web:  http://www.octave.org
> > How to fund new projects:  http://www.octave.org/funding.html
> > Subscription information:  http://www.octave.org/archive.html
> > -------------------------------------------------------------
> >
> >
> 
> -------------------------------------------------------------
> Octave is freely available under the terms of the GNU GPL.
> 
> Octave's home on the web:  http://www.octave.org
> How to fund new projects:  http://www.octave.org/funding.html
> Subscription information:  http://www.octave.org/archive.html
> -------------------------------------------------------------
> 
>



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------