Re: Reading array slow ( and dlmread bug )

[peter]

I tried a new construct this morning,
however it is still a factor of 10 slower
than the equivalent perl.

rdiff_2.m
--------------------------------
#!/usr/bin/octave -q
fid = fopen("new.out","rt") ;
while( (line = fgetl(fid)) != -1  )
 ;
endwhile

Peter

peter wrote:

> I am writing a small utility program to  compare
> files containing numerical data.
>
> I found that reading the files (with octave) was very slow
> compared to perl.  The format of the data  files is shown
> below.
>
> 38157    10872    33592    26089    47591    14720    45843    16639   
>   .....
>   ......
>   55002    29660    45918    52316      36063    56511    31388    
> 36950  
> I have enclosed a small function to generate
> the data (generate.m)
>
> I tried to use "load" and "dlmread". The execution speed
> is show below (slightly edited output).
>
> Sun Feb 20 18:40:50 GMT 2005
> octave dlmread
> ans = 100000  11   <=== incorrect size of array returned by "dlmread"
>
> Sun Feb 20 18:40:56 GMT 2005
> Octave load
> ans = 100000   10
>
> Sun Feb 20 18:41:17 GMT 2005
> perl
> Sun Feb 20 18:41:18 GMT 2005
>
> In this case "dlmread" took 6 seconds and "load" took 21
> seconds. The perl program took 1 second .....
>
> I think there is a bug in  "dlmread"  as empty space followed
> by "\n" is seen as a zero value. The size for the array  is therefore
> reported incorrectly.
>
> I think the difference is remarkable, given that
> the octave program is essentialy executing "C"
> code.
>
> Please find enclosed the code snippets I used.
>
> rdiff.sh
> -----------------------
> #!/bin/bash
> date
> echo "octave dlmread"
> ./rdiff.m
> date
> echo "Octave load"
> ./rdiff_1.m
> date
> echo "perl"
> ./rdiff.pl
> date
>
> rdiff.m
> ---------------------
> #!/usr/bin/octave -q
> a = dlmread("new.out"," ") ;
> size(a)
>
> rdiff_1.m
> --------------------
> #!/usr/bin/octave -q
> a = load("new.out") ;
> size(a)
>
> rdiff.pl.
> -------------------------
> #!/usr/bin/perl
> open(fid,"new.out");
>
> while($record = <fid>) {
>  @number = split(/[ \t]+/,$record) ;
>  #print("@number\n");
> }
> close(fid);
>
>
> generate.m
> -------------------------------
> #!/usr/bin/octave -q
>
> 1 ;
>
> function save_array(name,x)
>  fid = fopen(name,"wt") ;
>  for n=1:rows(x)
>    fprintf(fid,"%8d ",x(n,:)) ;
>    fprintf(fid,"\n") ;
>  endfor
>  fclose(fid) ;
> endfunction
>
> N = 100000 ;
> a = round(rand(N,10)*2^16) ;
> save_array("new.out",a) ;
>
>
>
> -------------------------------------------------------------
> Octave is freely available under the terms of the GNU GPL.
>
> Octave's home on the web:  http://www.octave.org
> How to fund new projects:  http://www.octave.org/funding.html
> Subscription information:  http://www.octave.org/archive.html
> -------------------------------------------------------------



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------