Re: Slowness in function 'open'

[Michael Creel]

Mark B. wrote:
> Hi,
>
> Is there a faster way to load a big text file composed only of integers? I
> am having severe performance penalties by using the load function
>
> These are the results of my tests:
>
> First test:
> ============================================
> Octave, version 2.9.12 (configured for "i686-pc-msdosmsvc")
>
> octave.exe:1> tic; load('mybigfile'); toc;
> Elapsed time is 98.314560 seconds.
>
> Matlab 7.3.0.267 (R2006b)
>
> > > tic; load('mybigfile');toc;
> Elapsed time is 27.923677 seconds.
>
> Octave, version 2.1.73 (i686-pc-cygwin)
>
> octave:1> tic; load('mybigfile');toc
> ans = 1117.5
> ============================================
>
> Between calls to the 'load' function I flushed the cache of the location
> where the file resides
>
> Second test:
> ============================================
> octave.exe:2> tic; load('mybigfile'); toc;
> Elapsed time is 93.594170 seconds.
>
> > > tic; load('mybigfile');toc;
> Elapsed time is 28.757878 seconds.
> ============================================
>
> (Note: I cannot save the file in binary format because of incompatibility
> with the C++ program that generates the text file)
>
> Is this a known issue?
>
> Thanks

Just to add a data point, I can load a 191MB text file in 19 s., using Octave 
2.9.12 on linux:
octave:6> tic; load a; toc
Elapsed time is 18.852654 seconds.
octave:7> ls -lh a
-rw-r--r-- 1 michael michael 191M 2007-06-22 09:43 a
octave:8>
I've always been anti-binary for storage of data that might be re-used sometime 
in the unforeseen future. But for short term data that gets thrown away after 
use, binary probably makes more sense.