Re: Slowness in function 'open'

[David Bateman]

Muthiah Annamalai wrote:
> 
> 
> On 6/21/07, *Mark B.* <address@hidden
> <mailto:address@hidden>> wrote:
> 
> 
>     Hi,
> 
>     Is there a faster way to load a big text file composed only of
>     integers? I
>     am having severe performance penalties by using the load function
> 
>     These are the results of my tests:
> 
>     First test:
>     ============================================
>     Octave, version 2.9.12 (configured for "i686-pc-msdosmsvc")
> 
>     octave.exe:1> tic; load('mybigfile'); toc;
>     Elapsed time is 98.314560 seconds.
> 
>     Matlab 7.3.0.267 (R2006b)
> 
>     >> tic; load('mybigfile');toc;
>     Elapsed time is 27.923677 seconds.
> 
>     Octave, version 2.1.73 (i686-pc-cygwin)
> 
>     octave:1> tic; load('mybigfile');toc
>     ans = 1117.5
>     ============================================
> 
>     Between calls to the 'load' function I flushed the cache of the location
>     where the file resides
> 
>     Second test:
>     ============================================
>     octave.exe:2> tic; load('mybigfile'); toc;
>     Elapsed time is 93.594170 seconds.
> 
>     >> tic; load('mybigfile');toc;
>     Elapsed time is 28.757878 seconds.
>     ============================================
> 
>     (Note: I cannot save the file in binary format because of
>     incompatibility
>     with the C++ program that generates the text file)
> 
>     Is this a known issue?
> 
>     Thanks
>     --
>     View this message in context:
>     
> http://www.nabble.com/Slowness-in-function-%27open%27-tf3960902.html#a11240251
>     Sent from the Octave - General mailing list archive at Nabble.com
>     <http://Nabble.com>.
> 
>     _______________________________________________
>     Help-octave mailing list
>     address@hidden <mailto:address@hidden>
>     https://www.cae.wisc.edu/mailman/listinfo/help-octave
> 
> 
> 
> Update your Octave version if possible. I think you should do the Octave
> developers a favor, by using a newer version of Octave 2.9.x or later.
> The comparison with Matlab, doesnt make much difference now that we are
> on a much improved version.

Muthu, If he is using 2.9.12 and MSVC build I don't think he can get any
newer without building from CVS himself :-)

> 
> I would also suggest you do a offline conversion program in Octave, that
> can load the text
> files and save it to binary format using, 
> save -binary cCaMeLCaPsMwItHuNgArIaNoTaTiOn
> 
> and then load binary files from Octave, in your 'main' script.
> 
> For me on the oldest Octave version I could find access to, 2.1.50
> a 1.4MB file loads as text in 0.399s; whereas the same file loaded as
> binary
> takes just 0.015370s.
> 
> I think thats x26 improvement for me.

Yeah, binary is better... The time difference with matlab is a factor of
5 and that's not greater, but is better than cygwin... Not sure why..

However,

>
>     (Note: I cannot save the file in binary format because of
>     incompatibility
>     with the C++ program that generates the text file)
>
>     Is this a known issue?


I'd be interested in knowing what this means, as I can't decipher it..

D.