On 6/21/07, Mark B. <address@hidden
> wrote:
Hi,
Is there a faster way to load a big text file composed only of integers? I
am having severe performance penalties by using the load function
These are the results of my tests:
First test:
============================================
Octave, version 2.9.12 (configured for "i686-pc-msdosmsvc")
octave.exe:1> tic; load('mybigfile'); toc;
Elapsed time is 98.314560 seconds.
Matlab 7.3.0.267 (R2006b)
>> tic; load('mybigfile');toc;
Elapsed time is 27.923677 seconds.
Octave, version 2.1.73 (i686-pc-cygwin)
octave:1> tic; load('mybigfile');toc
ans = 1117.5
============================================
Between calls to the 'load' function I flushed the cache of the location
where the file resides
Second test:
============================================
octave.exe:2> tic; load('mybigfile'); toc;
Elapsed time is 93.594170 seconds.
>> tic; load('mybigfile');toc;
Elapsed time is 28.757878 seconds.
============================================
(Note: I cannot save the file in binary format because of incompatibility
with the C++ program that generates the text file)
Is this a known issue?
Update your Octave version if possible. I think you should do the Octave developers a favor, by using a newer version of Octave
2.9.x or later. The comparison with Matlab, doesnt make much difference now that we are on a much improved version.
I would also suggest you do a offline conversion program in Octave, that can load the text
files and save it to binary format using,
save -binary cCaMeLCaPsMwItHuNgArIaNoTaTiOn
and then load binary files from Octave, in your 'main' script.
For me on the oldest Octave version I could find access to, 2.1.50
a 1.4MB file loads as text in 0.399s; whereas the same file loaded as binary
takes just 0.015370s.
I think thats x26 improvement for me.
What do you say?
-Muthu
PS: Im sorry if I posted this many times, just screwed it up