Re: Slowness in function 'open'

[Mark B.]

John W. Eaton wrote:
> 
> As others have pointed out, if you need a quick fix for your problem,
> you could use a filter of some kind to convert your data to binary.
> 
> Another option that I don't think I've seen mentioned yet is to filter
> your data so that it is prefixed with a header that looks like this:
> 
>   # name: x
>   # type: matrix
>   # rows: 100000
>   # columns: 1
> 
> (with the name and dimensions of your matrix substituted, of course).
> Then I think Octave's load function will be able to read the file
> faster.
> 

Thanks for the tip. Working with headers improved the performance a lot:

%%%%%%%%%%%%
%cache flushed before each of these commands

octave.exe:1> tic; load('mybigfile_hdr.txt'); toc
Elapsed time is 35.153114 seconds.

octave.exe:2> tic; save -binary 'mybigfile_hdr.bin' x; toc
Elapsed time is 44.581726 seconds.

octave.exe:3> tic; load('mybigfile_hdr.bin'); toc
Elapsed time is 4.553993 seconds.
%%%%%%%%%%%%

Interestingly, there was no improvement in performance when the header was
just:

# rows: <numrows>
# columns: <numcols>

Why is that?

I have another question about the headers. Is it possible to specify the
header information in a different file? For example:

load('thefile', '-header', 'thefile_header');

In this way the compatibility with Matlab (and other programs) with be
better ('thefile' will contain only the data)

If this is not possible at this time, please include it in your user's
wishlist. I guess I will have to wait until Octave 5 or something, but hey,
I had to ask anyway ;-)

On the other hand, saving the file in binary format is too time consuming.
Moreover, the size of the file is 50% larger (~45 mb) than the text file.
Loading a binary file the  next time is much faster of course, but given the
time needed to save the file is not worthwhile (in my specific case) unless
the file is going to be opened a second time.


John W. Eaton wrote:
> 
> Or, if you are interested in improving this bit of poor performance so
> that everyone benefits, the functions you are looking for are in
> src/ls-mat-ascii.cc.  
> 

Not in the short term, but probably later. At any rate, thanks for letting
me know the exact file that contains the code. It will save me  time
compared to looking for the file in the entire Octave's source code. You
should always do this if you want to tell somebody to contribute to a
specific feature if that person is willing to do so.

-- Mark

-- 
View this message in context: 
http://www.nabble.com/Slowness-in-function-%27open%27-tf3960902.html#a11259158
Sent from the Octave - General mailing list archive at Nabble.com.