Re: Slowness in function 'open'

[Robert A. Macy]

one more time...
octave 2.1.50 on WinXP 600MHz with 500MB RAM

to put the suggested header on there:
2.1.50
>> x=randn(1000,2000);
>> save -ascii "mybigfile5.txt" x;
...mybigfile5 is around 35.5MB
>> tic;load -force "mybigfile5.txt";toc*30/35.5
ans = 33.958
Wow, that really did speed up octave to give it the size of
the data set.

exit 2.1.50 and start 2.9.12...
octave-2.9.12.exe:1> tic;load("mybigfile5.txt");toc*30/35.5
same series of warnings, got before.  
error: load: failed to load matrix constant
error: load: trouble reading ascii file 'mybigfile5.txt'
error: load: reading file mybigfile5.txt

ok manually removed the first # line containing info, so
the first line is 
# name: x
and tried again, same list of error messages ?!

Robert

On Thu, 21 Jun 2007 20:37:19 -0400
 "John W. Eaton" <address@hidden> wrote:
> On 21-Jun-2007, Mark B. wrote:
> 
> | Does anybody else know how to improve these times?
> 
> Maybe.
> 
> As others have pointed out, if you need a quick fix for
> your problem,
> you could use a filter of some kind to convert your data
> to binary.
> 
> Another option that I don't think I've seen mentioned yet
> is to filter
> your data so that it is prefixed with a header that looks
> like this:
> 
>   # name: x
>   # type: matrix
>   # rows: 100000
>   # columns: 1
> 
> (with the name and dimensions of your matrix substituted,
> of course).
> Then I think Octave's load function will be able to read
> the file
> faster.
> 
> Or, if you are interested in improving this bit of poor
> performance so
> that everyone benefits, the functions you are looking for
> are in
> src/ls-mat-ascii.cc.  The function that reads rows and
> columns of
> numbers may be trying too hard to ensure that all rows
> have the same
> number of columns and to also counting the number of
> non-comment lines
> before it actually reads the data (so it can avoid
> resizing the output
> matrix, possibly multiple times).
> 
> jwe