Re: Slowness in function 'open'

[Jordi Gutierrez Hermoso]

About this load problem, which seems it's not the first time this has
come up... I have a few thoughts. I ran a quick test on a matrix I
created with Octave's rand(1000). The ASCII file ends up being 19
megs, and I named it "A.m".

On 21/06/07, John W. Eaton <address@hidden> wrote:
> Another option that I don't think I've seen mentioned yet is to filter
> your data so that it is prefixed with a header that looks like this:
>
>   # name: x
>   # type: matrix
>   # rows: 100000
>   # columns: 1

Unlikely. I still see a bit of bad performance with a matrix file
created by Octave, that does have this header information.

> Or, if you are interested in improving this bit of poor performance so
> that everyone benefits, the functions you are looking for are in
> src/ls-mat-ascii.cc.  The function that reads rows and columns of
> numbers may be trying too hard to ensure that all rows have the same
> number of columns and to also counting the number of non-comment lines
> before it actually reads the data (so it can avoid resizing the output
> matrix, possibly multiple times).

I looked at these functions. They seem indeed to be doing too much
work. Observe:

    octave2.9:1> tic, load A.m; toc
    Elapsed time is 7.975471 seconds.
    octave2.9:2> tic, system("foo"); toc #My own matrix read function,
                                         #source in url below [1]
    Elapsed time is 1.576744 seconds.

I can't quite understand what they're doing at first glance. But if
the problem is simply to read a matrix ASCII file, looks like there
might be a better way to do it.

The relevant bits of my sample code are in linalg.cpp, in the function
linalg::operator>>(...) for matrices. This is something for which I used
the GSL, so you'll need the GSL if you want to build my sample
code. Sorry for being lazy and not doing a standalone example without
the GSL.

Octave's algorithm does seem a bit strange, though. Why is it doing
two passes in the case that the file has no header? One pass should
suffice. You can read all the data into an std::list, which is what I
do, and there are a few simple checks to make sure that the formatting
of the ASCII matrix is correct. My code always ignores the Octave
header and all comment lines. This could be problematic for N-d arrays,
but at least for plain matrices works fine.

In fact, my code could probably be improved too, since it copies
matrices around unnecessarily. Avoiding that copying was stylistically
prohibitive for me.

Stylistically, I would rewrite Octave's code with std::stringstreams,
std::strings, and standard C++ library classes and algorithms, but
that's just because I have a deep disdain for all of C that isn't
needed for C++. I try to always avoid macros, for instance. To
paraphrase Stroustrup, macros are a deficiency in the code, the coder,
or the coding language. ;-)

HTH,
- Jordi G. H.

[1] source at
    http://platinum.linux.pl/~jordi/octave/matrix-read-example.tar.gz