Re: vectorization

[Dave Cottingham 2]

Nidjara wrote
> You guys are great!
> 
> I'm working with very large matrices (2-7 millions of rows), so this
> sub2ind and in general linear indexing couldn't work (the index numbers
> become too high).
> 
> Helios's solution works like charm, and it fills matrix so much faster
> than with loop (I'm using sparse matrix:). So thanks a lot.
> 
> Related to this. What I actually do in my program is: read large amount of
> data from a structured text file (graph .dot format), store it into sparse
> matrix G, and then do some linear algebra with G .
> 
> The code to load the data is:
> 
>       [L1, L2, L3]= textread(fileinput,'%d -> %d
> [w="%f"];','delimiter','\n}','headerlines',line-1);
>       G=sparse(L1+1, L2+1, L3, dim, dim);
> 
> Now the bottleneck is line with textread, which takes 99% of the code
> execution. Any ideas of a faster way to load the data? (I have data in
> .dot, graphml and .csv format)

I can't give a specific answer, as I've never used Octave for sparse
matrices, but I am painfully familiar with input being a bottleneck. And my
usual solution is this:

1. Figure out the format of an Octave save file for sparse matrices. This is
probably documented somewhere, but a quick way is to create a sparse matrix
and "save" it (don't use -ascii or -binary flags) and then look at the
resulting file and puzzle it out.

2. Write an external program to reformat your data into an octave save file.

3. "load" it.

"load" seems to be the quickest way to input data into Octave. About
-binary: if you can find the documentation for this format, it will probably
load faster. But if you don't find the documentation, it's harder to reverse
engineer the -binary format -- that's the only reason I suggested avoiding
it.



--
View this message in context: 
http://octave.1599824.n4.nabble.com/vectorization-tp4668064p4668080.html
Sent from the Octave - General mailing list archive at Nabble.com.