Re: [igraph] the old "cannot reserve space for vector" problem

[Pascal Jürgens]

Tamas,

thanks a lot for the clarifications and the numbers. I think I'll pass on 
calculating the entire matrix for now. Storing it twice means that swapping is 
guaranteed, which should prevent a serious slowdown.

However, I think that in case anyone attempts it, it might be interesting to 
look at the use of SSD disks for matrix storage. They might be just fast enough 
to switch the bottleneck back to the cpu.

Yours,
Pascal Juergens

On Jun 8, 2010, at 11:22 , Tamas Nepusz wrote:

> What I can tell you is that I successfully managed to calculate the 
> *diameter* of an undirected network with 2.1 million vertices and 7.5 million 
> edges -- this took about 11 days on a single core of an Intel Xeon X3360 
> running at 2.83 GHz. So, time complexity should not be a problem for your 
> network. I'm still worried about the storage requirements, because when using 
> the Python interface, the result matrix is temporarily stored *twice*, once 
> as a regular igraph_matrix_t and once as a Python list of lists which is 
> returned to Python. Of course the igraph_matrix_t is destroyed as soon as the 
> Python representation is ready, but still. I would probably try to be on the 
> safe side and build up the shortest path matrix line by line and store the 
> calculated lines on disk, so even if Python runs out of memory for some 
> reason, nothing is lost because all the earlier calculations are still saved.