Re: [igraph] Memory-Efficient Graph Saving in Python

[Tamas Nepusz]

Hi,

> Trying to save a large graph object using iGraph 0.7 in python 2.7. When I
> try to save my graph with the following command:
> 
> myGraph.save('graph.pkl', format = 'pickle')
The pickle format is probably one of the most heavyweight formats because it
can be used to store arbitrary Python objects (well, only those that support
pickling) so it has to do a lot of extra bookkeeping behind the scenes. (For
instance, to ensure that cases when object A holds a reference to bject B which
then holds another reference to object A does not result in an infinite loop).
Therefore, I wouldn't use this format unless some of your graph/vertex/edge
attributes are so complex that they can be stored by pickling only.

(By the way, your graph is not duplicated in memory when it is saved -- the
reason why your memory usage explodes is because of this extra bookkeeping that
Python has to do).

Depending on what sort of vertex and edge attributes you have, you could try
storing your graph in NCOL format (if you have at most a "name" attribute for
vertices and a "weight" attribute for edges). If NCOL is not enough (because
you have more edge attributes to store), you could try GML or GraphML instead.
GraphML is going to take a lot of space (because it is XML-based), but you can
always compress it with gzip afterwards. The Python interface of igraph also
supports "graphmlz" as a format, which is basically gzipped GraphML,
transparently compressed and decompressed behind the scenes.

T.