Re: [bug-recutils] Optimizing memory consumption

[Michał Masłowski]

Hi.

> I used Valgrind's massif tool in order to profile the usage of the heap
> by recsel, and got the following results running recsel on a file
> containing 20000 records (3.8 Mb):

Not used this tool before, will be useful.

> There are no memory leaks.  As you can see, the peak indicates a total
> of memory used of around 121Mb in the heap.  This is simply too much for
> a 4 megabytes recfile, so we must come with some optimizations in terms
> of memory consuption.

There are some leaks with othoer options (at least with some indexing
changes), planning to find and fix more after the index support is
complete.  I don't expect this to have significant effects.

> For example, an obvious optimization is to use a pool of strings to
> store the field names.  The pool would be shared by all the records
> pertaining to some record set.

Field sources might be more commonly duplicated.  Similarly for values
of some fields.

Seen many places in librec doing deep copies of objects, this would
probably need reference counting or a more complex solution to use
references to the same objects instead.

pgpPdzZ2TLeJn.pgp
Description: PGP signature