[bug-recutils] Optimizing memory consumption

[Jose E. Marchesi]

Hi all.

I recently got a very interesting report: doing a recsel in a 200Mb
recfile takes more than 4Gb of RAM, with the result of the process being
killed by the kernel:

[60738.722562] Out of memory: Kill process 20130 (recsel) score 780 or
sacrifice child
[60738.722570] Killed process 20130 (recsel) total-vm:4841960kB,
anon-rss:3509936kB, file-rss:628kB

I used Valgrind's massif tool in order to profile the usage of the heap
by recsel, and got the following results running recsel on a file
containing 20000 records (3.8 Mb):

--------------------------------------------------------------------------------
Command:            recsel 20000-records.in
Massif arguments:   (none)
ms_print arguments: massif.out.3560
--------------------------------------------------------------------------------


    MB
120.8^                                                                    #   
     |                                                    ::::::::::::::::#   
     |                                                  @::               #   
     |                                                  @::               #:  
     |                                                 :@::               #:  
     |                                                 :@::               #:  
     |                                                ::@::               #:  
     |                                                ::@::               #:  
     |                                               :::@::               #:  
     |                                              ::::@::               #:: 
     |                                             @::::@::               #:: 
     |                                         @@@:@::::@::               #:: 
     |                                    @@@@@@@@ @::::@::               #:: 
     |                               @@@@@@@@@@@@@ @::::@::               #:: 
     |                           @:@@@ @@@@@@@@@@@ @::::@::               #:: 
     |                       @@@@@:@@@ @@@@@@@@@@@ @::::@::               #:::
     |                  @@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@::               #:::
     |              @@@@@@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@::               #:::
     |         @@@@@@@@@@@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@::               #:::
     |    @:@@:@@@ @@@@@@@@@@@@@@@:@@@ @@@@@@@@@@@ @::::@::               #:::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   1.901

  0              0                0                0             0            0
  1     23,918,655        1,065,632          633,598       432,034            0
 [...]
 59  1,460,090,910      119,858,072       72,510,893    47,347,179            0
 60  1,476,532,025      125,330,256       75,860,862    49,469,394            0
*61  1,942,067,951      126,676,744       76,685,098    49,991,646            0
 62  1,958,509,017      108,537,280       65,788,890    42,748,390            0
 63  1,974,950,089       90,398,536       54,762,626    35,635,910            0
 64  1,991,391,167       72,259,752       43,658,413    28,601,339            0
 65  2,007,832,250       53,735,576       32,429,103    21,306,473            0
 66  2,024,273,329       35,380,312       21,481,457    13,898,855            0
 67  2,040,714,408       17,025,096       10,401,197     6,623,899            0


There are no memory leaks.  As you can see, the peak indicates a total
of memory used of around 121Mb in the heap.  This is simply too much for
a 4 megabytes recfile, so we must come with some optimizations in terms
of memory consuption.

For example, an obvious optimization is to use a pool of strings to
store the field names.  The pool would be shared by all the records
pertaining to some record set.

More ideas?

-- 
Jose E. Marchesi         http://www.jemarch.net
GNU Project              http://www.gnu.org