Re: GC Warning: Repeated allocation of very large block

[Damien Mattei]

what is strange is how the Boehm garbage collector can have noticed that
this repeated allocation of memory was a bug?, as it could have be
intentional and normal....
Damien

On Wed, Sep 21, 2022 at 11:42 AM Damien Mattei <damien.mattei@gmail.com>
wrote:

> i think the warning is not a false true because the program at reached in
> 10 days almost 3Gb :
>
>  PID UTIL.     PR  NI    VIRT    RES    SHR S  %CPU  %MEM    TEMPS+ COM.
>
>
>  332467 mattei    20   0 3262260   2,8g   3744 R 215,0  18,4  32277:54
> guile
>
> stopping it has released 3 Gb of memory, the problem can belong from the
> hash table which is declared (not defined) with one identifier at top-level
> but created in a function used many times in a loop so there was many
> hash-table creates and the haskeys-values where not deleted when leaving
> the function Quine-Mc-Cluskey that uses the hash-table because there is no
> documentation and that in SRFI 69 i forget to do that, a simple function in
> SRFI like this one could be useful:
>
> ;; delete all the elements of a hash table but not the hash table itself
> (will be done by garbage collector)
>
> (define (hash-table-clear! ht) (map (lambda (key) (hash-table-delete! ht
> key))
>   (hash-table-keys ht)))
>
> i have modified and ran the program again, and will see if the message
> from garbage collector happens again (in a week of computation...) i will
> also check the whole program that seems to expand symbolic expressions a
> lot when i have time.
>
> Regards,
> Damien
>
> On Mon, Sep 19, 2022 at 7:26 PM Damien Mattei <damien.mattei@gmail.com>
> wrote:
>
>> thank you, i think my algorithm is right but can be improved, i have to
>> work on it,activate tracing to see where is the big expression list and if
>> it is possible to correct that. There is a main loop in code and the
>> warning only appear when data is bigger so i do not think of any memory
>> leak (only have alittle doubt on an hastable which is global variable and
>> reused:
>> (when debug-mode (display-nl "Quine-Mc-Cluskey:"))
>>  {minterms-ht <- (make-hash-table)}  ;; need to be cleared at each run
>> (init-hash-table-with-set-and-value minterms-ht minterms #f)
>> this :
>> ;; the hash table for minterms, better to be a top-level definition,it's
>> nightmare otherwise...
>> (declare minterms-ht)
>> {minterms-ht <- (make-hash-table)} ;; Scheme+  syntax
>> expression create a new hash table with pointer minterms-ht
>> this is done at each loop in the for each Cn
>> and i should clear it before but there is no instruction to do that in
>> the SRFI 69:
>> https://srfi.schemers.org/srfi-69/srfi-69.html
>> ???
>> I will also check the still running memory used by program.
>> Damien
>>
>> On Mon, Sep 19, 2022 at 1:44 PM Olivier Dion <olivier.dion@polymtl.ca>
>> wrote:
>>
>>> On Mon, 19 Sep 2022, Damien Mattei <damien.mattei@gmail.com> wrote:
>>> > is this message appearing when a single scheme variable reach a given
>>> > size?
>>>
>>> This message is from the bdwgc and not from Guile itself.  From their
>>> documentation:
>>> --8<---------------cut here---------------start------------->8---
>>> The garbage collector generates warning messages of the form:
>>>
>>>
>>>     Repeated allocation of very large block ...
>>>     May lead to memory leak and poor performance
>>>
>>>
>>> when it needs to allocate a block at a location that it knows to be
>>> referenced
>>> by a false pointer. These false pointers can be either permanent (e.g.
>>> a static integer variable that never changes) or temporary. In the latter
>>> case, the warning is largely spurious, and the block will eventually
>>> be reclaimed normally. In the former case, the program will still run
>>> correctly, but the block will never be reclaimed. Unless the block is
>>> intended
>>> to be permanent, the warning indicates a memory leak.
>>>
>>>   1. Ignore these warnings while you are using GC_DEBUG. Some of the
>>> routines
>>>   mentioned below don't have debugging equivalents. (Alternatively,
>>> write the
>>>   missing routines and send them to me.)
>>>   2. Replace allocator calls that request large blocks with calls to
>>>   `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You
>>> may
>>>   want to set a breakpoint in `GC_default_warn_proc` to help you
>>> identify such
>>>   calls. Make sure that a pointer to somewhere near the beginning of the
>>>   resulting block is maintained in a (preferably volatile) variable as
>>> long
>>>   as the block is needed.
>>>   3. If the large blocks are allocated with realloc, we suggest instead
>>>   allocating them with something like the following. Note that the
>>> realloc
>>>   size increment should be fairly large (e.g. a factor of 3/2) for this
>>> to
>>>   exhibit reasonable performance. But we all know we should do that
>>> anyway.
>>>
>>>
>>>         void * big_realloc(void *p, size_t new_size) {
>>>             size_t old_size = GC_size(p);
>>>             void * result;
>>>             if (new_size <= 10000) return(GC_realloc(p, new_size));
>>>             if (new_size <= old_size) return(p);
>>>             result = GC_malloc_ignore_off_page(new_size);
>>>             if (result == 0) return(0);
>>>             memcpy(result,p,old_size);
>>>             GC_free(p);
>>>             return(result);
>>>         }
>>>
>>>
>>>   4. In the unlikely case that even relatively small object (<20 KB)
>>>   allocations are triggering these warnings, then your address space
>>> contains
>>>   lots of "bogus pointers", i.e. values that appear to be pointers but
>>> aren't.
>>>   Usually this can be solved by using `GC_malloc_atomic` or the routines
>>>   in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc.
>>>   Sometimes the problem can be solved with trivial changes of encoding
>>>   in certain values. It is possible, to identify the source of the bogus
>>>   pointers by building the collector with `-DPRINT_BLACK_LIST`, which
>>> will
>>>   cause it to print the "bogus pointers", along with their location.
>>>   5. If you get only a fixed number of these warnings, you are probably
>>> only
>>>   introducing a bounded leak by ignoring them. If the data structures
>>> being
>>>   allocated are intended to be permanent, then it is also safe to ignore
>>> them.
>>>   The warnings can be turned off by calling `GC_set_warn_proc` with
>>>   a procedure that ignores these warnings (e.g. by doing absolutely
>>>   nothing).
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> It's my understanding that either A) there's a leak or B) its a false
>>> positive.
>>>
>>> > i wanted to debug and trace because i know this algorithm expand a lot
>>> > expressions (perheaps too much) sometimes (but it is an NP-problem and
>>> > exponential, so perheaps stop the program is the only solution and run
>>> it
>>> > on more little data)
>>> > i do not think there could be a memory leak in a recursive scheme
>>> > program,
>>>
>>> Gotta be careful with a conservative GC I guess.  Your memory will get
>>> reclaim at some point, but it's not guaranteed when since any value on
>>> the C stack -- and global variables -- could reference your allocation
>>> by accident.  So if by accident there's a static constant value in C
>>> that has the value of an allocation by the GC, it will never reclaim it.
>>> Since it's the Scheme stack and I suppose it's tail call optimized, that
>>> should not impact the GC.
>>>
>>> The most important thing would be to check the memory usage of the
>>> program with a tool like `htop'.  If there's leak, you will see that the
>>> memory usage percentage keep increasing.
>>>
>>> --
>>> Olivier Dion
>>> oldiob.dev
>>>
>>