Reducing iconv-induced memory usage

[Ludovic Courtès]

Hello!

As Andy noted in the past, iconv conversion descriptors associated with
ports take up a lot of malloc’d memory, that only gets freed when
finalizers are run.  On GNU/Linux, a UTF-8 → UTF-8 C.D., which does
nothing, mallocs 180 KiB (!), according to the program attached.  So the
problem is acute.

So I think we should special-case UTF-8 I/O to not use iconv at all.

For output, it’s easy since we already do the conversion to UTF-8 in
‘display_string’.  For input, it’s a bit more work because input byte
streams have to be checked for invalid sequences.

I’m working on a patch but I’d like to get initial feedback and also
about whether it should wait until after 2.0.1 or not.

Thanks,
Ludo’.

#include <iconv.h>
#include <malloc.h>

static size_t total;

static void * (*prev_hook) (size_t, const void *);

static void *
m (size_t s, const void *c)
{
  __malloc_hook = prev_hook;
  printf ("alloc %zi\n", s);
  void *r = malloc (s);
  total += s;
  __malloc_hook = &m;
  return r;
}

static void
my_init_hook (void)
{
  prev_hook = __malloc_hook;
  __malloc_hook = &m;
}

void (*__malloc_initialize_hook) (void) = my_init_hook;

int
main (int argc, char *argv[])
{
  total = 0;
  iconv_open ("UTF-8", "UTF-8");
  printf ("allocated %zi B\n", total);
  return 0;
}