Huge file adventure (+patch)

[Dmitry Antipov]

Recently I have got installed 16GB RAM in my laptop and, of course,
have tried to open 8GB ASCII text file to see what happens (now it's
just 1/2 of RAM, so why not?). After opening, I did some editing,
then wait for auto-save and ... got "Memory exhausted" message. Short
investigation quickly shows that I need ... 24GB of RAM to handle
such a file :-(. Here is why:

1) Fdo_auto_save calls Fwrite_region for the whole file, which
   issues e_write (.., start=1, end=8G, ...).

2) CODING_REQUIRE_ENCODING decides that it's time to do some
   encoding, and encode_coding_object allocates 8G destination
   buffer (from coding.c):

   8335    else if (EQ (dst_object, Qt))
   8336      {
   8337        ptrdiff_t dst_bytes = max (1, coding->src_chars);
   8338        coding->dst_object = Qnil;
   8339        coding->destination = xmalloc (dst_bytes); /* HERE */
   8340        coding->dst_bytes = dst_bytes;
   8341        coding->dst_multibyte = 0;
   8342      }

3) Finally encode_coding_object tries to create 8G Lisp string
   to hold the result (from coding.c):

   8351    if (EQ (dst_object, Qt))
   8352      {
   8353        if (BUFFERP (coding->dst_object))
   8354          coding->dst_object = Fbuffer_string ();
   8355        else
   8356          {
   8357            coding->dst_object
   8358              = make_unibyte_string ((char *) coding->destination,
   8359                                     coding->produced); /* HERE */
   8360            xfree (coding->destination);
   8361          }
   8362      }

So 8G for buffer text + 8G for encoding buffer + 8G for the result.
Since `coding->destination' is freed immediately after creating Lisp
string, I need 16G for some period of time but with short 24G peak.

And, of course, there is a patch to address an issues described above.
Comments are very welcome because I'm not hooked too much in coding
machinery (yet).

Dmitry

e_write_and_encode.patch
Description: Text Data