bug#36431: Crash in marker.c:337

[Stefan Monnier]

>> We start by inserting the new bytes at the *beginning* of the gap, but
>> when we do the move_gap_both this moves those bytes to the *end* of the
>> gap (where decode_coding_gap expects them, apparently), so when we
>> decode we always have to move all the inserted bytes, right?
>
> Yes.  This is so we don't need to know up front how many bytes will be
> read from the file.

OK, so IIUC:
- we insert the new bytes at the beginning of the gap, in order to have
  room to grow if there are more bytes than expected, and also in case
  there are fewer bytes than expected (in which case we'd otherwise
  have to move the bytes we just read so they properly end at the end
  of the gap).
- decode_coding_gap wants the new input bytes to be at the end of the
  gap, so that we can put the decoded chars at the beginning of the gap
  and as one grows the other shrinks, so we don't need space for "IN +
  OUT" bytes but only for "OUT" bytes.  Is that right (I'm trying to
  find some comment or other evidence that this is the case, but
  haven't found it yet).

IOW, it should be possible to optimize the common case by reading the
new bytes into the end of the gap to avoid moving everything in the
common case (if the number of bytes read is different from originally
expected, we'll have to do extra work, but for the common case where we
know the file size upfront and it doesn't change while we read it, this
will save us some work).

But the effort is probably not worth the trouble: a memmove of a few
gigabytes costs relatively little compared to the cost of actually
decoding those same gigabytes.


        Stefan