Re: MPS: Loaded pdump

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Ok, I think I'll try to get rid of the pdump root. Maybe I'll regret it,
>> and maybe I'll fail, but I feel the root is not the right thing to have.
>> (I'll do in in a branch from the branch in the fork here.)
>>
>> Step 1:
>>
>> Make the loaded dump traversable by dumping igc_headers.
>>
>> How easy or not that is is hard to say. On one hand, there seems to be
>> dump_object_begin, which looks promising, but I already know one case
>> (hash tables), where extending that function to write igc_headers might
>> not be sufficient. And where is one exception there are more. And flags
>> like pack_objects could interfere, and so on.
>>
>> Doenside: igc_header is then even more set in stone. Removing the header
>> for conses becomes even more work. One would need a new section in the
>> dump, just for conses.
>>
>> Step 1.5:
>>
>> When (1) is done, one could make the dump an exact root and use
>> dflt_scan on it. (That's better than an ambig root, but I don't want the
>> root because the world is stopped when roots are scanned.)
>>
>> Step 2:
>>
>> Walk through the dump and make copies of dumped objects in MPS memory.
>> Record a mapping from dumped -> MPS object. Then walk through the copied
>> objects and replace references to dumped objects.
>>
>> This is a bit hand-wavy. At some point when a dump is loaded, things in
>> Emacs are set up to refer to objects in the dump. We will have to do our
>> copy before that and somehow make sure the copy is used instead of the
>> dump.
>
> An update on what's going on:
>
> I now have a branch in my fork that builds with and without MPS, where
> the MPS dump contains igc headers and the non-MPS dump doesn't.
>
> Next step will be do ensure that the hot section of the dump is indeed
> traversable as I want, and fix what's wtong. So we are at step 1.25 or
> so.
>
> Things are a bit slower right now for procrastination reasons. The
> pdumper is boring as hell :-/. Did I mention that code generation would
> be a nice thing? Anyway.

I've now pushed someting like step 1.45.

Random notes:

- The dump for MPS now contains the start offsets of igc objects. The
  existing object_starts relocs cannot be used because they are for Lisp
  objects only.

- Each igc object in the dump has an igc_header. The function
  pdumper_visit_object_starts can be used to traverse them in a loaded
  dump. I chose this interface because it cannot be made sure that igc
  objects occupy a continguous region in the dump, at least not with
  surgery on the pdumper.

- Obarrays are probably not yet handled right. I couldn't bring me to do
  this yet. As you know my fork doesn't have obarrays, but uses CL
  packages which use hash tables. Any takers?

- There are a number of igc_obj_types of the form IGC_OBJ_DUMPED_xy. I'm
  not sure what do with these.

- I saw that igc_header::pvec_type is used for something, and shoujld
  probably tell that was meant as an debugging aid (for the case of
  hitting IGC_OBJ_FWDs, so that more easily see what type was forwarded.
  This could also be seen fromt he vectorlike header. I think at some
  we should remove pvec_type in favor of more hash bits. Whatever, not
  so important.

- I commented out from .gitignore the .patch files. This is because
  ignoring .patch files disables some very handy Magit functionality wrt
  patch handling.

Otherwise this is not yet used. I checked with an MPS and non-MPS
build (with checking=all).

Taking the next step will be difficult. Copying object from the dump to
MPS and fixing references from one graph to the other requires writing a
gazillion functions to do that. At least at the moment that's something
that exhausts me just be thinking of it.

Whatever. Happy photosynthesizing on a sunny day!