Re: [MIT-Scheme-devel] A tale of four builds.

[Taylor R Campbell]

I'm about to commit a bunch of changes to the build system (and a few
other changes interspersed), largely in the top-level makefile rules.
The main change is that we first build a compiler toolchain from the
sources in the tree, and store it in tools/, and then we build
everything using that compiler toolchain.

`make all-native' (or just `make', for the native-code case) will
automatically build the tools.  The first time around it will clean
the tree[*] and build the tools; the next time around it will use the
toolchain it build the first time.  You can blow away the tools with
`rm -rf tools'[**].

Here are some more detailed notes.  Please let me know if you have any
questions!


* Goals

- Improve parallelization.  The new makefile rules have fine-grained
dependencies, so `make -jN' should work safely.  I already did this
for the native build, but the new makefile rules also work for SVM and
almost work for LIARC.  The old SVM and LIARC rules are still used by
default, but there are only a few issues to fix before that can be
changed.  Eventually everything should use the new rules.

- Enable cross-compilation.  The first step is now to build a compiler
toolchain, stored in src/tools/, with which to build the whole system
afterward.  This can be a native compiler, in the sense of that it
runs on the same architecture as the code it generates, or it can be a
cross-compiler, if you configure with `--enable-cross-compiling'.

- Reduce incoherent circularity in the build.  The parts that need to
run in a host Scheme are clearly laid out, and the whole system is
consistently compiled with the same compiler toolchain.


* To do

- [easy] Install .pkd files referring to the installed systems.
- [medium] Teach CREF, SF, and LIAR to do portable FASDUMP and FASLOAD.
- [easy] Eliminate FINISH-CROSS-COMPILATION once the above works.
- [easy] Remove old SVM and LIARC rules once the above works.
- [medium] Move runtime/syntax-*.scm into a separate syntax/ component.
- [hard] Teach MIT Scheme about a sensible approach to macro phases.
- [hard] Go through Edwin, LIAR, &c., and fix all their macros.
- [?] Rewrite microcode makefile, split modules into separate
directories, split the host/target C compiler for liarc-{cc,ld},
replace the double-indirected Makefile.in.in by a Makefile.in, fix
config.status and other GNU conventions...

The toolchain is built against runtime/runtime.pkg as it appears in
the source tree, which may be very different from the host's runtime.
Instead, the host's runtime/runtime-unx.pkd should have been installed
into the MIT Scheme library directory, so that the toolchain (and
anything else) can be built against that.

Of course, CREF is a botch that desperately needs to be replaced by a
more principled system for organizing programs, but the general
concept of installing a file like runtime/runtime-unx.pkd is sound, so
it should figure into our build system.

Cross-compiling to SVM works OKish because the non-compiled-code fasl
files are compatible.  But all other cross-compilation is broken: the
.bin files won't load in the cold load to FINISH-CROSS-COMPILATION.
Instead, we need to eliminate the FINISH-CROSS-COMPILATION step and
teach the cross-compiler to just use a portable FASDUMP.

The LIARC build almost avoids all of the issues, but when compiling
IMAIL -- which is the only part of the system that it doesn't work to
cross-compile, because it depends on hairy macros from Edwin -- CREF
tries to fasload some .pkd files which it chokes on.  With a few
tweaks, I can persuade it to get these from the built-in objects --
but those tweaks are bad.

Finally, we need a principled way to say `When you compile IMAIL, you
need to use macros defined in Edwin.' that doesn't involve loading
Edwin itself into the compiler program, like Scheme48 and Racket have.
I started working on one of these years ago but never made progress,
partially because I didn't understand how to think about build systems
at the time.

Unfortunately, I don't foresee when I might have time to do any of
these tasks; I've already spent more time than I have...


* Developing the tools

If you want to iterate development with the tools, you can do this:

   make -f Makefile.tools               # Build the tools, and then
   make -f Makefile.tools save          # save the objects, so that... (*)
   make clean           # Clean the tree for a fresh build.
   make all-native      # Build, and find you broke the compiler or macros.
   make -f Makefile.tools clean         # (*) ...you can clean up the tree
   make -f Makefile.tools restore       # and get them back in order to
   <fix the bug in the toolchain>
   make -f Makefile.tools               # incrementally rebuild the tools.

You can also save and restore the the objects from the normal makefile
using `make save' and `make restore', so you can alternate back and
forth with `make -f Makefile.tools save && make clean && make restore'
and `make save && make -f Makefile.tools clean && make -f
Makefile.tools restore'.

And, of course, you can use `-jN' with `make -f Makefile.tools' too.


[*] Cleaning the tree to build the tools should not be necessary, but
currently is so because we don't support separate source and object
directories.

[**] I haven't updated the relevant clean target(s) (maintainer-clean,
I guess) to do this, sorry.