[Gcl-devel] Re: confusion over patch suggestion

[Camm Maguire]

Greetings!

First of all, Tim, none of this pertains to 'flaws' in axiom IMHO, but
rather current weaknesses in GCL, which can nevertheless thankfully be
overcome without too much work.

There are two issues:

1) To use an externally installed GCL:
        a) One has no access to unixport/makefile for the purposes of
           patching the link command line generating raw_gcl and
           adding the EXTRAS of sockio-c.o and cfuns-c.o.  Instead,
           one can make use of GCL's compiler::link function to do the
           same within lisp from an externally installed GCL.  This
           happens at the 'lisp' image creation stage, and the syntax
           is:
                (compiler::link 
                        nil 
                        ;;;
                        ;;; an empty list of compiled lisp module pathnames
                        ;;;
                        "${OUT}/lisp" 
                        ;;;
                        ;;; the output image pathname
                        ;;;
                        "(setq compiler::*default-system-p* t)" 
                        ;;;
                        ;;; init code to run after the lisp modules' init;
                        ;;; *default-system-p* is explained below
                        ;;;
                        "${OBJ}/${SYS}/lib/cfuns-c.o 
${OBJ}/${SYS}/lib/sockio-c.o ${OBJ}/${SYS}/lib/libspad.a")
                        ;;;
                        ;;; extra objects and libs to link in via ld
                        ;;;
  
        b) One wants to be able to save a local image (lisp, bootsys,
        depsys, interpsys) and still be able to access pertinent files 
        in the GCL tree, e.g. collectfn (now renamed to
        gcl_collectfn), and cmpinclude.h if compiling with :system-p
        t, as is required at present on 5 GCL architectures.  GCL now
        keeps a persistent *system-directory* variable to enable
        finding these files.  Hence these two patches, with comments
        interspersed:

--- src/interp/util.lisp.pamphlet       28 Aug 2003 14:23:40 -0000      1.1
+++ src/interp/util.lisp.pamphlet       13 Sep 2003 18:59:49 -0000
@@ -68,10 +68,14 @@
   ;; perform system initializations for building a starter system
   (init-memory-config)
   #+:AKCL

        ;;; when building GCL from source, the final image has
        ;;; *system-directory*, *lib-directory*, *load-path* and a 
        ;;; few others set to find files in the built source tree.
        ;;; When GCL is installed, *system-directory* refers to the
        ;;; newly installed location.  Currently we still rely on the
        ;;; shell script wrapper to set *lib-directory* and
        ;;; *load-path*, but this should be cleaned up at some point 
        ;;; soon.  As axiom calls the images without the wrapper
        ;;; (which invokes the image with command line args setting 
        ;;; these variables), this patch uses *system-directory* to
        ;;; find the gcl_collectfn.lsp source file, copy it into the
        ;;; image's destination directory, and compiles it there
        ;;; so it will be found in the image's default *load-path*.
        ;;; This could be improved when we get a persistent
        ;;; *lib-directory*, in which case the copy and recompile
        ;;; can be avoided.
        
-  (let ((collectfn (concatenate 'string (string lsp) "/cmpnew/collectfn")))
-   (unless (probe-file (concatenate 'string collectfn ".o"))
-     (compile-file collectfn))
-   (load collectfn)
+  (let ((collectfn (concatenate 'string si::*system-directory* 
"../cmpnew/gcl_collectfn.lsp"))
+       (collectfn1 (concatenate 'string obj "/" sys "/interp/collectfn")))
+   (with-open-file (st collectfn :direction :input)
+      (with-open-file (st1 (concatenate 'string collectfn1 ".lsp") :direction 
:output)
+       (si::copy-stream st st1)))
+   (unless (probe-file (concatenate 'string collectfn1 ".o"))
+     (compile-file collectfn1))
+   (load collectfn1)
    (compiler::emit-fn t)
    (mapcar
      #'load


--- src/interp/Makefile.pamphlet        5 Sep 2003 22:14:19 -0000       1.3
+++ src/interp/Makefile.pamphlet        13 Sep 2003 18:59:49 -0000
@@ -611,8 +613,10 @@
        @ echo '#+:akcl (setq compiler::*suppress-compiler-notes* t)' >> 
${OUT}/makeint.lisp
        @ echo '#+:akcl (si::gbc-time 0)' >> ${OUT}/makeint.lisp
        @ echo '#+:akcl (setq si::*system-directory* "${SPAD}/bin/")' >> 
${OUT}/makeint.lisp
-       @ (cd ${OBJ}/${SYS}/bin ; \
-         echo '(progn (gbc t) (load "${OUT}/makeint.lisp") (gbc t) 
(user::spad-save "${SAVESYS}"))' | ${LISPSYS} )
+#      @ (cd ${OBJ}/${SYS}/bin ; \
+#        echo '(progn (gbc t) (load "${OUT}/makeint.lisp") (gbc t) 
(user::spad-save "${SAVESYS}"))' | ${LISPSYS} )


+       @ (cd ${OBJ}/${SYS}/bin ; \
+         echo '(progn (gbc t) 

                ;;; Something in makeint.lisp is resetting the
                ;;; *system-directory* variable, and this patch just restores
                ;;; it in the final image.

                        (setq x si::*system-directory*)
                        (load "${OUT}/makeint.lisp") 
                        (setq si::*system-directory* x)
                        (gbc t) (user::spad-save "${SAVESYS}"))' | ${LISPSYS} )

        @ echo 6 ${SAVESYS} created
        @ cp ${SAVESYS} ${AXIOMSYS}
        @ echo 6a ${AXIOMSYS} created


2) Currently on ia64, mips(el), alpha, and hppa, GCL cannot save
   system images with loaded compiled lisp object files (.o files).
   On these systems, GCL must use compiler::link to make images
   containing compiled lisp object files.  One can also use this
   method on other systems including i386, but it is not necessary.
   Eventually we hope the necessity for this step will go away.  To
   achieve this will require modifications to the BFD library for
   these platforms, either upstream, or via a GCL patch.

   Luckily the alternate image creation method is almost transparent
   to the user.  In fact, perhaps with your help, it can be made
   completely transparent, and these separate arch-dependent image
   building steps can be avoided.  The idea is that one has a normal
   GCL session, compiling files, processing other input, etc., and
   instead of executing save-system to dump the image, passes a list
   of the compiled lisp object modules to compiler::link together with
   whatever other interpreted initialization code was run in the
   session.  compiler::link will then build a 'raw' image by linking
   the lisp objects to the gcl objects with ld, and then feed this
   image as input the interpreted code already run in the current
   session.  Any invocations of 'load' in this second run
   automatically bypasses the actual load step, and instead merely
   initializes the compiled lisp object which is already linked into
   the running executable.  At the end of this input, the image is
   saved with save-system.

   We made an improvement in this procedure for axiom, and that was
   the si::*collect-binary-modules* setup.  Basically, one sets this
   variable to t at the beginning of a session, and the variable
   si::*binary-modules* will contain a list of all loaded binary
   compiled lisp modules at the session's end, ready to pass as is to
   compiler::link.  If in addition I could figure out a reliable way
   of saving the input stream for rerunning later, then I could simply
   redirect save-system on these troublesome architectures to a call
   to compiler::link and all would be transparent.  

   For compiled lisp object modules to be ld linkable, they must be
   compiled with the :system-p flag set to t.  This can most easily be
   achieved by setting compiler::*default-system-p* to t.

   As it is now, the following is needed on ia64, mips(el), alpha, and 
   hppa, with comments interspersed:

--- src/boot/Makefile.pamphlet  28 Aug 2003 12:15:28 -0000      1.1
+++ src/boot/Makefile.pamphlet  13 Sep 2003 18:59:46 -0000

###
###                       object list       output name   post init  code
###
+CMD0=   (compiler::link (quote (${OBJS1})) "${SAVESYS}" "(setq 
compiler::*default-system-p* t)")


-CMD0=  (progn (mapcar (function (lambda (x) (load  x))) (quote (${OBJS1}))) 
(system::save-system "${SAVESYS}"))
+#CMD0= (progn (mapcar (function (lambda (x) (load  x))) (quote (${OBJS1}))) 
(system::save-system "${SAVESYS}"))

        

--- src/interp/Makefile.pamphlet        5 Sep 2003 22:14:19 -0000       1.3
+++ src/interp/Makefile.pamphlet        13 Sep 2003 18:59:49 -0000
@@ -558,8 +558,10 @@
        @ echo '(load "${OUT}/c-util")' >> ${OUT}/makedep.lisp
        @ echo '(unless (probe-file "${OUT}/g-util.${O}") (compile-file 
"${OUT}/g-util.${LISP}" :output-file "${OUT}/g-util.${O}"))' >> 
${OUT}/makedep.lisp
        @ echo '(load "${OUT}/g-util")' >> ${OUT}/makedep.lisp
-       @ (cd ${MNT}/${SYS}/bin ; \
-          echo '(progn (load "${OUT}/makedep.lisp") (spad-save "${DEPSYS}"))' 
| ${LISPSYS})
+#      @ (cd ${MNT}/${SYS}/bin ; \
+#         echo '(progn (load "${OUT}/makedep.lisp") (spad-save "${DEPSYS}"))' 
| ${LISPSYS})
+       @ ( cd ${OBJ}/${SYS}/bin && \


                ;;; I've spread out the code in this patch and
                ;;; interspersed comments


+               echo '(setq si::*collect-binary-modules* t) 
                        ;;
                        ;; keep track or all loaded .o files
                        ;;
                      (load "${OUT}/makedep.lisp")
                        ;;
                        ;; run the code generating the depsys image
                        ;;
                      (compiler::link 
                        ;;
                        ;; make a new image with compiler::link
                        ;;
                        (remove-duplicates
                           si::*binary-modules* :test (quote equal))
                                ;;
                                ;; remove duplicates from the list of .o files
                                ;;
                        "$(DEPSYS)" 
                        ;;
                        ;; output image name
                        ;;
                        "(setq si::*collect-binary-modules* t)
                                ;;
                                ;; feed the following to the image
                                ;; linked with ld -- first make sure
                                ;; we don't actually load any more .o files
                                ;;
                        (load \"$(OUT)/makedep.lisp\")
                                ;;
                                ;; then run the image generation code
                                ;; again, on this pass, all attempts
                                ;; to load .o files should instead
                                ;; merely initialize files already
                                ;; linked in
                                ;;
                        (gbc t)
                                ;;
                                ;; collect garbage
                                ;;
                        (when si::*binary-modules* 
                                (error si::*binary-modules*))
                                ;;
                                ;; Bail if we loaded a module on this run
                                ;;
                        (setq si::collect-binary-modules* nil
                              si::*binary-modules* nil)
                                ;;
                                ;; stop collecting loaded modules
                                ;;
                        (gbc t)(setq compiler::*default-system-p* t)"
                                ;;
                                ;; set the system-p flag on all
                                ;; future compiles with this image
                                ;;
                         "" 
                                ;;
                                ;; no extra libs needed
                                ;;
                        nil
                                ;;
                                ;; don't run the init code
                                ;; for the linked in modules
                                ;; explicitly via the function
                                ;; si::user-init, but rather 
                                ;; tranparently redirect load
                                ;; requests to initialization
                                ;; requests in order as they
                                ;; occur when processing the 
                                ;; standard input
                                ;;
                        )' | ${LISPSYS})
        @ echo 4 ${DEPSYS} created
 
 @
@@ -611,8 +613,10 @@
        @ echo '#+:akcl (setq compiler::*suppress-compiler-notes* t)' >> 
${OUT}/makeint.lisp
        @ echo '#+:akcl (si::gbc-time 0)' >> ${OUT}/makeint.lisp
        @ echo '#+:akcl (setq si::*system-directory* "${SPAD}/bin/")' >> 
${OUT}/makeint.lisp
-       @ (cd ${OBJ}/${SYS}/bin ; \
-         echo '(progn (gbc t) (load "${OUT}/makeint.lisp") (gbc t) 
(user::spad-save "${SAVESYS}"))' | ${LISPSYS} )
+#      @ (cd ${OBJ}/${SYS}/bin ; \
+#        echo '(progn (gbc t) (load "${OUT}/makeint.lisp") (gbc t) 
(user::spad-save "${SAVESYS}"))' | ${LISPSYS} )
+       @ (cd ${MNT}/${SYS}/bin && \
+         echo '(setq si::*collect-binary-modules* t)
                        ;;
                        ;;  collect a list of loaded binaries
                        ;;
                (setq x si::*system-directory*)
                        ;;
                        ;; save the *system-directory* setting
                        ;; as makeint is changing it
                        ;;
                (load "${OUT}/makeint.lisp")
                        ;;
                        ;; run the image generation code
                        ;;
                (setq si::*system-directory* x)
                        ;;
                        ;; restore *system-directory*
                        ;;
                (compiler::link 
                        ;;
                        ;; link a new image
                        ;;
                        (remove-duplicates si::*binary-modules* :test
                                (quote equal)) 
                        ;;
                        ;; including unique .o files loaded this session
                        ;;
                        "$(SAVESYS)" 
                        ;;
                        ;; output to this image
                        ;;
                        "(setq si::*collect-binary-modules* t)
                        ;;
                        ;; and fed input of, checking for erroneous loads
                        ;;
                        (load \"$(OUT)/makeint.lisp\")
                        ;;
                        ;; run the image generation code again
                        ;;
                        (when si::*binary-modules* 
                                (error si::*binary-modules*)
                        ;;
                        ;; Bail if we loaded instead of init'ed
                        ;;
                        (setq si::collect-binary-modules* nil 
                              si::*binary-modules* nil)
                        ;;
                        ;; stop collecting loaded pathnames
                        ;;
                        (setq compiler::*default-system-p* t)
                        ;;
                        ;; don't forget to use :system-p
                        ;;
                        (gbc t)
                        ;;
                        ;; collect garbage
                        ;;
                        )" 
                        "$(OBJ)/$(SYS)/lib/sockio-c.o
                                $(OBJ)/$(SYS)/lib/cfuns-c.o 
$(OBJ)/$(SYS)/lib/libspad.a" 
                        ;;
                        ;; here we need extra C libraries
                        ;;
                        nil
                        ;;
                        ;; initialize via redirecting load commands
                        ;; when proicessing the input, not 
                        ;; explicitly via a si::user-init function
                        ;;
                )' | ${LISPSYS})
        @ echo 6 ${SAVESYS} created
        @ cp ${SAVESYS} ${AXIOMSYS}
        @ echo 6a ${AXIOMSYS} created


With this setup, axiom can compile with externally installed GCL on
(with a slight bit of anticipation) all 11 Debian architectures and
amd64.   A case can clearly be made that GCL should handle all this
arch specific stuff itself, but this is the reality at present.  I'm
certainly open to suggestions.  I could provide a lisp variable
indicating that the current arch cannot save loaded .o files if
desired -- this would obviate the need for a separate determination of
the architecture in a configure script.   Or, since all your image
generation code is nicely packaged into an external file, I could
write a superset function to save-system and compiler::link for use
in all cases.  The only tricky part in complete transparency is saving
input fed in not from a file, but via a terminal, and presering the
modules list across multiple invocations of load, save, rerun.  Things
can definitely be improved, but so far there has been slight demand,
and the existing per application work-arounds have proved adequate.
And GCL has more pressing ANSI issues to attend to.

Anyway, the more important patches from axiom's point of view are,
IMHO, the 1) issues.  With these applied, axiom no longer need
maintain a subbuild of GCL, at least on i386 and the like. Possibly
going further, the 2) issues could either be applied in all cases,
only the needed cases, or not at all, leaving it up to debian/rules to
do for Debian distros only.  Whatever you feel is best.

Take care,

Tim Daly  <address@hidden> writes:

> Camm, Mark,
> 
> I'm sure the patch issue is clear to both of you but I'm feeling 
> a bit lost over what the issues are, what flaws you are trying to
> correct, what your suggestions are and what the options you
> are suggesting imply. Camm, could you explain it at a more
> 3rd grade level? (I guess the coffee isn't strong enough this 
> morning). I'm not familiar with Debian. I'm not clear how Axiom's
> socket code is supposed to get put into GCL without change. I'm
> fairly certain (but willing to be convinced) that "configure" is
> not going to work with the Axiom build (I read the docs, wrote an
> example, and abandoned the approach last year).
> 
> Tim
> address@hidden
> 
> 
> 
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah