[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Gcl-devel] Roadmap
From: |
Camm Maguire |
Subject: |
Re: [Gcl-devel] Roadmap |
Date: |
05 Jun 2002 18:58:04 -0400 |
Greetings, all, and thanks for your feedback!
Just to clarify one thing here -- I'd like to use the 2.4.x numbering
to refer to stabilization of Dr. Schelter's work. I think 2.5.0
already should have new capabilities, as it does currently to my
understanding. But this does remind us that we need to port a few of
the bug fixes currently in cvs and release a 2.4.2. Would anyone like
to volunteer a patch, taken from a subset of 'cvs -rdiff Version2_4_1
HEAD' or whatever the cvs command would be? I know of one clear error
which affected bignums. In addition, we have yet to fix 'declare
integer' to add gmp support. I've worked on this a bit. I think we
should ensure that 2.4.2 will build maxima.
As for 2.5.0, I'm just summarizing the results here, but it seems the
consensus is that we should adopt the following:
Pre-2.5.0:
Major:
1) Clean builds with -Wall
2) resize default memory parameters and stack sizes
3) Check licensing
4) Incorporate pcl/clcs in some fashion (i.e. perhaps as loadable .o
files)
Minor/Optional:
5) Whatever architecture patches I can get working in time.
6) compressed info file reading
7) working defsystem
8) clocc tests at build time.
Post-2.5.0 (with comments):
1) Boehm gc option -- As I get more used to the code, the more
impressed I am by various gcl features, which I think we should
bring to the attention of our marketing department :-).
One of these is the current mm design. I had suspected some
mm/gc bug when I ran into difficulty porting gcl to other Debian
architectures, but this has never been verified. I've found posts
by Schelter testifying to there being no leaks. Far beyond this,
we have an *exact* gc system to my understanding, the only negative
of which I've read about is its difficulty of implementation :-).
For those non-gc experts like myself, this means that the entire
heap is explicitly walked with every possible pointer location
checked, as opposed to making assumptions from a perspective of
ignorance as to the actual data structures occupying the heap.
In addition, at least on Linux ix86, the system tracks when
each page is written to by marking them all PROT_READ and catching
the SIGSEGV, then changing the perms to add PROT_WRITE and marking
the page for later gc processing. Unwritten pages can be gc'ed
much faster. This 'stratified' gc seems like a great idea, but I
still do not understand everything.
Once more, all pages are marked to hold only a certain class
of lisp objects. Don't know exactly how this is exploited yet, but
it certainly should have advantages over the alternative.
Finally, malloc/free *appears to work just fine* across gc!!!!
I haven't completely verified this statement, but I think the way
it works is like this: mallocs return a pointer to a contiguous
block of memory, which is then 'consed' into a malloc list in the
lisp heap. This effectively protects it from gc, and *I think* the
page type t_contiguous is spared relocation as other items undergo
gc. I used to think one had to trap mallocs and redirect to a
static array if one wanted to preserve storage across function
calls with malloc, but I've tested eliminating this redirect, and
all works apparently perfectly. I'm just about to axe the
do_bfd_malloc I added before -- the structures which bfd_init
mallocs internally are apparently protected for the duration of the
program from corruption by gc!
If this is correct, we have the following tremendous advantage
-- we can use third party libraries without caring if they malloc
or not, potentially opening up a vast array of standard C library
functionality to gcl. My only hesitation in this conclusion is the
fact that Dr. Schelter had forked the gmp code and replaced an
internal malloc with alloca, saying it could cause a problem for
gcl. But others have simply linked against -lgmp and proceeded
without issue. Perhaps the malloc emulation was not completed when
Dr. Schelter incorporated gmp?? We need some good bignum test code
to explore this issue. Vadim?
2) UFFI -- forgot to mention this in the roadmap, but it fits in as
others have noted with the additional bindings I was thinking of.
I wish someone in the know could explain to me why a system can't
just generically access external libraries with a syntax identical
to the Fortran/C syntax, and have the internals of the system block
gc for the call and connect the right addresses together. Having
to write a lisp wrapper for every external routine is going to get
lisp nowhere fast in terms of new users, it would appear. I know
my idea is naive, but I still don't see the issue in principle.
3) external libgmp support -- as mentioned in 1) we may effectively
already have this. I just wanted to mention that what I'd like to
do, at least for an installable system package like the Debian
package, is to compile the gmp as shared libs for each
subarchitecture relevant to the general arch, and install these
libs in such a way so that gcl would use the fastest appropriate
one given the running CPU. The method of doing this has already
been worked out for atlas on Debian, so should be straightforward.
And I'm guessing that this would just about make gcl as fast as
possible for bignums with minimal effort to the user.
4) ecls issues -- I'll try to fire off a note at some point to the
developers.
5) Makefiles -- moderately important, straightforward, tedious, adding
nothing new in terms of functionality.
6) 64bit ports -- I feel we need to be ready for this. Luckily, there
is an alpha-osf1.h already.
This brings up a related issue: fasloading. In the world of
lisp, people seem to find the ability to dynamically load and
run .o files an extremely useful, distinctive reason to use
lisp in the first place. And gcl certainly has the hooks to
do it fast, though portability suffers. The most portable way
to do this in the old code appears to be with dlopen. The
disadvantage here is that one cannot save the system image
with the loaded files, at least if Dr. Schelter's faq is still
accurate. The bfd work I've been doing would appear to extend
current x86 behavior to *many* systems portably, at no cost in
speed, but perhaps some increase in memory. Here are the
choices we face:
1) No one cares about anything but x86, put this on the back
burner.
2) People like to load, but don't care about saving the image,
and are content to link the .o files explicitly if they
wish. Use dlopen for portability.
3) People like this loading to be as fast and small as
possible -- put the reloc information by hand into
sfaslelf.c as Dr. Schelter did.
4) People want the speed, but the size is not so critical.
Use bfd everywhere, and take advantage of a probable speed
increase as the symbol lookup proceeds through a hash table
instead of a sorted binary tree.
Just to report here -- Daniel Jacobowitz's suggestion on
clearing the cache worked perfectly on ppc. We now have a ppc
port just as good as the others. But everywhere expect on
ix86, I'm still running into a maxima compilation problem.
The let macro defined in the build gets corrupted to "let."
somewhere.
7) Rainer's check -- maybe keep this out of the distribution, but I
need it to narrow down the issues I'm seeing in 6)
8) Performance --
a) someone needs to comment on the *speed* of the current gc,
as opposed to the number of gc's typically encountered in
normal use. I've heard conflicting reports, and would love
some numbers.
b) I feel we could easily access ISA extensions for bignums,
and possibly also matrix operations via blas/atlas, but I'm
not really sure if anyone particularly cares.
c) Over the very long term, one could use gperf to make a
perfect hash table for the saved_gcl image to be used in
loading, lex/yacc to build a very fast lisp reader/parser,
etc.
Sorry for putting this all in one place. I'm partly writing this to
provide a reference for our future work. Comments/ideas of course
most welcome.
Take care,
--
Camm Maguire address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Re: [Gcl-devel] Roadmap,
Camm Maguire <=