RE: [Gcl-devel] Recent HEAD changes

[Mike Thomas]

Hi Camm.

Thanks for the detailed answer.  Time is short so I'm responding as best I
can.

| > 1. Your newly checked in test for the TYPE_BITS macro causes a
| fatal end to
| > the configuration script.
| >
|
| ??? Need more detail here -- what is the problem?

Fixed now - don't worry - inlining pwd into the C source like that gives a
bad path.  I substituted "./" for "`pwd`/".

|
| > 2. I'm getting a negative CSTACK_ADDRESS -2147483648
| >
|
| This is normal if accurate.

main.c: In function `main': main.c:294: warning: this decimal constant is
unsigned only in ISO C90

|
| > 3. Are these sensible values?
| >
| >   #define CSTACK_TOP 0x80000000
| >   #define CSTACK_RANGE 0x40000000
| >
|
| Yes, It appears as if your C stack counts down from 0x80000000,
| leaving the range above this value fair game for immediate fixnums.


Actually it counts down from the values shown below in the default case:

=========================================================================
$ cat junk.c
#include <stdio.h>
void test_fun ( void )
{
    int test_fun_var1;
    int test_fun_var2 = 0;
    fprintf ( stderr,
              "FUNCTION AUTOMATIC VARS:   test_fun_var1 %x, test_fun_var2
%x\n",

              &test_fun_var1, &test_fun_var2 );
}

int main(int argc, char **argv)
{
  int test_var1;
  int test_var2=0;
  void *heap = malloc ( 1000 );
  fprintf ( stderr,
            "FUNCTION TEXT:             test_fun %x, main %x,\n"
            "MAIN AUTOMATIC VARS:       test_var1 %x, test_var2 %x,\n"
            "MSVCRT heap:               (WARNING: NOT GCLs heap) %x\n",
            &test_fun, &main,
            &test_var1, &test_var2,
            heap );
  test_fun ();
  return 0;
}

$ gcc junk.c -o junk

$ ./junk.exe
FUNCTION TEXT:             test_fun 401290, main 4012c5,
MAIN AUTOMATIC VARS:       test_var1 22ff74, test_var2 22ff70,
MSVCRT heap:               (WARNING: NOT GCLs heap) 3d3d20
FUNCTION AUTOMATIC VARS:   test_fun_var1 22ff24, test_fun_var2 22ff20

=========================================================================


| >
| >   (Perhaps some comments in "h/gclincl.h" would help?)
|
| Yes!  Could you suggest some?

Probably the description you've posted in this email or else the short
comments you provided for each would be a good place to start.  If you like,
I'll convert your notes myself as it is probably better if an "outsider" to
your thought processess summarises.  I'll check it in when I put my own
changes back into CVS.

|
| >   There is no "ldd" on Windows so the shared lib heap ceiling test is
| > meaningless.   DLLs are loaded above the main program text etc
| and below the
| > heap.
|
| How can one discover this programmatically?

You probably can't in a completely failsafe manner - heap can come from a
number of sources on Windows (eg the MSVCRT runtime, or several calls in the
Win32 heap management API) - and don't forget that any DLL's are completely
sealed up - so they are calling their own heap allocation functions (any of
the above) completely separately from GCL's own heap allocation mechanism.
It appears from the above test program that the MSVCRT heap (ie, memory
return by malloc()) is living between the C Stack and the program TEXT.

| In short, during my visit to UT, it has become apparentl that there
| are many 'serious' GCL users, by which I mean industrial users pushing
| the limits of GCL on their machines, pushing to n-way 64bit GCL with
| gigs of ram, etc.

Yes, I agree that this is very exciting news and certainly worthy of our
attention as you are currently doing.  Just remember that I'm not half as
energetic and knowledgable as yourself, so I'm always ambling along far
behind you sniffing flowers on the hillside rather than working assiduously
on GCL!!

| I've tried to redesign the memory configuration
| process to make the most of the address space available.

You certainly have!

| Given DBEGIN and MAXPAGE, try to grab as much C stack as possible to
| avoid artificial limits on recursion depths.  Bound the C stack if it
| counts down by the max heap (and ideally if I can figure out how to do
| it, by the upper bound of the shared library memory, though this would
| have to be done at runtime, as we can change this via compiler::link).

The C Stack bounds for Windows can be calculated from the top of the C stack
and the size set at link time (the stack grows down).  There is no such
thing as an "unlimited" C/executable stack on Windows.  Indeed,
compiler::link can be used to set a new C stack by passing the linker a
command line option like that below  (LD_STACK_FLAGS in configure.in, set
from CSSIZE/enable_cssize).

| If the C stack counts up from high addresses, bound it to half its
| range to make room for an immediate fixnum table unless disabled.
| >From this, get a max used mem address, which I'm incorrectly calling
| CSTACK_TOP for the moment, though it need not be this if the C stack
| is below DBEGIN.  I assume that the C stack start and direction are
| system immutables.

This last assumption is fine on WIndows.

|
| Given this, use the remaining high addresses, which are inaccesbile to
| the heap under any circumstances, for immediate fixnums.

Anything below DBEGIN is inaccessible to the GCL heap on Windows, although
DLL's are presumably being loaded into the memory hole between the program
text etc and DBEGIN.

| Getting this all to work portably will take a little sweat, but I
| don't think too much.  We're close to autoconfiguring all of this
| across the Debian 12.

Now for the Windows 5 or 6.

| > 5. On Windows, the C stack lives below the program text, data
| and heap and
| > its size defaults to CSSIZE 1073741824 but can currently be altered by
| > configure to some other value as required.
|
| OK, as you might have notices, everyone can configure CSSIZE now too.
| Your defaults might no longer be needed, but in any case would be
| great if you could give it a look over to streamline things to the
| extent possible.

Windows defaults to a 2 MByte stack but that value (or some new size) needs
to be passed into GCL for NULL_OR_ON_CSTACK to use (at least).

| We also have sigaltstack based continuation on c stack overflow in
| main.c.  Might want to check how this would work on mingw too.

I'm ifdefing it out as memory violations are handled differently on Windows
and the solution to this kind of problem is bound up with the solution to
SGC and is currently in my too-hard basket.

| > 6. The Lisp heap start is normally hard-wired by our
| configuration script to
| > start at DBEGIN =  0x1a000000.  I am thinking that I could
| lower that start
| > point or just remove it altogether - the heap start address can
| be altered
| > automatically as the system tries to grab heap memory anyway
| according to
| > whether it succeeds in getting the blocks it asks for.  How this would
| > affect unexec I am uncertain at this stage.
| >
|
| My understanding is that DBEGIN should never move after compile time.
| I would suggest setting it as low as possible, and ideally, having
| this determined from configure magic to flow with future changes to
| the mingw address space layout.

At risk of belabouring a pedantic but crucial general point - that is not
"mingw address space layout" but "Windows program address space layout" - ie
it applies to any Win32 compiler/linker we may care to use with GCL -
Cygwin, MinGW32, MS VC++, Borland etc.

|   It sounds as if all your obstacles
| are below DBEGIN anyway, meaning you can likely have the largest GCL
| heap of all the ports!

That depends on the version of Windows and also how the system is
configured.  A typical NT or XP system has 2 G of space accessible to the
process itself (and 2 Gig for the system in relation to that process - not
sure what that is - presumably process specific system DLL heap/data), but
can be configured to have 3 G at boot time (with just 1 Gig for the system).
Win95/98/ME is similar but laid out a bit differently with different rules -
basically processes can wipe out the OS.

Apparently on Win64 we will be talking up to 7 or 8 Terabytes depending on
the processor type.  GCL has a long way to go before it will be a native 64
bit app on Windows (I expect that it should run fine as a 32 bit app on
Win64) - mainly because we will need to find a free 64 bit native C
compiler.


| But I don't understand your statement that the
| C stack is below the text if you are getting a negative
| CSTACK_ADDRESS.

See the test program above and it's output.  Apparently a bug in
configure.in?  I don't understand how an address can be negative in the
first place except by a C signed/unsigned type problem?  If this is not
really an address, then I think it should be renamed to avoid confusion.


Cheers

Mike Thomas.