Re: Update on Gash and Gash-Utils

[Janneke Nieuwenhuizen]

Timothy Sample writes:

Hello Tim!

> This is an update on porting Gash and friends to Mes.  Here’s my latest
> branch: https://git.ngyro.com/mes/log/?h=wip-gash-utils
>
> It adds another 62 commits on top of ‘wip-gash’.  Git says:
>
>> 60 files changed, 2761 insertions(+), 183 deletions(-)

Wow...

> With these changes, I am able to use Mes, Gash, and Gash-Utils to
> configure, build, and install packages!

That's...pretty awesome!  We had this "dream", and now...

> The most glaring shortcoming is that I had to punt on Awk by including
> Gawk in the test environment.

Any idea what's missing?  Is it just too slow?  Gash' awk does work when
running Gash-Utils on Guile right?

> Good thing we have a long history of building old versions of Gawk!  ;)

Hehe, yeah git is patient.

> New stuff
> ---------
>
> The main improvement is probably the new system interfaces.  We now have
> ‘stat’, ‘lstat’, ‘rename-file’, ‘mkdir’, ‘rmdir’, ‘umask’, ‘utime’
> (almost – see below), ‘sleep’, ‘link’, and ‘symlink’.

Nice, and that makes a lot of sense; Of course Gash needs such things.

At first tried to keep Mes and the default libc *really* minimal, that
was at the time that there was no M2-Planet and I thought we would have
to redo Mes's simple C in assembly by hand.

> I added support for regular expressions by bringing in Dorai Sitaram’s
> pregexp (Portable regular expressions for Scheme) code.  I had to make a
> minor adjustment to it, and wrap it in a Guile-like interface.  That
> interface also has a little hack to deal with “basic” regular expression
> syntax.  Grep and Sed seem to be happy with it.

That's really very nice; I didn't know this was possible.  Do you have
any idea of its performance, and/or did you consider bringing a C regexp
library in?  I guess this was just the simplest thing to do right now,
and much preferrable over translating C to M2-Planet...

> I also added true bytevector support.

Yeah!  You found some remnants of a half-baked attempt of mine ;)

> This means is that Scheme can manipulate ‘TBYTES’ objects via
> ‘make-bytevector’, ‘bytevector-u8-ref’, and ‘bytevector-u8-set!’.

Ah...right.  That's cool!

> I added bytevector I/O procedures, too, but they are written in terms
> of reading and writing bytes (that is, they don’t take advantage of
> the fact that we have contiguous buffers to do fast I/O).  One fallout
> of this is that I deleted MesCC’s pretend bytevectors.  Rather than
> rewrite them on top of true bytevectors, I just had MesCC make lists
> directly.  The reason for this is that it assumes the objects made by
> the pretend bytevector interface are lists later on!

OK.

> We have more code from Guile, too.  SRFI 37 gives us ‘args-fold’, and I
> was able to mostly adapt ‘file-system-fold’ to work on the directory
> support added in the last round for Gash.

Oh my...

> A few smaller items include ‘sort’ (via a SRFI 132 stub), ‘and-let*’,
> setters for immutable records, and ‘set-fields’.  There are more list
> and string procedures and bug fixes, but you can read them in the commit
> log.
>
> Lastly, I made modules the default and disabled the “psyntax” test like
> we talked about.

Yay!  Mes is gaining some serious Guile compatibility :)

> Problems
> --------
>
> When I wrote ‘utime’, I wrote it in terms of the ‘utimes’ system call.
> Now I’ve learned that this is an older system call that is not supported
> of RISC-V.  I will have to rewrite it in terms of ‘utimensat’ (which is
> not hard, but I don’t have time for it ATM).

Ok.

> There’s a weird ‘xxx! multi-squash!’ commit.  There was a bug in the
> changes I made to ‘primitive-load’.  This commit needs to be split up
> and squashed in a few places.  I can do it, but we should coordinate on
> the history rewrite.

Sure, I'll keep that in mind.  As you know, I would really like to
release 0.25 with some basic RISCV-64 support first anyway.  I would
have thought we'd have a 0.25 at least a month ago but I'm kind of
waiting for a consistent set of stage0 releases, with M2-Planet 1.11.0.

The good news here, is that meanwhile we fixed quite some RISCV64 and
other stray 64bit problems that would otherwise have gone into 0.25.1.
Not sure what's holding up 1.11.0, though.

> Awk is going to be a bear to port.  In Awk, I rebind ‘let’ and ‘let*’ to
> have SRFI 71 semantics (nicer syntax for ‘let-values’, essentially).
> However, Mes has very loose ideas about macro binding.

Hmm...

> Rebinding ‘let’ in Mes seems to have global effects.  This introduces
> a circular dependency.  What I want: my ‘let’ → ‘let-values’ → normal
> ‘let’.  However, on Mes my ‘let’ and normal ‘let’ are unified.  I’m
> sure there’s a way to hack around this, but – to be frank – I’m tired
> of “hacking around”.  I’m trying to decide if I could find it in me to
> roll up my sleeves and fix the root problem here....  (Guix code will
> be very hard to run without solid macro support.)

Yeah; my initial macro kludge seemed to have worked too well; I've given
it multiple attemts to rewrite it properly but failed to produce
something better until now.

There's two big things in Mes that I'm still a bit scared about but
should probably be done at some point.  A better macro implementation
should also drastically improve the speed at which something like
psyntax runs (if not making it run correctly in the first place).  Then
there's the idea of a scheme compiler, of course, to fix our performance
problems.  Possibly the Guile-Steel efforts could bring some inspiration
here...

> Questions
> ---------
>
> Besides the bytevector stuff, I wrote some weird code to expose bindings
> in the root module.  See ‘define-in-root’ in “boot-5.mes”.  I could
> follow this pattern to expose more things in the root module while
> having the benefit of the module system to define them.  For example,
> instead of defining ‘string-every’ in “srfi-13.mes” and manually
> unpacking the optional arguments, I could define it using ‘define*’ in
> “srfi-13.scm” (using whatever modules I want) and then expose it in the
> root module.  Thoughts on this approach?  Is the syntax too much?

That sounds like a good idea.  Especially with Mes maturing, we need to
be much less careful at being minimalistic, I think.  In any case,
always "do the simplest thing that could possibly work", and then
refactor mercilessly ;-)

> Something like this would really help with Guile compatibility in the
> long term (and simplify the Gash and Gash-Utils Mes shims in the short
> term).

Sure, go for it!

> Conclusion
> ----------
>
> I’m assuming the next steps (to get from 0.25 to 0.26) will be to
> integrate this work into Guix.

Yes.  This is a big reason for me to want to have a 0.25 that goes into
Guix first.  Now that we have the possibility of using sources from GIT
in commencement (<https://issues.guix.gnu.org/64708#6>) we're not
strictly bound to releases anymore but...yeah.

If integrating mes 0.26 doesn't work right away, it's nice to have 0.25
integrated and know that it worked.

> My idea for that is to get to a point where, after Mes is built, we do
> the rest of the bootstrap with Gash and Gash-Utils on top of Mes
> (using ‘%bootstrap-guile’ only for orchestrating the build scripts).

Yes.

> With that working, we could coordinate releasing versions of Mes,
> Gash, and Gash-Utils that support that outcome.  Of course, I’m happy
> to defer to the maintainer if there are other plans.  ;)

:)

> Lastly, FYI, the relevant branches for Gash and Gash-Utils
> ‘wip-modular-mes’ and ‘wip-mes’ respectively (on Savannah).

A truly beautiful piece of work, thank you so much for working on this.

Greetings,
Janneke

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com