Re: The sad state of locking -> it almost works

[xystrus]

On Fri, Apr 12, 2002 at 09:30:17PM -0400, Alain Magloire wrote:
> > 
> > FWIW, what I've been up to is more-or-less re-implementing something
> > along the lines of the API of the original libmailbox.  There are a
> > couple of reasons I've been doing this.  The foremost reason is purely
> > as an academic exercise.
[SNIP]
> > Anyway, you guys are doing some really cool things with mailutils, and
> > I like the idea of url access especially.  But I'd rather see that
> > implemented in a different layer, on top of the physical access layer.
> > It seems to me mailbox kinda tries to handle everything all at once,
> > and I'd rather break everything down into layers -- physical layer,
> > application layer (like dealing with MIME and e-mail addresses and
> > lions and tigers and)...  
> 
> Good points, we will do that: two layers, the first one dealing with the
> mailbox format nothing fancy i.e. get the job efficiently,
> the second layer: the framework to do all that fancy things.

I was originally thinking of 3 layers -- the third to deal with
accessing e-mail over a network.  Though now that I think more on it,
I don't think this is quite so much a "layer" as I think the code for
this module (or set of modules) is largely seperate and distinct,
sitting at the same level as the physical mailbox access stuff.  I
think a diagram might look something like this:

              --------------------    -------------------
              |                  |    |                 |
  ACCESS      | physical mailbox |    | network mailbox |
  LAYER       |  access module   |    |  access module  |
              |                  |    |                 |
              --------------------    -------------------
                           \            /
                            \          /
                             \        /
                        ---------------------
                        |                   |
  APPLICATION           |    application    |
  LAYER                 | data manipulation |
                        |     module(s)     |
                        |                   |
                        ---------------------

So I guess not so much three layers as two, with two seperate modules
(well, groups of modules... more on that in a moment) in the access
layer, and one or more module in the applicaton layer.  I might also
do well to rename physical mailbox access as on-disk mailbox access...
I think both that and network access qualify as physical access,
though of a very different nature.  

Having not ever really dealt with writing an e-mail application, I
haven't really given much thought to what kinds of modules might be
needed in that second layer.  I guess functions for processing MIME
components of a message would certainly need to be one of those
modules, and I suspect you'd need functions for parsing e-mail
addresses...  prolly some other stuff.

Ok, so the "modules" shown above in the access layer are not really
individual modules, but sets of modules.  For example, the physical
mailbox access "module" would contain modules for accessing mbox, mbx,
mmdf, maildir, mh, etc.  The network mailbox access "module" would
contain modules to handle POP3 connections, IMAP connections, SMTP
connections, etc.  They're not modules but groups of modules.

Ideally, the code for all of these modules is written modularly enough
that no module in the access layer need know anything about the code
in any of the other modules.  The modules in the application layer
need know /only the exported API from the access layer modules/.  With
one exception, that is: they all obviously need to know the definition
of a mailbox_t structure.
         
[Also possibly worth noting that, while clearly mail clients and access
servers will each do both functions, I think of network access as
being mostly a function of the client applications, whereas on-disk
access is more a function of the server applications.  All I'm really
saying here is that to a server, the on-disk view of a mailbox is more
important, whereas to a client, the on-disk view of a mailbox is
becoming decreasingly important, as POP and especially IMAP become
more and more widely adopted.  I don't know if that makes any sense to
anyone but me, but if not that's ok.  If it doesn't make sense to
you, just ignore this whole paragraph.  :) ]

> As you are pointing out one should be able to use say pop3 without
> having to go through the entire framework layer.

Yeah, I agree that makes sense.

> So if you are working on one of the formats: POP3, Mailbox, Maildir,
> IMAP4, MH etc ..  it probably can be reuse once we move on to the
> second stage.

Well, right now I'm concentrating on mbox, because I've been somewhat
fascinated with performance issues surrounding that format ever since
the first time I started using UW-IMAP.  I'm experimenting with a few
things: As I've already mentioned several times, I'm playing with MMIO
vs. file I/O.  I'm  also screwing around with ways to eliminate temp
file usage by re-writing the mbox file *in place* and only re-writing
that which is essential to be re-written.  My current theory is that
performance can be enhanced significantly by doing this, rather than
by re-writing the whole mailbox to a temp file and moving it back to 
where it goes...  I've also got a similar idea about how to modify
message status (and possibly other frequently modified headers, like
X-UID) without re-writing ANY of the mailbox, other than the status of
the given message itself, which would obviously increase the
performance of such an operation tremendously.  If you like, I can go
into more detail about my theories on this.  But right now, I need to
eat some lunch...  ;-)

Once I get done monkeying around with all that, I'd probably be
interested in tackling maildir, as I think it also represents an
interesting academic challenge (though I gather Dan J. Bernstien is a
genius, and simply reading his code would be sufficiently
instructional).  I think I've also mentioned to you off list my
interest in IMAP.  However, I start a new job tomorrow (yay! a
paycheck!) so my time to mess with all this will be much more limited
now.  And I can't really pull 25 hr. coding stints like I did the
other day... ;-)  So I'll probably be messing with mbox for about
another month or so.

> You are probably a little harsh on our first attempt, but it's not
> that bad and we learn a lot from it and most of the code will
> probably be backported with some care not too make the same mistakes
> 8-)

Oh well hey, it was not my intent to be at /all/ harsh.  I just find
the current incarnation of mailbox doesn't quite suit my needs (and
also that with what I will learn in the process, I will personally
benefit greatly from trying to implement most of this stuff from
scratch myself).  Like I said, I think you guys are doing a lot of
interesting and useful things with the library.  I would also comment
that I have not looked all that closely at very much of the existing
code (though some of it), so I don't really have the insight with
which to fairly make harsh criticism! ;-)  

Comments on all of the above are quite welcome, and I take criticism
quite well (even harsh criticism, usually :) so feel free...

Xy