Re: mailbox (was Re: intro)

[Alain Magloire]

> 
> On Wed, Mar 13, 2002 at 08:56:58AM -0500, Alain Magloire wrote:
> > > Actually I've been poking through the libmailbox code a little bit,
> > > and I think I would like to play with that for a while.  On the whole
> > > it looks to me like it's pretty well written.  Of particular interest
> > 
> > If mail handling was that simple 8-).
> > No, you are looking at the wrong code, libmailbox was our first
> > version way back, it is obsolete now.  The code is in mailbox/*
> > all the added made it, .. a "little" more complex 8-).
> > This is what I meant by things getting too complex, but I understand
> > why the complexity and thought we could have done a better job
> > handling this .. but that's allright, next time around with the experience 
> > 8)
> 
> Ok, now I understand your comment, as well as your mentioning you wanted
> to separate the functionality of the lower APIs from the POP and IMAP
> stuff.  FWIW, I agree with you...  I think you were on the right track
> with libmailbox.  It's missing a lot of stuff, and there's no hint of
> signal handling in it at all, but as far as mailbox access I think
> the API you defined is perfect.  All the other stuff should be built
> ON TOP of that, i.e. handling MIME is not specifically a part of
> mailbox access...  I think it would be best to have the API you 
> originally defined for accessing mail folders, and then build seperate
> APIs that sit on top of that API for the other stuff.  Though there may
> be issues I'm overlooking, and hopefully the list will point those out
> to me.

OK, I'm in my "Retrospection mood", ... so ..

The design is actually quite simple:
- a number of low level API to handle different mailboxes.
  POP3 IMAP4 MBOX(unix) MH MAILDIR etc ..
  Rationale: since they are so different, better not try to mold
  to a restrictive API, but rather let it be specific to the format.

- A framework that will provide a unify way to access mailboxes,
  the type of mailbox is hidden and a general API is/was provided for
  all. mailbox_t
  Rationale: this will let any utilities be able to handle POP3,
  IMAP etc .. for free.

- URL a uniq way to identify a mailbox:
   imap4://address@hidden

- A bunch of other objects to handle mail related tasks, encoding
  mime handling(mime_t), streams(stream_t) etc ...

Somewhere things took a bad turn when facing the harsh titanic world
and all sort of imcompatibilities, handling of async I/O, thread-safe
etc ..
Ok, I'm exagerating a little 8).

For example MH provides "sequences" which does not quite
map on all mailbox, or IMAP provides enormous flexiblity(some say
complexity) like unsolicited responses, SEARCH capa.  Disconnected
mode, synchronisation, and I'm glossing over many things.

C is/was ubiquitous and was choosen as the language .. or put it
another way .. we did not know C++ 8).

Would C++ made a difference? I do not know, maybe.

In any case, now that we have gain some experience, we are looking
at ways to do things better(maybe the simplistic approach of libmailbox)
- First freetime is limited and its a lot of work
- Second we still can learn for the current mailbox.

> > > to me is what effect switching to MMIO would have on performance and
> > > memory utilization, for both small mailboxes and very large mailboxes.
> > > I may play with this, and then report my results back here, if there's
> > > interest.  In theory, MMIO should have a noticable positive impact on
> > > performance, at least in the case of machines with plenty of RAM.  
> > 
> > Is MMIO, Memory Map files? if yes we do that for Read-only files.
> 
> Yes.  MMIO = Memory-Mapped I/O.
> 
> > But it turns out that the performance gain is relatively small
> > for big files(lots of page fault) and most of the time the spool
> > is on remote machine access across NFS.
> 
> Right... you do need to have a lot of memory for the gains to be
> noticeable.  It still shouldn't be slower, unless you really have
> way too little memory in your server.  If that's the case, BUY SOME
> RAM!  It's dirt cheap.

I only got 4 slots on my PC.  And,(this is not address to you of course 8)
buying RAM is no excuse for sloppy programming.

> Somebody mentioned having a configure option for disabling MMIO later
> in this thread, and I'd agree with that.  But (without having done
> any empyrical testing) I suspect that for most cases, MMIO will be
> faster, and I think the code is easier to write too, so it should
> be the default.

Well, I use to maintain GNU grep not so long ago and I had to
put a --mmap do disable the use of mmap() it was counterproductive
and rather a penality on many cases. When you were grepping thousands
of files it's suprising how many page faults arise and it turns
out that a well tune I/O buffer could do as well and perfrom in all
situations including remote filesystems.

> > > I'm also interested in the mbox_expunge() function... I haven't
> > > gotten around to looking closely at it, but I did notice a comment
> > > that claims it breaks splendidly.  Can someone comment on how it
> > > breaks?  Is this still a problem?  I've also already written some
> > > functions that more-or-less implement expunging messages from mbox
> > > folders using MMIO, so it should be reasonably easy to convert what
> > > I have into a form that makes sense for libmailbox...
> > 
> > Interresting, it is a lot of moving around.  Did you notice
> > any gains for large mailbox, with random delete messages?
> 
> Honestly, I didn't do a lot of comparison testing.  At the time, I
> was more interested in just figuring out how to do it than comparing
> performance against non-MMIO methods.  As I mentioned, MMIO is fairly
> new to me...  I'm also not sure about portability issues, but since
> you're already using it, it would seem that's not a problem...

The problem is when you delete or adding things randomly(Random Access)
on a file, its lot of moving around.  Take a loot at
mailbox/mapfile_stream.c(_mapfile_write()).

> > > It also seems that support for the vaunted Maildir and MH folders
> > > isn't finished.  Is someone actively working on these?  I've been
> > > meaning to learn more about how Maildir is implemented anyway, so I
> > > thought I might have a look at finishing that up.
> > 
> > MH was done by sergey.
> > I do not think anyone is working on maildir.  To do maildir
> > you will have to implement must functions mark as "stub"
> > in mailbox.c in a file mbx_maildir.c file, see mbx_pop.c,
> > mbx_mh.c, mbx_mbox.c for some complex examples.
> > 
> > > Thoughts?
> > 
> > Not easy, and one shot of tequilla will not be enough 8)
> 
> Well last night was mudslides... ;-)

8)