Re: IMAP server (was: Re: [Savannah-hackers] gnu-socketserver)

[Alain Magloire]

Bonjour

> >>> "Alain Magloire" <address@hidden> 26-Feb-01 4:40:49 PM >>>
> 
> >Somebody asked me if it was possible to write a 
> >full JavaMail using mailutils as the back end.  
> >The answer was yes.  His ideas were to cut 
> >development time by implementing most of the 
> >functions with JNI wrappers calling mailutils.
> >Although feasible it is not a good idea, JavaMail 
> >provides a very nice framework which is _pure_ 
> >JAVA(i.e portable).
> 
> Javamail provides a horrible framework... I can't stand it
> personally.

8-), 8-)
In that case, I think you are "oversimplifying"(Is that a word ?)
True, some design decision were bad but there are some good ideas.

> 
> However JNI would be terribly slow and not very nice.

Yes synchronizing the JNI backend mail handling with the Java code
can quickly be a no fun job.

> Sun's Javamail is slow enough without getting JNI involved.

Now this is a suprising statement : "Sun's Javamail slow".
I've never use it in any applications, why is it slow ?
Surely the bottleneck is the network, for example, downloading MEGS of
messages from a POP server, then the framework being slow.

I do not see any rationale in such statement.

> 
> The oje project (at sourceforge but soon part of GNU I hope) is
> developing a free (LGPLed) version of Javamail. That's where my IMAP
> provider fits in.

OJE, that's news to me.  Jeff that's a another we should put in the list
of "competitors" ;-).

- how OJE, will be faster then JavaMail ? rather how will it achieve
the speed that you claim JavaMail suffers so much ?

- You say that JavaMail is horrible, but yet you are in a project
to clone it! that seems  odd.  I would have expect that you target the
problems that JavaMail suffers and come up with some other ways(API) to
address them.

> >A full IMAP server/client, will be hard on the skin, 
> >IMAP4 protocol is complex compare, to say, POP3.
> >I'm sure you are quite aware of the rfc2060(IMAP4), 
> >the coming IMAP5 and all the protocol extensions 
> >(idle, threading, etc ...)
> 
> I wasn't aware of IMAP5... is it much different? When's it due? If
> not for a while then that's why I haven't heard of it - I don't let
> things come on my radar till they've matured a bit  /8->
> 

It is still a draft, under multiple revisions, you can find a copy
at the IETF.  But I beleive there is a link on the IMC(www.imc.org)
Go to the internet drafts.

> I've been hacking various IMAP4 tools (mostly in Java) for a couple
> of years now. I know the protocol pretty well. 
> 
> I think clients are harder than servers, the server just has to
> package up the data in it's store into s-expressions and keep a
> running overview of thinks like incomming messages. It's quite a nice
> protocol from a server point of view. 

8-) If you like lisp  yes.
BODYSTRUCTURE is my personal favorite when it comes to implement a client.
At first, I wanted to be renegade and not use recursive calls.

> 
> Where it gets hard is in parsing all those differents kinds of data.
> Why couldn't they just stick to s-expressions for everything? Having
> said that clients aren't so difficult in Java because the Kawa system
> (a GNU tool) provides an s-expression reader which works fine for IMAP
> parsing.

Agreed, If you have a good parser engine extracting the info, the sun
will shine again.

> >: use files as messages or files as folders
> >Not sure I follow here, if you have a Unix mailbox 
> >format, you are going to explode the messages into files ?
> 
> Don't think aboput the traditional mailbox format with this server.
> It's a mailstore with only ONE interface: IMAP. All work is carried
> out through the protocol interface.

Ack.

> In one implementation of the backend the store would be one directory
> per user, each user having under that directory one directory for each
> folder and one file for each message. Things like flag options would
> be stored in special files not available to the user through the IMAP
> interface.

Comment:
Many implementations take the approach of not saving the information
on the mailbox, this seems to gave them some speed gain, by choosing
a suitable format.  But maintaining a external files can be a burden,
and since the info is maitained elsewhere the users are lock with a particular
implementation/program.

> In another implementation the user's mailbox might be trad unix
> mailboxes, or MMDF files or some other wierd and wonderfull file
> format but it is always hidden completly from the user by IMAP. The
> user won't have rights to the filesystem directories which make up the
> mailstore.

Agreed, decoupling your backend this way give the highest level of
flexibility.  I can imagine that one day you may also  have SQL database
as the backend mailbox.

> Personally I'm only interested in the one-file==one-message system
> because that will be real fast. It has a downside that it's wastefull
> on inodes which is why I'm abstracting the interface to the backend;
> people who want to store their mail on an NFS server can write an
> inode econmoic backend. Personally I think huge mail stores are a bad
> idea... multiple servers work better.

Ack, ack.  It also makes it cleaner for locking, less race conditions.
IIRC, news(NTP) did have problems with this approach one file per message.
Mh and qmail uses this approach.

> BTW a side effect of my "you only talk to this through IMAP" is that
> it eliminates all locking problems. Since there is only one process
> accessing the IMAP store we don't have to worry about locking at all.

And also it gets rid of the security with setgid().
Are there other security concerns ?  Do you have to setuid()
when appending a email , for example?

> Yes. I understand that. But there's no reason that you can't have
> delivery via IMAP. That's what append is for. The delivery agent just
> has to have the rights to do the delivery.
> 
> Here's how a delivery agent works:
> 
> - exim recieves mail for nferrier
> - exim works out that nferrier's mail goes to imap server "nicimap"
> - exim calls delivery agent
> - delivery agent opens IMAP connection to nicimap
> - del-agent logs into IMAP as super user
> - del-agent selects folder "/nferrier"
> - del-agent appends incomming mail to inbox

I think your saying mail deliver to the local machine, I was refering
to general email that I want to deliver to address@hidden where xxx.yy
may have MX records, or may not be available so the sender will have
to queue  etc ...

> 
> At the last stage the user might want some scripts to execute (for
> filtering) so we'll pass the incomming mail through some filters
> written in Scheme.
> 
> Geddit?
> 

filtering, search, spam control capabilities etc .. are they also part
of this ?

> >So far, I've never see the behaviour of a server written 
> >in pure Java put under heavy load i.e if the webservers 
> >of google.com or yahoo and the databases were written 
> >in pure java, what would be the performance?
> 
> Try www.talk21.com. We've got 3 million subscribers, a *lot* of hits
> and a complex app. It's all pure Java (except the IMAP and LDAP
> servers).
> 

A good start, indeed.  The numbers are impressive.

But I need some hard evidence.  Example, a pure java database
on an industrial enviroment etc ...
JVM technology came  along way and are getting better but compare
to a well crafted/optimise C program ... I would like to see more numbers.

> >I've written some industrial programs with Java in 
> >another life but the emphasize was on AWT and nice 
> >little windows.  The heavy stuff was still in C/Ada 
> >and fortran.
> 
> Java on the server side is perfectly acceptable. It's robust and it's
> fast. I think you'll be surprised.

Yes,  there is no reason with a good JIT to not have some
acceptable performance.

> >The problem beside the performance concern is 
> >security.  Servers usually fork() for security concerns, 
> >many daemons have to setuid()/setgid() to do some 
> >priviledge operations.  Threads  do not isolate enough
> >since it will affect all the other threads with nasty race 
> >conditions.
> 
> Yes. That's true. Java can't help you there. I'll have to run my
> server as root. 
> 
> But I'm not thinking about my server in the way that you think about
> normal Unix services. I will dedicate a whole server to it (and
> probably several servers in the future). 

Comment:
Threads are not the panacea.  If one is not carefull using threads it
may actually __slow__ things down, because of the synchronisation needed
to protect the shared resources and predicates.  On a single CPU a mutex
may be implemented with a compare_exchange instruction that blocks the bus.

Its cool when you have 2 CPUs and more and your working threads can be
scheduled on a CPU each an run happy.  It's great if you have lot of IOs
and you could use the idle CPU meanwhile.  It's another story if you have
CPU bound calculations and spending time sync the threads.

> >Thank you for sharing them.  Any URL handy ?
> >The project sound very interesting/promessing.
> 
> Part of the reason that there's no URL handy is because I've not
> pesuaded anyone else to help. I'm trying to get the owner of talk21 to
> fund the development (as I could do it myself in about a month).

Cool.

-- 
au revoir, alain
----
Aussi haut que l'on soit assis, on n'est toujours assis que sur son cul !!!