Re: IMAP server (was: Re: [Savannah-hackers] gnu-socketserver)

[Nic Ferrier]

>>> "Alain Magloire" <address@hidden> 26-Feb-01 9:17:40 PM >>>

>> Sun's Javamail is slow enough without getting JNI involved.

>Now this is a suprising statement : "Sun's Javamail slow".
>I've never use it in any applications, why is it slow ?
>Surely the bottleneck is the network, for example, downloading 
>MEGS of messages from a POP server, then the framework 
>being slow.
>I do not see any rationale in such statement.

Perhaps it's rather a blase statement. I find it very frustrating to
work with, particularly the IMAP provider but also the actual API
implementation.

The implementation causes many unnecessary operations and much
unnecessary caching. This slows it down, for what I'm doing anyway. 

For example, when I get an inputstream for reading a message I want
to get the inputstream and not a stream sitting on top of a buffer.
The base64 decoding works on thw whole buffer - it's not stream
based!

Also when they do buffering is crazy. Sometimes the message is read
in just to establish the size of the message! It's wierd.

There are lots more examples... basically it annoys the hell out of
me.


>- how OJE, will be faster then JavaMail ? rather how 
>will it achieve the speed that you claim JavaMail suffers 
>so much ?

Some of the speed problems are architectural so we can't do anything
about those. Where we can we've improved Javamail by reducing caching
so that the user gets to choose how things are done.


>- You say that JavaMail is horrible, but yet you are 
>in a project to clone it! that seems  odd.  I would have 
>expect that you target the problems that JavaMail suffers 
>and come up with some other ways(API) to address them.

No. Implement Javamail first and then join the JSR to change
JavaMail. 

I want to have as many ideas as possible about how we can improve
Javamail without necessarily breaking it so I can take those ideas
into the JCP and we can get Sun and everyone else to sign up to it. I
can see Javamail changing substantially over the next couple of
years.


>> I think clients are harder than servers, the server just 
>> has to package up the data in it's store into s-expressions 
>> and keep a running overview of thinks like incomming 
>> messages. It's quite a nice protocol from a server point of 
>> view. 

>8-) If you like lisp  yes.

I do.

>> In another implementation the user's mailbox might 
>> be trad unix mailboxes, or MMDF files or some other 
>> wierd and wonderfull file format but it is always hidden 
>> completly from the user by IMAP. The user won't have 
>> rights to the filesystem directories which make up the
>> mailstore.

>Agreed, decoupling your backend this way give the 
>highest level of flexibility.  I can imagine that one day 
>you may also  have SQL database as the backend mailbox.

It could be done but I can't understand why you'd want to do it.
Files are very close to messages in terms of "object-ness". The reason
people don't use files very much is because they've got problems with
scalability and locking.

I'm trying to solve both those problems with a twisted architecture.


>And also it gets rid of the security with setgid().
>Are there other security concerns ?  Do you have to setuid()
>when appending a email , for example?

Nope. The mail is written (low level) as one user (whoever is running
the JVM hosting the IMAP server object). The IMAP level user stuff is
applied within the server using the registry of user's stored in the
mailstore (note the registry might also be abstracted so this could
become LDAP or something... but I don't see an immediate need for
that).


>> Yes. I understand that. But there's no reason that 
>> you can't have delivery via IMAP. That's what append 
>> is for. The delivery agent just has to have the rights to 
>> do the delivery.

>I think your saying mail deliver to the local machine, 
>I was refering to general email that I want to deliver 
>to address@hidden where xxx.yy may have MX records, 
>or may not be available so the sender will have to queue  
>etc ...

As you say: I'm talking about delivery from the MTA to the mailstore.
Delivery to the host should always be via SMTP, any other way would be
crazy. 

But how to get from the SMTP server to the mailstore? Normally this
is where the locking problems creeps in because the SMTP server just
"delivers" the message - that is it actually writes the message data
to the user's mailbox using file i/o.

I'm putting the IMAP protocol between the SMTP MTA and the
mailstore.


>filtering, search, spam control capabilities etc .. are they 
>also part of this ?

search I haven't thought about. 

Spam control should be a combination of MTA (eg: exim has some good
spam handling) and user filters.

Filters are of course part of this. Since we have a structured
delivery method we can have easily applied filters. Filters again, are
stored in the user's folders so delivery/filtering can occur
recursively.

ie:
- append to INBOX fires "append" filter which causes delivery target
to be changed to "myfolder"
- append to "myfolder" fires "myfolder"s append filter which...
etc...


>But I need some hard evidence.  Example, a pure java 
>database on an industrial enviroment etc ...
>JVM technology came  along way and are getting better 
>but compare to a well crafted/optimise C program ... I would 
>like to see more numbers.

I'm not aware of any pure Java SQL engines. Java may not be well
suited to SQL implementation... I've not really thought about it.
Object databases are of course popular. I've never worked with one
though... I prefer things on a lower level.

I appreciate you have concerns about Java... the point is that it's
easier to write a well crafed Java program than a well crafted C
program. It takes me less time anyway.

I don't think there's a big enough issue about speed to stop one
taking advantage of the increase in development efficiency. But that's
just a feeling really.

The IMAP server doesn't really need to be written in Java, and maybe
with C I could make it quicker... but I can just compile the end
result with GCJ and I've got very fast code so why should I use C?


>Threads are not the panacea.  If one is not carefull 
>using threads it may actually __slow__ things down, 
>because of the synchronisation needed to protect the 
>shared resources and predicates.  On a single CPU a 
>mutex may be implemented with a compare_exchange 
>instruction that blocks the bus.
>Its cool when you have 2 CPUs and more and your 
>working threads can be scheduled on a CPU each an 
>run happy.  It's great if you have lot of IOs and you could 
>use the idle CPU meanwhile.  It's another story if you have
>CPU bound calculations and spending time sync the threads.

Except with multiple CPUs you still have the problem of shared
state.

I know what you're saying but Java helps here too. Java is really
just a layer... if I wanted world class performance from a particular
piece of hardware I would reimplement the Java thread layer to benefit
my application based on what the hardware could offer me.

Perhaps a green thread system is best for machine X, perhaps a thread
system based on OS processes.... because Java is just an abstract
layer you can choose.


But these are religous arguments really... I'm not trying to persuade
you to change development of the GNU-IMAP server. It couldn't be done
in Java because it needs to do things in a certain way.

I'm aiming at a totally different market. I'm aiming at large mail
sites trying to turn them into clusters of IMAP servers.


Nic

PS why do you keep saying "Ack!"? Is this because you don't like the
detail?