bug-mailutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [bug-mailutils] iterating through large mailbox: memory


From: Alain Magloire
Subject: Re: [bug-mailutils] iterating through large mailbox: memory
Date: Tue, 15 Mar 2005 14:17:26 -0500 (EST)

> 

Sorry for the delay

> Thank you for the explanation, Alain!  This is helpful.  (I just now
> saw your reply, sorry.  For some reason I didn't get a daily digest
> after my posting.)
> 
> > (2) and a certain number of headers are save/allocated for fast processing 
> > so
> > when you ask for the Subject/From/Date etc ... the lib will not go back to 
> > the
> > disk .. to slow.
> 
> I will try your code changes to reduce the memory consumption from this.
> 

Cool, if you do let us know how it went, maybe we can improve things
base on your experience.

> 
> > (1) each message is allocated a structure that will hold the offsets of the 
> > message
> 
> Okay, this makes sense.
> 
> However, for some applications, I'd like to make one forward pass
> through the mailbox, message by message, and I don't need to back up. 
> I would get what I need from each message, then call the 'next
> message' method of the iterator.  I'd be willing to accept that I lose
> the caching that is normally done.
> 

I see what you mean. The code was not designed this way,
any access will trigger a parsing of the entire mailbox.
Maybe we can have a flag like LAZY_HEADER_CACHING so
the no caching will be done on the header.
But you have to realize we still need to parse the mailbox, i.e
if you do:
        mailbox_get_message(..., 45 000, ..);
Even if you know that there is 45 000 messages and it probably
simpler to start at the end 8) ... Anyway I'm not sure on
how to do all those optimisation without the code being bloated
or hindering the usage for other clients with no such requirements.

> It's similar to a SAX style of parsing for XML.
> 
> I'd be happy to test our your new code whenever it is ready.
> 

Not for a while, limited time ... and I need to bounce some
ideas on Sergey/Jordy/Jeff/Wojciech ...  heads.

But you comments brought to light restrictions and inflexibility
in the library.






reply via email to

[Prev in Thread] Current Thread [Next in Thread]