[bug-mailutils] email fixed, and IMAP fixed?

[Gary Potwin]

Hi, Sergey.  Thank you for your response.  I'm sorry that I have not
replied in so long, but I got buried in work again, and it took time
to figure out what is below.

Item 1 {

> OK. To begin with, please ensure that your mails have a valid email
> address in the `From:' header. The address <address@hidden> is not
> valid and cannot be replied to. That was the reason my mail from
> August 13 got bounced back to me.

I'm sorry about the above error.  I had to recompile nmh, and study the
source code for Exim version 3.36 to put in the correct line in the
exim configuration file, to fix the error.  It should be coming through
OK now with "From: address@hidden".

}

Item 2 {

> For any questions related to GNU Mailutils, I'd suggest to use
> <address@hidden>. It is read by many people, so that
> somebody among them may help you in case I am absent.
>
> It would also be advisable to subscribe to that list.

I am now using the above address.  I did review the above list for
June 2009 through the present, and I did not see the below items
addressed.

}

Item 3 {

>> 1. I needed to see what was being communicated between the IMAP server and
>> the mail program.  I couldn't figure out how to set "imap4d_transcript" from
>> the command line,
>
> It is set in imap4d configuration file, in section "server", like this;
>
> server 0.0.0.0:143 {
>   transcript yes;
> }

I tried putting exactly what you have above into the file

"/usr/local/etc/mailutils.rc"

and ran the following command

imap4d --config-lint --config-verbose

which gave the following output

imap4d: Info: parsing file `/usr/local/etc/mailutils.rc'
# 1 "/usr/local/etc/mailutils.rc"
server 0.0.0.0:143 {
# 2 "/usr/local/etc/mailutils.rc"
  transcript yes;
};
/usr/local/etc/mailutils.rc:1: unknown section `server'
imap4d: Info: finished parsing file `/usr/local/etc/mailutils.rc'

So then I tried the following in "/usr/local/etc/mailutils.rc"

program imap4d server 0.0.0.0:143 {
  transcript yes;
}

and again ran the following command

imap4d --config-lint --config-verbose

which gave the following output

imap4d: Info: parsing file `/usr/local/etc/mailutils.rc'
# 1 "/usr/local/etc/mailutils.rc"
program imap4d server 0.0.0.0:143 {
# 2 "/usr/local/etc/mailutils.rc"
  transcript yes;
};
/usr/local/etc/mailutils.rc:2: unknown keyword `transcript'
imap4d: Info: finished parsing file `/usr/local/etc/mailutils.rc'

I understood the documentation to say that this method of turning on
the transcript won't work when invoking imap4d from inetd, which is
exactly what I have to do.  So I went back to what I described to you
previously

>> 1. I needed to see what was being communicated between the IMAP server and
>> the mail program.  I couldn't figure out how to set "imap4d_transcript" from
>> the command line, so, in imap4d/util.c, I commented out four
>> "if (imap4d_transcript)" statements so that their respective "mu_diag_output"
>> statements would always run, and I set "transcript = 1".  Now I can see the
>> communications.

and I found that this works fine in spite of the fact that I am invoking
imap4d from inetd.  So is there a way to turn on the transcript from
the configuration file or from the command line, and have it work under inetd?

}

Item 4 {

>>> 2. The problem I am having with the directories graying out may be due to my
>>> inexperience at this, since I have never before attempted to set up an IMAP
>>> server.

>> I don't quite understand what exactly "graying" means. Is it some way to
>> represent folder attributes? If so, what attributes? 

> As I was getting ready to resend
> the email, I realized that I had left out the following important information:
> for each grayed-out directory that I refer to below, the IMAP server is
> sending the "Noselect" switch along with it, which I suspect is why the mail
> program displays them as grayed-out.

There is a typo in the above: "Noselect" should really have been "NoSelect".
Under both Microsoft Windows and the X System under Linux, icons are
displayed which will do something if you click (or double-click) on them.
Depending on the circumstances at the time, the icons may be normally
displayed, which will result in the above action occurring, or the icons
may be displayed dimly or in gray, which is what I meant by "graying out"
above, in which case you will not be able to click or double-click on them
and have the usual action happen.  As best I understand so far, this has
nothing to do with folder attributes.  It turns out that my suspicion above
regarding the "NoSelect" switch was correct.  Please see Item 6 below for
what I did about it.

}

Item 5 {

>> Would you kindly tell me what directory structure is normally
>> expected by an IMAP server,

> There are no restrictions on mailbox and directory structure. Any
> directory hierarchy is understood.

>> source code package), and some of them refer to a namespace that would start
>> with "mh:" or "#mh" in my case.  Should I put one of these in as a namespace
>> parameter in the mail program?

> No, the "namespace" directive has nothing to do with the mailbox types.

The above were just my mistaken attempts to try to get the IMAP server
to recognize my MH mail folders.  I completely concur with what you stated.
Please see Item 6 below for what I did that allowed the IMAP server to
see my MH mail folders.

}

Item 6 {

>> Here is the path I have been using with MH for years:
>>
>> /home/gary/Mail
>>
>> When the email program sends this path to the IMAP server, this forces the
>> scheme to be "file",
>>
>> (in function "url_parse0" in "mailbox/url.c")
>>
>> which doesn't match any of the formats
>>
>> (mbox, mh, or maildir (from include/mailutils/registrar.h, called by
>> "mu_register_local_mbox_formats" from imap4d/imap4d.c))
>>
>> checked by "mu_record_is_scheme" in "mailbox/registrar.c",
>>
>> (I really don't yet understand how the following code works:
>>
>> if (record->_is_scheme)
>>   return record->_is_scheme (record, url, flags);
>>
>> (The only definition I could find for "_is_scheme" is in "struct _mu_record"
>> in "include/mailutils/registrar.h", and I don't know what "Stub functions to
>> override. The default is to return the fields." means.)

> The `_is_scheme' is a "virtual method" which is defined in the library
> responsible for the particular mailbox implementation (e.g. in
> libmu_maildir, for mailboxes in maildir format).

>>
>> Using checking statements (mu_diag_output), I found that the if statement
>> following the above one is never accessed.)
>>
>> called from "mu_registrar_lookup_url" in "mailbox/registrar.c",
>> called from "mu_registrar_lookup" in "mailbox/registrar.c",
>> called from "mu_folder_create_from_record" in "mailbox/folder.c",
>> called from "mu_folder_create" in "mailbox/folder.c",
>> called from "imap4d_list" in "imap4d/list.c".

I believe that I was correct as far as I had gone, but I thank you for your
explanation of "virtual method" above, which enabled me to continue the
investigation.  The above code is set up to check the URL against the
3 records that were set up in "mu_register_local_mbox_formats":
first, mbox, and then, mh and maildir.  My present understanding of the
mbox system is that mbox mail is always stored in one or more mbox format
files.  To the best of my knowledge (man 5 mbox), there is no such thing
as an mbox format subdirectory or folder.  The code that checks for mbox
format is in libproto/mbox/folder.c.  When this code is called to check
what really is an mh format subdirectory, lines 93-95 incorrectly identify
it as an mbox subdirectory.  When the mbox code later on tries to deal with
this subdirectory, it realizes that this is not a valid mbox file, even
though it is a subdirectory, so that's why the "NoSelect" switch gets sent
out for this subdirectory.  What I did was to stop lines 93-95 from being
compiled by adding

#if 0

just before line 93, and adding

#endif

just after line 95.  This forces the mbox URL check to fail for any folder
or subdirectory, thereby allowing the next record to run, which allows the
mh record to correctly identify this as an mh subdirectory.  With the above
change, imap4d did correctly see all of my mh mail, and I am now able to
access this mail using both Linux and Microsoft Windows email programs,
which was my objective from the beginning.

Did I make any mistakes in any of the above?  In stopping lines 93-95
above from being compiled, have I made imap4d work for my purposes, while
introducing some other problem that I simply have not stumbled onto yet?
If the above really is correct, perhaps this really is a bug, and perhaps
the above should be incorporated into a bugfix and/or the next release.

}

Thank you, Sergey, for all your help, and for your patience with my
ignorance and lack of experience.

Looking forward to your response,

Gary

------------------------------------------------------------------------------

Date: Tue, 18 Aug 2009 22:57:15 +0300
From: Sergey Poznyakoff <address@hidden>
To: "Gary Potwin" <address@hidden>
Subject: Re: [bug-mailutils] is Sergey Poznyakoff OK? 

Hi Gary, 

> Hi, Sergey.  Thank you for your response.  I would like to clear up, as best
> we can, these email problems, and then, by all means, I would like to
> get back to the IMAP server questions.

OK. To begin with, please ensure that your mails have a valid email
address in the `From:' header. The address <address@hidden> is not
valid and cannot be replied to. That was the reason my mail from
August 13 got bounced back to me.

> I noted (see headers below) that you responded
> to the message that I sent to address@hidden on August 17.  Did you
> ever receive the previous, nearly identical, message that I sent to 
> address@hidden on August 13?

Yes, I did.

> Please let me know if my emails are getting
> through, and which email address you would prefer that I use.

For any questions related to GNU Mailutils, I'd suggest to use
<address@hidden>. It is read by many people, so that
somebody among them may help you in case I am absent.

It would also be advisable to subscribe to that list.

Regards,
Sergey

------------------------------------------------------------------------------
Date: Tue, 18 Aug 2009 09:40:45 +0300
From: Sergey Poznyakoff <address@hidden>
To: <address@hidden>
Subject: Re: [bug-mailutils] is Sergey Poznyakoff OK? 
In-reply-to: Your message of Mon, 17 Aug 2009 09:36:29 -0400
         <address@hidden>
References: <address@hidden>

Hi Gary,

> 1. I needed to see what was being communicated between the IMAP server and
> the mail program.  I couldn't figure out how to set "imap4d_transcript" from
> the command line,

It is set in imap4d configuration file, in section "server", like this;

server 0.0.0.0:143 {
  transcript yes;
}

> 2. The problem I am having with the directories graying out may be due to my
> inexperience at this, since I have never before attempted to set up an IMAP
> server.

I don't quite understand what exactly "graying" means. Is it some way to
represent folder attributes? If so, what attributes? 

> Would you kindly tell me what directory structure is normally
> expected by an IMAP server,

There are no restrictions on mailbox and directory structure. Any
directory hierarchy is understood.

> (I really don't yet understand how the following code works:
> 
> if (record->_is_scheme)
>   return record->_is_scheme (record, url, flags);

The `_is_scheme' is a "virtual method" which is defined in the library
responsible for the particular mailbox implementation (e.g. in
libmu_maildir, for mailboxes in maildir format).

> source code package), and some of them refer to a namespace that would start
> with "mh:" or "#mh" in my case.  Should I put one of these in as a namespace
> parameter in the mail program?

No, the "namespace" directive has nothing to do with the mailbox types.

Regards,
Sergey

-------------------------------------------------------------------------------

To: Sergey Poznyakoff <address@hidden>
Subject: Re: [bug-mailutils] is Sergey Poznyakoff OK? 
In-reply-to: <address@hidden> 
References: <address@hidden> <address@hidden>
Comments: In-reply-to Sergey Poznyakoff <address@hidden>
   message dated "Thu, 13 Aug 2009 11:27:48 +0300."
Date: Thu, 13 Aug 2009 21:19:39 -0400
From: Gary Potwin <address@hidden>

Hi, Sergey.  It is very good to hear from you again.  I'm very glad you haven't
been run over by a truck, or some other bad thing.  A colleague of yours from
GNU Mailutils sent me the following email just before you sent your email.
Between the == lines is my response, and what he sent.

==============================================================================

Hello, Alfred M. Szmidt.  Thank you for your reply, and for helping forward
the message to Sergey Poznyakoff.  He responded shortly after you did.

Sincerely yours,

Gary Potwin

-----------------------------------------------------------------------------

He is just busy.

Author: Sergey Poznyakoff <address@hidden>
Date:   Thu Aug 13 00:08:33 2009 +0300

    Fixes in configure.ac

===============================================================================

>From the looks of it, the last email that I sent to you at
address@hidden somehow had most of the latter part deleted, because
that is where I had included the two emails that I had sent to you before.
As I did previously, those emails are included below.  Those emails explain
where I am so far with the IMAP server problems.  I look forward to your
response.

Thank you,

Gary

------------------------------------------------------------------------------

Hi Gary,

> I have been trading emails with Sergey Poznyakoff for several months 
> concerning
> the GNU Mailutils IMAP server.  As can be seen below, he usually responds very
> quickly, but I have not had any response from him since I sent him one email 
> on
> July 6, 2009, and another on July 20, 2009, which is why I am trying this 
> other
> email address that he informed me of in a previous email.

Strange, I did not receive those two postings. Could you please forward
them again?

> Has anything bad happened to him?

I'm OK, thanks for bothering about me. I really appreciate that. The
reason why our communication broke is purely technical, and I apologize
for it. I'll investigate my mail system to prevent such things from
happening again.

So, what is your progress with Mailutils?

Regards,
Sergey

------------------------------------------------------------------------------


To: address@hidden
Reply-To: address@hidden
Fcc: 
/home/gary/Maildata/Computer/Software/Linux/GNU/Mailutils/Sergey_Poznyakoff/address@hidden
Subject: is Sergey Poznyakoff OK?
--------
Hello, my name is Gary Potwin.  As you can see from the emails copied below,
I have been trading emails with Sergey Poznyakoff for several months concerning
the GNU Mailutils IMAP server.  As can be seen below, he usually responds very
quickly, but I have not had any response from him since I sent him one email on
July 6, 2009, and another on July 20, 2009, which is why I am trying this other
email address that he informed me of in a previous email.  Has anything bad
happened to him?  If he is overwhelmed with other work, I can certainly
understand that.  If he is OK, would someone please forward this email to him?
I don't know whether my emails are getting through.  Either way, would someone
please let me know if he is OK or not?

Thank you,

Gary Potwin

------------------------------------------------------------------------------

To: Sergey Poznyakoff <address@hidden>
Subject: Re: using IMAP on MH emails 
In-reply-to: <address@hidden> 
References: <address@hidden> <address@hidden> <address@hidden> <address@hidden>
Comments: In-reply-to Sergey Poznyakoff <address@hidden>
   message dated "Tue, 12 May 2009 15:56:56 +0300."
Date: Mon, 20 Jul 2009 11:24:58 -0400
From: Gary Potwin <address@hidden>


Hi, Sergey.  Below please find a copy of the email that I sent to you two
weeks ago.  Maybe the email just got lost somewhere, or maybe you were on
vacation.  Either way, I hope you are OK.  As I was getting ready to resend
the email, I realized that I had left out the following important information:
for each grayed-out directory that I refer to below, the IMAP server is
sending the "Noselect" switch along with it, which I suspect is why the mail
program displays them as grayed-out.

Looking forward to your reply,

Gary

------------------------------------------------------------------------------

To: Sergey Poznyakoff <address@hidden>
Subject: Re: using IMAP on MH emails 
In-reply-to: <address@hidden> 
References: <address@hidden> <address@hidden> <address@hidden> <address@hidden>
Comments: In-reply-to Sergey Poznyakoff <address@hidden>
   message dated "Tue, 12 May 2009 15:56:56 +0300."
Date: Mon, 06 Jul 2009 16:42:39 -0400
From: Gary Potwin <address@hidden>

Hi, Sergey.  Thank you for your reply.  I'm sorry it has been so long since I
emailed you, but I was buried in work by my job, so this had to wait.  Per
your instructions, I did download Mailutils 2.0, compiled it, and it does
"run" on my 75 MHz Pentium.  I am still having the same problems as before,
where the directories are grayed out.  Here's what I've discovered up to this
point:

1. I needed to see what was being communicated between the IMAP server and
the mail program.  I couldn't figure out how to set "imap4d_transcript" from
the command line, so, in imap4d/util.c, I commented out four
"if (imap4d_transcript)" statements so that their respective "mu_diag_output"
statements would always run, and I set "transcript = 1".  Now I can see the
communications.

2. The problem I am having with the directories graying out may be due to my
inexperience at this, since I have never before attempted to set up an IMAP
server.  Would you kindly tell me what directory structure is normally
expected by an IMAP server, and what are normally expected settings in the
email program?  Here is the path I have been using with MH for years:

/home/gary/Mail

When the email program sends this path to the IMAP server, this forces the
scheme to be "file",

(in function "url_parse0" in "mailbox/url.c")

which doesn't match any of the formats

(mbox, mh, or maildir (from include/mailutils/registrar.h, called by
"mu_register_local_mbox_formats" from imap4d/imap4d.c))

checked by "mu_record_is_scheme" in "mailbox/registrar.c",

(I really don't yet understand how the following code works:

if (record->_is_scheme)
  return record->_is_scheme (record, url, flags);

(The only definition I could find for "_is_scheme" is in "struct _mu_record"
in "include/mailutils/registrar.h", and I don't know what "Stub functions to
override. The default is to return the fields." means.)

Using checking statements (mu_diag_output), I found that the if statement
following the above one is never accessed.)

called from "mu_registrar_lookup_url" in "mailbox/registrar.c",
called from "mu_registrar_lookup" in "mailbox/registrar.c",
called from "mu_folder_create_from_record" in "mailbox/folder.c",
called from "mu_folder_create" in "mailbox/folder.c",
called from "imap4d_list" in "imap4d/list.c".

I read all of the rfc documents in doc/rfc (thank you for including them in the
source code package), and some of them refer to a namespace that would start
with "mh:" or "#mh" in my case.  Should I put one of these in as a namespace
parameter in the mail program?

As you can see, I tried to solve this on my own, but so far I can't.  If any
of the above questions are answered somewhere on the web, I would appreciate
knowing the appropriate URL.  I spent many hours on Google, but failed to find
any information like this.

Thank you for any advice you can give,

Gary

------------------------------------------------------------------------------

Hi Gary,

> I am trying to learn Linux, so I presently have Debian 3.1 installed on a
> 75 MHz Pentium with 128M of ram.  All I have done so far is to use the
> packages that are provided with this distribution, and the 0.6.1 version
> of GNU Mailutils is what is provided as a package with that distribution.
> This machine does not meet the hardware requirements of any of the more
> recent Debian distributions, which is why I haven't upgraded.  Would this
> machine be able to run Mailutils 2.0?

Yes, I believe it will.  

>If so, I'll have to investigate how install it.

The usual procedure is: downoload the tarball, then do:

tar xf mailutils-2.0.tar.gz
cd mailutils-2.0
./configure [OPTIONS]
make
make install

where [OPTIONS] stands for a list of options you pass to configure to
customize the installation.  After unpacking, you'll find in the
directory mailutils-2.0 files INSTALL and README with the detailed
instructions.

You will need the usual build tools for the above procedure
(i.e. cc, ld, make).

> I then created an IMAP account on the Mozilla Mail (version 1.7.8) that is
> also installed on the 75 MHz Pentium by means of the Debian 3.1 package.
> I am able to see the various directories of my mail, but they are all
> grayed out, and I can't subscribe to them, so I can't see any messages.
> Again, I don't know if this is due to needing to upgrade or not.  If you
> can advise me on this, I would be grateful.

I have never encountered such symptoms, but they may well be due to
obsolete version of imap4d.  Version 0.6.1 is an early alpha version,
that contained many bugs.  Try the latest one instead.  If installing
from sources does not suit you, I can try to make a binary package for
you, but that may take some time.

Regards,
Sergey