[bug-mailutils] Mailutils 1.9.90

[Sergey Poznyakoff]

Hello,

I am pleased to inform you that GNU Mailutils version 1.9.90 is available
for testing from ftp://alpha.gnu.org/gnu/mailutils.

Since the last stable version the package has undergone a number of major
changes, which justify the bump in minor (and the planned increase in major)
version number. The rest of this message describes these changes in detail.

Due to the excessive amount of changes I had to sacrifice documentation
in favor of developing and testing. The mailutils.texi document in its
current state should be considered obsolete and inaccurate. I will try
to fix this as soon as my time and load will permit. In the meantime,
this message aims to provide the necessary (albeit rather terse)
descriptions and hints.  

Should you need any clarifications or more accurate descriptions, feel
free to ask me.

As of today, this version of GNU Mailutils is runing on several
production servers, mostly under GNU/Linux and FreeBSD. Any efforts to
test it on another platforms will be greatly appreciated.

The rest of this message concerns new features and user-visible changes in
GNU Mailutils 1.9.90. To facilitate reading it is formatted to fit Emacs
outline-mode.

* New configuration file format.

** Configuration files.

Each Mailutils utility (excepting MH suite and mail) will look for its
configuration in the following places:

1. Site-wide configuration file $sysconfdir/mailutils.rc, where $sysconfdir
stands for the system configuration directory, as set on configuration time
(default is /usr/local/etc). 

2. If the above file contains an `include' statement (see below) with a
directory name as its argument, the file <progname> in this directory,
where <progname> stands for the utility name with all directory components
removed. For example, if mailutils.rc contains

 include /etc/mailutils.d;

and /etc/mailutils.d is a directory, then pop3d will look for
/etc/mailutils.d/pop3d. 

3. Per-user configuration file: ~/.mailutils/<progname>, where <progname>
is the same as in (2).

4. An additional configuration file specified by the command line option
`--rcfile'.

** Configuration file format.

Empty lines are ignored. Comments are introduced by any of the following:

1. Any characters after an `#' or '//' until the next newline are
ignored.
2. Any characters between '/*' and '*/' are ignored.

Anything that is not an empty line or a comment must be a valid configuration
statements. A configuration statement can occupy one or several lines and must
be terminated with a semicolon.

A _simple_ configuration statement consists of a keyword followed by a value.
Examples of valid simple configuration statements:

# A numeric value:
max-children 69;

# A string value. If it consists of alphanumeric characters and punctuation
# it is not necessary to quote it:
pidfile /var/run/pop3d.pid;

# A string value split over several lines (as in C):
auth "User-Name = ${user}, User-Password = ${passwd}, "
     "Service-Type = Authenticate-Only";

A _block_ configuration statement consists of a keyword, optionally followed
by a block tag, followed by a list of configuration statements, enclosed in
curly braces. An example of a valid block configuration statement:

logging {
  facility local1;
  tag "GNU pop3d";
};

** The `program' statement.

The `program' statement is a block statement that introduces configuration
statements specific for a certain utility. It has the following form:

program <progname> {
  <list...>
};

Where <progname> is the utility name with all directory components
removed, and <list...> is a list of configuration statements to apply to that
program. The `program' statement can be used only in the site-wide
configuration file. An example of a `program' statement:

program pop3d {
  logging {
    facility local3;
  };
};

** The `include' statement.

include <file-or-directory>;

If <file-or-directory> is a regular file, its contents will be read and
processed in place. Otherwise, if it is a directory, the file <progname>
will be looked up in that directory and read in, if the file exists. The
`include' statement with a file as its argument is allowed in any
configuration file. The form with a directory as its argument is allowed
only in the site-wide configuration file.

** Per-utility documentation.

Each GNU Mailutils utility is self-documented. Run it with the command
line option `--config-help' to get an annotated list of configuration
statements valid for this utility.

** Related command-line options.

 --no-site-rcfile       Do not load site-wide configuration file.
 --no-user-rcfile       Do not load per-user configuration file.
 --rcfile=FILE          Load this configuration file.
 --rcfile-lint          Check configuration file syntax and exit.
 --rcfile-verbose       Verbosely log parsing of the configuration files.

* Programs

** Debugging and online help

Each Mailutils utility understands two additional command line
options:

 --debug-level=LEVEL    Set Mailutils debugging level.
 --debug-line-info      Show source info with debugging messages.

(see also `** Global debugging and verbosity settings.', below)

** URL parameters.

Additional mailbox URL parameters `type', `user' and `param' can
appear in any local URLs. For example, the following configuration
statement:

mailbox {
  mailbox-pattern "maildir:/usr/local/var/mail;type=index;param=2;"
                  "user=${user}"; 
};

sets the default mailbox type to `maildir' and tells that the user's
mailbox is located in a two-level indexed directory hierarchy. The
keyword ${user} will be replaced by the user login name.

** MAIL and FOLDER environment variables.

These variables are consulted only if mail-spool directory is not
explicitely set either in the configuration files or in the command
line. This is different from the previous versions.

** New utility `maidag'

Maidag is a MAIl Delivery AGent. It is a general-purpose MDA able to
run in both traditional and LMTP mode and to deliver mails to various
mailbox formats.  These formats, among others, include remote+smtp and
remote+sendmail (see `New mailbox types.', below), which are
equivalent to forwarding a message over SMTP.  Thus, maidag supercedes
both `mail.local' and `mail.remote' (which are, however, still included
for backward compatibility).  Maidag is also able to process incoming
messages using Sieve or Scheme scripts and, based on results of this
processing, to take a decision on whether to actually deliver and
where to deliver them.

Users of mail.local and mail.remote are advised to switch to maidag,
because it offers greater flexiblity and is much more reliable.

** Comsat

The `comsat' utility is able to notify about messages delivered to
a mailbox of any `local' type, i.e.: UNIX mailbox, MH and Maildir.
This is implemented using quick access mode.  See `Quick access to a
message', below.

The communication protocol has been modified to make this possible.
The traditional comsat protocol is supported as well.

** New Sieve action `pipe'

Syntax: pipe [:envelope] <command line: string> 

This action executes the given <command line> and pipes the message to
its standard input. If the :envelope tag is given, the envelope of the
message is piped as well.

** Client SMTP STARTTLS support

** Support for new protocols: POPS (pops://) and IMAPS (imaps://),

** LDAP support (authentication and authorization).

** Support for TCP wrappers.

The support for TCP wrappers is added to the daemon programs (imap4d,
pop3d, maidag). The support is controlled at compile time by the
--with-tcpwrappers command line options to configure.

By default, it is enabled if libwrap presence is detected. 

A set of configuration file statements is provided for fine tuning TCP
wrappers at run-time:

tcp-wrappers {
  # Enable TCP wrapper access control.  Default is "yes".
  enable <arg: boolean>;
  # Set daemon name for TCP wrapper lookups.  Default is program name.
  daemon <name: string>;
  # Use this file for positive client address access control (default:
  # /etc/hosts.allow).
  allow-table <file: string>;
  # Use this file for negative client address access control (default:
  # /etc/hosts.deny).
  deny-table <file: string>;
  # Log host allows at this syslog level.  See logging { facility } for a
  # description of argument syntax.
  allow-syslog-level <level: string>;
  # Log host denies at this syslog level.  See logging { facility } for a
  # description of argument syntax.
  deny-syslog-level <level: string>;
};

For example, the following statement disables TCP wrappers:

tcp-wrappers {
 enable no;
}; 

See also ACL, below.

** pop3d and imap4d ports.

Pop3d and imap4d can be configured to listen on several different
IP addresses/ports (or even local sockets) simultaneously. See
`pop3d --config-help', for details.

** pop3d: Fixed APOP handling.

** imap4d supports PREAUTH mode.

Three mechanisms are provided for authentifying the connection in
PREAUTH mode:

 1. stdio - PREAUTH mode is enabled automatically if imap4d is started
    from command line in interactive mode (-i command line
    option). The current login name is used as the user name.

 2. ident - The remote machine is asked about the requester identity
    using the identification protocol (RFC 1413). Both plaintext and
    DES encrypted replies are understood.

 3. prog - Imap4d invokes an external program to authenticate the
    connection. The command line is obtained from the supplied string,
    by expandind the following variables:

      1) ${client_address} - Remote IP address in dotted-quad notation;
      2) ${client_port}    - Remote port number;
      3) ${server_address} - Local IP address;
      4) ${server_port}    - Local port number.

    If the connection is authenticated, the program should print the
    user name, followed by a newline character, on its standard
    output and exit with code 0.

    Otherwise, it shoud exit with a non-zero exit code.
    
* Libraries

** Diagnostic and debugging functions essentially rewritten.

A set of debugging macros, MU_DEBUG0 through MU_DEBUG11, is provided.
New functions mu_debug_printf and mu_debug_vprintf allow for flexible
formatting of debugging messages. The mu_debug_t object features
built-in line bufferring.

The new function mu_debug_check_level(d,l) returns true if the logging
level currently set for debugging object `d' matches that given by
`l'.

New diagnostics functions are available, among them:

  void mu_diag_vprintf (int level, const char *fmt, va_list ap);
  void mu_diag_printf (int level, const char *fmt, ...);
  void mu_diag_voutput (int level, const char *fmt, va_list ap);
  void mu_diag_output (int level, const char *fmt, ...);

A new header file, mailutils/diag.h, declarations of these and other
related functions.
  
The functions mu_error and mu_verror are shortcuts for
mu_diag_output (MU_DIAG_ERROR, ...)  and mu_diag_voutput (MU_DIAG_ERROR, ...),
correspondingly.

The function mu_error_set_print is deprecated as well as the
mu_error_pfn_t type. They are, however, still supported. The following
approach is recommended to use instead of mu_error_set_print:

      mu_diag_get_debug (&debug);
      mu_debug_set_print (debug, new_printer, NULL);

** Header iteration

New function `mu_header_get_iterator' returns an iterator for this
header object. Thus, the following code loops over all headers:

      mu_iterator_t itr;
      mu_header_get_iterator (header, &itr);
      for (mu_iterator_first (itr); !mu_iterator_is_done (itr);
           mu_iterator_next (itr))
        {
          const char *hdr, *val;
          mu_iterator_current_kv (itr,
                                  (const void**)&hdr,
                                  (void**)&val));
          /* Do something with hdr and val */                     
        }
      mu_iterator_destroy (&itr);

** Global debugging and verbosity settings.

These settings provide default values for mu_debug_t objects created
by various library objects. The following functions are provided for
dealing with global debugging level:

   unsigned mu_global_debug_level (const char *object_name);
   int mu_global_debug_set_level (const char *object_name, unsigned level);
   int mu_global_debug_clear_level (const char *object_name);

** New function mu_mailbox_sync

It supercedes mu_mailbox_save_attributes, which is now considered
deprecated.

** Observable event handling

Each event type is associated with an event-specific data
pointer. This pointer is passed to event handling functions along with
an opaque function-specific data pointer. This affects the following
functions:

  mu_observer_set_action
  mu_observer_set_action_data (New function)
  mu_observable_notify

New type of event, MU_EVT_MESSAGE_APPEND, is signalled whenever a new
message is appended to the mailbox.

** Quick access to a message

A set of functions is provided for so-called `quick access' to mail
messages.

Quick access is used when an application needs to quickly read one
message from the mailbox. This mode is aimed to avoid any additional
computing overhead, such as scanning the mailbox, etc. In particular,
this mode is used by comsat. 

To use quick access functions, the mailbox must be opened with flags
MU_STREAM_READ|MU_STREAM_QACCESS.  The function
mu_mailbox_quick_get_message can then be used to retrieve the message:

   int mu_mailbox_quick_get_message (mu_mailbox_t mbox,
                                   mu_message_qid_t qid,
                                   mu_message_t *pmsg);

The message is identified by the second argument, qid.

The function mu_message_get_qid can be used to retrieve
mu_message_qid_t identifier for a message:

   int mu_message_get_qid (mu_message_t msg, mu_message_qid_t *pqid);

** New `aget' and `sget' accessors for mu_url_t

The following new accessors are provided:

extern int mu_url_sget_scheme  (const mu_url_t, const char **);
extern int mu_url_aget_scheme  (const mu_url_t, char **);  
  
extern int mu_url_sget_user  (const mu_url_t, const char **);
extern int mu_url_aget_user  (const mu_url_t, char **);  

extern int mu_url_sget_passwd  (const mu_url_t, const char **);
extern int mu_url_aget_passwd  (const mu_url_t, char **);  
  
extern int mu_url_sget_auth  (const mu_url_t, const char **);
extern int mu_url_aget_auth  (const mu_url_t, char **);  

extern int mu_url_sget_host  (const mu_url_t, const char **);
extern int mu_url_aget_host  (const mu_url_t, char **);  
  
extern int mu_url_sget_path  (const mu_url_t, const char **);
extern int mu_url_aget_path  (const mu_url_t, char **);  

extern int mu_url_sget_query  (const mu_url_t, const char **);
extern int mu_url_aget_query  (const mu_url_t, char **);  

int mu_url_sget_fvpairs (const mu_url_t, size_t *, char ***);
int mu_url_aget_fvpairs (const mu_url_t, size_t *, char ***);

** ACL

A set of functions implements general-purpose access control lists.
They are declared in the header file mailutils/acl.h.  Currently,
these functions support IPv4 and UNIX address families.  Support for
more address families, in particular IPv6, will be added in future.
The following actions can be defined in ACLs:

 1. mu_acl_accept       - Accept connection.
 2. mu_acl_deny         - Deny connection.
 3. mu_acl_log          - Issue a log message, using mu_diag.*
                          diagnostics functions.
 4. mu_acl_exec         - Execute an external program. 
 5. mu_acl_ifexec       - Execute an external program and accept or
                          deny connection depending on its exit code.

An example of `acl' configuration statement:

acl {
  # Provide a log message, compatible with previous MU versions:
  # '${address}' will be replaced by the actual IP address in dotted-quad
  # form.
  log from any "Connect from ${address}";
  allow from 213.130.0.0/19;
  deny from any;
};

** KWD

A set of functions is provided for translating string values to
integer tokens and vice-versa using simple translation tables.
These functions are declared in the header file mailutils/kwd.h.

** Vartab

A set of functions is provided for performing variable expansion in
strings. These functions are declared in the header file
mailutils/vartab.h. Variable references have the form `${name}' for
multi-letter variables, and $n for single-letter ones. For compatibility
with previous versions, the single-letter form %n is also supported.
          
** Incompatible change in mu_url_get_.* return value

Any mu_url_get_.* accessors return MU_ERR_ENOENT if the corresponding
field is not present in the object. Previous versions of GNU Mailutils
returned 0 and stored empty string in the output buffer.

** New mailbox types.

Two new mailbox types, "remote+smtp" and "remote+sendmail", are
supported. These are `append-only' mailboxes, i.e. the only operation
that can be done over them is mu_mailbox_append_message.  E.g.,
appending to the URL `remote+smtp://127.0.0.1:24' is equivalent to
sending a message using mailer `smtp://127.0.0.1:24'.

** New argcv functions.

-  int mu_argcv_get_np (const char *command, int len,
                        const char *delim, const char *cmnt,
                        int flags,
                        int *pargc, char ***pargv, char **endp);

This function is an alternative entry point to
mu_argcv_get/mu_argcv_get_n functions. The resulting argv will contain
non-whitespace delimiters only if flags contains the bit
MU_ARGCV_RETURN_DELIMS.

-  void mu_argcv_remove (int *pargc, char ***pargv,
                         int (*sel) (const char *, void *), void *);

Removes from pargc/pargv all elements for which the sel function
returns true.
                         
** New registry functions.

- int mu_registrar_lookup_url (mu_url_t url, int flags,
                               mu_record_t *precord, int *pflags);
                             
** Parsing of URLs similar to file:///a/b. 

It is parsed as an absolute file name `/a/b'.

Previous versions incorrectly understood such an URL as `a/b'
(relative file name). 

** Remove v0.6 compatibility layer.

** Function mu_mail_directory is removed.

** New function mu_mailbox_url.

Regards,
Sergey