Documentation

[Neil Jerram]

Many people have commented following my proposal for a standard format
for docstrings in Scheme files.  Thanks to everyone for that.

This email is an attempt to define a Scheme-level mechanism for
documentation that meets everyone's concerns.  I'll present the
mechanism first and then talk about how it relates to snarfing and the
various points that have been raised.

  - procedure with setter: documentation-string OBJ
      Return or set OBJ's docstring.

      (`documentation-string' is implemented as an object property
      using `make-object-property'.)

  - procedure with setter: meta-information OBJ
      Return or set OBJ's meta-information.  The format of the
      meta-information is a list with alternating keywords and values,
      e.g. (#:author "Ford K. Truman" #:compliance 'r5rs).

      (`meta-information' is implemented as an object property using
      `make-object-property'.)

  - procedure: document! OBJ DOCSTRING . META-INFORMATION
      Convenience procedure for setting OBJ's docstring and
      meta-information in a single expression.

      For example:

        (document! documentation-string "\
        Return or set OBJ's docstring."
        #:author "Ford K. Truman")

  - module option: #:documentation-filter
      This module option is used to specify the procedure that should
      be used to display documentation for objects defined in this
      module.  The documentation filter procedure is called with two
      arguments -- the object whose documentation is to be displayed,
      and an output port -- and should display the object's
      documentation to the specified port with no leading or trailing
      blank lines.

      The default documentation filter displays the specified object's
      docstring, with no additional processing.

  The specification of the `help' macro is unchanged from the current
  spec, but its implementation needs to be modified so as to use the
  appropriate documentation filter for each object whose documentation
  help displays.

  Note that `documentation-string' replaces all the existing
  documentation mechanisms, namely:
    (procedure-documentation proc)
    (procedure-property proc 'documentation)
    (object-property object 'documentation)

      
So, how does this relate to the various points that have been raised?

- Use of comments vs. a string as the first line of a lambda body.

  Personally, I don't like the ambiguity involved in using the first
  line of a lambda body as a docstring.  However, with the mechanism
  above, it is possible to define a syntax which permits this and
  makes the heuristics involved explicit:

    (define-syntax define/documented
      (lambda (x)
        (syntax-case x ()
          ((_ (name formals ...) docstring body1 body ...)
           (string? docstring)
           (syntax (begin
                      (define (name formals ...) body1 body ...)
                      (document! name docstring)))))))

  Similarly for `lambda/documented'.

- Associate docstrings with values rather than with names.

  The mechanism above does this.  Note, however, that it is sometimes
  the right thing to document a name rather than a value, e.g. to
  document an exported variable whose value can be set by the module
  user to change some aspect of the module's operation.  This is
  achieved by documenting a symbol value:

    (document! 'case-fold-search "...")

- Use gettext to support i18n rather than a home grown system.
- Use of markup language, both Texinfo and others.

  These decisions are delegated to the documentation filter procedure
  that a particular module uses.  This gives a lot of flexibility; two
  examples are (i) deciding to tailor the format of the documentation
  output according to the type of output port (HTML output port???);
  (ii) ability to process semantic markup like Emacs' \\[command-name]
  as well as display markup.

  Note that, in the particular case of Texinfo, the `makeinfo' step,
  that is currently done as part of building Guile, could instead be
  done at runtime by a documentation filter.

- Docstring visibility, interaction with module system etc.

  This falls out naturally as per the existing `help' implementation.

- How does snarfing as a concept relate to file loading and operations
  like `C-M-x'?

  Given the above mechanism, the snarfing of docstring comments in
  Scheme files can be seen as part of the documentation filter
  processing, or as a step that is performed ahead of time in order to
  facilitate the documentation filter processing.

  One possibility which seems quite nice to me is that the output of
  the snarfing operation is a Scheme file consisting of a sequence of
  `(document! ...)' expressions.  In this case, the documentation
  filter code would simply load this file in the context of the
  corresponding module.

  As regards `C-M-x': on a `define/documented' expression, it updates
  both the definition and the associated documentation; on a
  `document!' expression, it updates the documentation associated with
  the specified value; etc., as you would expect.

  Re-snarfing a file's docstring comments requires a cache-discard
  mechanism which is the responsibility of the documentation filter to
  support.  `load' per se should not do anything special here;
  however, one could imagine a load hook procedure that would cause
  a module's documentation filter to discard its cache.

- How to snarf/support docstrings that appear in the expansion of
  macro calls?

  Snarfing - no support.  Macro definitions can include
  `define/documented' and such like.

- Use of Guile to write snarfing tools.

  No issue here, and I think it's a good idea.

- Do we aim to move towards literate programming and/or automatic
  reference manual generation?

  I think this is beyond the scope of this proposal.  Certainly it's
  conceivable that the documentation filter for a particular module
  could share common code with a program designed to produce a manual
  from the module comments.


Looking forward to your comments!  Best regards,

        Neil