Proposed new M4 feature -- dependency rule creation

[David Warme]

Hi M4 fans.

Many of us have long wished that M4 would automatically generate
Makefile dependencies.  This missing feature has arguably been on the
TODO list for 15 years or more.

Earlier this year I submitted a patch for it.  More recently,
Lorenzo Di Gregorio independently submitted a different patch.  So the
two of us went off-list to merge our work.  In the process, we
discussed some very wide-ranging practical experience regarding
automatic generation of dependency rules, and different ways of
integrating such processes into Makefiles.

Ultimately, we decided to "reboot" -- in other words, to abandon both
our specific patches, and to start over with a purely functional
design.  That process went like this:

1. Start with a "mature" implementation of "makedep" functionality
   (we used GCC version 4.4.5).

2. Make all necessary adjustments appropriate to the context of M4.

3. Distill it all down into a concrete, written functional
   specification.

Lorenzo and I had quite a bit of back and forth iteration during steps
2 and 3.  In order to spare the M4 developer community much of the
same effort that Lorenzo and I went through, we present below brief
rationale for each of the design decisions we made while arriving at
the final functional specification, which we present last.

The purpose of this post is to solicit further input from the M4
developers to identify any "blind spots" that Lorenzo and I might have
overlooked.  Hopefully we can avoid rehashing decisions already made,
unless they involve a significant error or omission on our part.

Thank you, and sincerely,

David Warme


A brief summary of GCC's existing "makedep" functionality is as
follows:

========================================================================

gcc-4.4.5 provides the following "makedep" type features:

-M      Output dependency *instead* of compiling.

-MD     Generate dependency AND compile file in one run.

-MM     Omit system include files (and files included only indirectly
        from such system include files).

-MMD    Like -MD, but omits system includes like -MM.

-MF path
        Write dependency rules to given path instead of stdout.

-MG     Assume missing headers are generated files and add them as
        dependencies anyway.

-MP     Add a phony target for each dependency other than the main
        file (i.e., every file that is the subject of an #include
        directive).

-MT target
        Specify the dependency rule target verbatim.  Multiple "-MT
        target" options are OK, and are concatenated with separating
        spaces.

-MQ target
        Like -MT, but quotes characters that make handles specially.

========================================================================

We then identified the following simplifications / modifications that
are appropriate in the context of GNU m4:

1. GNU m4 has no well-established body of "system" m4 include files.
   There is therefore no reason to provide -MM or -MMD capability.

2. In m4, the macro expanded output always goes to stdout.  In order to
   prevent mingling of macro expanded output with dependency rule
   output, the generated dependency rules are always written to a
   separate file specified analogously to "gcc -MF file.dep".

3. GNU m4 writes all macro expanded output to stdout, so m4 never really
   knows the name of the target file being produced.  (Consider, e.g.,
   when stdout is the write end of a pipe.)  The user must therefore
   always explicitly specify the target to use in the generated rule, in
   a manner analogous to "gcc -MT target".

4. The ability to specify "-MT target" multiple times is unnecessary.
   The same effect can be obtained using -MT once with a target string
   of, e.g., "target1 target2 target3".

5. Given "-MT target", there is no pressing need for "-MQ target":
   a. Such quoting would cater to the details of one specific "make"
      implementation (potentially at the expense of all other makes).
   b. The user can perform this quoting operation externally to m4 and
      pass the resulting target string in analogously to
      -MT "$requoted_target_string".

6. In m4, there is a need to provide two versions of the -MG flag:
        a. One that affects include(file_name), and
        b. One that affects sinclude(file_name).
   Use of either flag makes the macro expanded output incorrect, since
   the missing file is replaced with an empty file during that run of
   m4.  To completely round out this functionality, we also provide:
        c. An option that affects files listed on the command line, and
        d. An option that means "all" (a, b, and c).
   Any missing file that becomes a dependency under any of these
   options appears in the generated dependency exactly as given to the
   include() or sinclude() macros, or on the command line, respectively;
   no prefixes from any -Isearch_dir options are applied.  (Consider the
   alternative: if one or more -I options are specified, how would M4
   know which (if any) prefix to use for a file that is not found in
   any of these places?)  These options assume, of course, that the
   missing files will ultimately NOT include() or sinclude() any
   additional files -- if they do, then these additional files will be
   missing from the generated dependency rule.

7. In GCC, there can be a huge difference in CPU time between -M mode
   (generate dependencies only -- only the preprocessor is run) versus
   -MD mode (generate dependencies AND perform compilation -- where all
   preprocessing, compilation, optimization and assembly are performed).
   This performance difference is lacking in m4, since the entire macro
   expansion process must still be performed in either case.  So long as
   the user has the ability to independently generate/keep or discard
   both output streams (the macro expanded output and the dependency
   rule output), then there is no need for distinct -M versus -MD modes
   in m4.  A single mode retains the fundamental capability to run m4 in
   a manner that generates: (a) macro output only, (b) dependencies only,
   (c) macro output together with dependencies, or (d) neither.

8. As a consequence of items 2, 3 and 7, invoking m4's "makedep"
   feature is therefore always analogous to

                gcc -MD -MF dependency_file -MT target

   where dependency_file and target must always be explicitly
   specified.

9. We cannot replicate the GCC option syntax in m4 because -Manything is
   already allocated to existing m4 features.  The present design uses
   options that all match the pattern "--makedep*".  Some of the
   resulting option names are very long.  Note that users typically
   will not be typing these options very often -- they will most often
   be issued either (a) from Makefile "boilerplate" (e.g., pattern
   rules), or (b) by other tools such as automake.  There are no
   single-character versions of these options -- such short forms should
   be reserved for options that users would type more frequently.

10. The -MP option of GCC actually causes a "phony" target dummy rule
    to be created for each file that is the subject of at least one
    #include directive.  Files mentioned on the GCC command line do
    not get a dummy rule (unless they are also #include'd from
    somewhere).  These semantics are somewhat subtle and inflexible.
    We address this in m4 by providing separate options controlling
    dummy rule generation for include(), sinclude() and files listed
    on the command line.  There is also a single "all" option that
    enables dummy rule generation for all three classes at once.

These rationale have given rise to the following proposed user interface
for the GNU m4 "makedep" feature.  It is phrased similarly to what we
would expect to appear in the documentation of these options in the
updated GNU m4 manual.

========================================================================

        Proposed Functional Specification
        ---------------------------------

Options controlling generation of Makefile dependency rules:

Makefile dependency rules can be automatically generated by specifying
both the --makedep=PATHNAME and --makedep-target=TARGET options.

--makedep=PATHNAME causes m4 to generate a dependency rule into the file
  specified by PATHNAME.  Macro expansion output is still written to
  stdout as normal.  This option is analogous to the -MF option of GCC.

--makedep-target=TARGET specifies TARGET to be the target of the
  generated dependency rule.  The string TARGET is used verbatim.  The
  string TARGET can contain several logical targets separated by spaces.
  It is the user's responsibility to properly express characters that
  make handles specially (such as '$', or spaces within file names).
  Since m4 sends its macro expansion output to stdout, it never really
  knows the name of the target file being generated, so the target must
  always be specified explicitly by the user with this option.  This
  option is analogous to the -MT option of GCC (except that GCC allows
  -MT to be specified multiple times).

Note that the --makedep=PATHNAME and --makedep-target=TARGET options
must either (a) both be specified, or (b) neither be specified.  They
cannot be used independently of each other.

The following additional options can also be used when dependency rules
are being generated (i.e., these options are only valid when both
--makedep=PATHNAME and --makedep=TARGET have also been specified):

--makedep-gen-missing-include causes m4 to assume that any file
  included via the include() macro that is missing (i.e., does not
  exist) is an automatically generated include file.  M4 includes such
  missing files as dependencies in the generated rule regardless.  In
  this case the dependency appears exactly as specified in the argument
  to include() and is not modified by any -Isearch_dir prefixes.  Note
  that the macro expansion output generated to stdout will be incorrect
  when this happens because the missing include file is assumed to be an
  empty file.  This option causes the m4 include() macro to behave like
  sinclude(), except that a warning message is produced on stderr to
  indicate that the requested file was missing.  This option is
  analogous to the -MG option of GCC.

--makedep-gen-missing-sinclude causes m4 to assume that any file
  included via the sinclude() macro that is missing (i.e., does not
  exist) is an automatically generated include file.  M4 includes such
  missing files as dependencies in the generated rule regardless.  In
  this case the dependency appears exactly as specified in the argument
  to sinclude() and is not modified by any -Isearch_dir prefixes.  Note
  that the macro expansion output generated to stdout will be incorrect
  when this happens because the missing include file is assumed to be an
  empty file.  This option does not alter sinclude()'s behavior of
  silently ignoring requests to sinclude() files that do not exist.

--makedep-gen-missing-argfiles causes m4 to assume that any file listed
  on the command line that is missing (i.e., does not exist) is an
  automatically generated file.  M4 includes such missing files as
  dependencies in the generated rule regardless.  In this case the
  dependency appears exactly as specified on the command line and is not
  modified by any -Isearch_dir prefixes.  Note that the macro expansion
  output generated to stdout will be incorrect when this happens because
  the missing file is assumed to be an empty file.  A warning is
  produced on stderr for each missing command line file handled in
  this manner.

--makedep-gen-missing-all is equivalent to specifying all three options:
  --makedep-gen-missing-include, --makedep-gen-missing-sinclude, and
  --makedep-gen-missing-argfiles.

Note that the above --makedep-gen-missing-* options assume that the
missing files will ultimately NOT include() or sinclude() any additional
files -- if they do, then these additional files will be missing from
the generated dependency rules.

The following options control the generation of "phony" targets for
certain classes of dependencies.  These dummy rules are used to work
around errors make gives if you remove files without updating the
Makefile to match.  Dependencies that match one or more of these classes
cause a single dummy rule to be generated for them:

--makedep-phony-include causes m4 to generate a "phony" target for each
  file that is the subject of an include() macro.  This option is
  analogous to the -MP option of GCC.

--makedep-phony-sinclude causes m4 to generate a "phony" target for each
  file that is the subject of an sinclude() macro.

--makedep-phony-argfiles causes m4 to generate a "phony" target for each
  file that is specified on the command line.

--makedep-phony-all is equivalent to specifying all three options:
  --makedep-phony-include --makedep-phony-sinclude --makedep-phony-argfiles.

========================================================================