Auto-generation of dependencies for intermediate files from implicit rules

[Cameron McCormack]

Hi all.

I'm having troubles working out a way to generate (and use) dependency
information for my project, where the targets are all built using
implicit rules.

(Please excuse the long-windedness.)

--- Description of my project

I want to generate HTML files from XML documents scattered around a
directory tree.  There are a number of different types of XML documents
here and I have a list of what type of file each can be transformed
into.

As an example sufficient for the description of this problem, these are
the types of files I have in my directories:

  - .html files
       These just contain HTML output. They can't be transformed into
       anything else.
  - .page files
       XML documents which contain textual information and can be
       transformed into .html files.
  - .subjects files
       XML documents which contain a list of subject codes and names.
       These can be transformed into .page files.
  - .layout files
       XML documents which include .page files and describe their
       physical arrangement.  Can be transformed into .page files.

The ultimate goal each time is to generate an HTML version of a
particular XML document (either a .page or a .subjects file).

Each generated file, though, is given a slightly different filename.  If
I have the file index.page, and I want to generate an HTML version of
this, the resulting file should be called ".index.html.cache" (it is a
cache of the transformation to HTML).

If I wanted to generate an HTML version of a file semester1.subjects,
first this would be transformed into ".semester1.page.cache", then this
file would be transformed into ".semester1.html.cache".

--- What I have so far

I have a set of implicit rules to do the transformations, which look
like this:

  # Rules to generate .html files from .page files
  
  %.html.cache : %.page
        $(XFORM) $< xsl/page-html.xsl > $@

  %.html.cache : %.page.cache
        $(XFORM) $< xsl/page-html.xsl > $@

  # Rules to generate .page files from .subjects files

  %.page.cache : %.subjects
        $(XFORM) $< xsl/subjects-page.xsl > $@

  %.page.cache : %.subjects.cache
        $(XFORM) $< xsl/subjects-page.xsl > $@

These rules work fine.  If I have the file "semester1.subjects" in the
directory, and I type "make .semester1.html.cache", the two commands are
run:

  $(XFORM) semester1.subjects xsl/subjects-page.xsl > .semester1.page.cache
  $(XFORM) .semester1.page.cache xsl/page-html.xsl > .semester1.html.cache

and then the intermediate file .semester1.page.cache is deleted.  (By
the way, is there some way to tell make not to delete this intermediate
file generated by implicit rules?  It would be handy to keep this one as
well.)

--- The problem

My .page files can contain an <include/> element which will insert
another XML document at that position.  Any XML document can be included
like this, as long as it can be transformed into a .page file somehow.

For example, my uni.page file might contain

  <include href="semester1.subjects"/>

which means that semester1.subjects must first be transformed into a
.page file (called .semester1.page.cache) and then included at that
position.

I need a way of saying that the uni.page depends on
.semester1.page.cache (and thus, implicitly, semester1.subjects).

Generating the list of dependencies for a .page file is not hard.  I
have a script that will do that, and output a list of *.page.cache and
*.page files that the original .page file depends on.

What I'm having trouble doing is including this dependency information
in my Makefile.

The typical method of auto-dependency generation mentioned here[1]:

  SRCS = foo.c bar.c ...

  %.P : %.c
        $(MAKEDEPEND)
        @sed 's/\($*\)\.o[ :]*/\1.o $@ : /g' < $*.d > $@; \
                rm -f $*.d; [ -s $@ ] || rm -f $@

  include $(SRCS:.c=.P)

won't work for me, because I don't have a predetermined list of files
(foo.c bar.c) which I am generating dependencies from.

At the start of the make process, I won't necessarily have all of the
*.page.cache files from which dependencies need to be extracted, since
*.page.cache files will be intermediate files generated when converting
some other XML document into HTML.

Effectively, every time a *.page.cache file is generated, its
dependencies must be recorded and included.  Something like this would
be good, but doesn't work:

  .%.page.cache : %.layout
        $(XFORM) $< xsl/layout-page.xsl > $@
        $(EXTRACTDEPS) $@ > address@hidden
        include address@hidden

If, extending my example from above slightly further, we have these
files:

  semester1.subjects
    containing a list of subject information

  master.layout
    has layout information in it that will be transformed into
    .master.page.cache containing:
        <include href="semester1.subjects"/>
    Thus, .master.page.cache depends on ".semester1.page.cache".

If I type "make .master.html.cache", to generate an HTML version of the
master.layout file, this is what should happen:

  - .master.html.cache doesn't exist, so make determines that it can use
    the implicit rule to generate it from .master.page.cache.

  - .master.page.cache doesn't exist, so make determines that it can use
    the implicit rule to generate it from master.layout.

  - $(XFORM) gets run to transform master.layout into
    .master.page.cache.

  - $(EXTRACTDEPS) gets run to extract the dependencies from
    .master.page.cache.  These dependencies are written to
    .master.page.cache-deps:

        .master.html.cache : .semester1.page.cache

    and then included.
    
  - Now that .master.page.cache exists, make can use it to generate
    .master.html.cache.  But the dependency which we just extracted
    and included is unfulfilled, so make needs to generate them.

  - .semester1.page.cache doesn't exist, so make determines that it can
    use the implicit rule to generate it from semester1.subjects.

  - Now that we have .semester1.page.cache, make can generate
    .master.html.cache from .master.page.cache (which depended on
    .semester1.page.cache).

(Phew!)

--- The upshot of this

If my example was too convoluted to follow, I think it boils down to
this: dependencies need to be generated and included for intermediate
files.  These intermediate files aren't known until the actual make
process has begun, and determined from the implicit rules that they need
to be generated.

Hopefully someone can help me!

Thanks,

Cameron

[1] http://www.paulandlesley.org/gmake/autodep.html

-- 
Cameron McCormack
  // address@hidden
  // http://www.csse.monash.edu.au/~clm/
  // icq 26955922