Re: Dependencies and other issues with generated files

[Nick Bowler]

Hello David,

On 2012-06-27 18:13 +0200, David Sveningsson wrote:
> I'm having some problems when generating source files. I have written a
> tool which generates both a c source-file, a header and dependencies of
> the source, all at the same time using a single command.

There is an entire chapter of the Automake manual about exactly this:

  §28.9 Handling Tools that Produce Many Outputs
  https://www.gnu.org/software/automake/manual/automake.html#Multiple-Outputs

The short version is that it is hard, but possible, to do this in make.
Modifying your tool so that it can be invoked multiple times, producing
one output file per invocation will be much simpler, although this is
not always a satisfying option.

None of the examples in the manual use suffix rules, but it's
straightforward to do so.  However, note that many non-GNU make
implementations are not smart enough to figure out chains of
suffix rules correctly (if at all), such as

 .o -> .c -> .foostamp -> .foo

so you may need to add some explicit prerequisites to your Makefile to
avoid such chains.

Some specific comments about your current approach:

> After stripping of the flags my rules basically look like this:
> 
>     SUFFIXES = .foo

Most of the time, Automake is smart enough to add the required suffixes
automatically, so you do not need to specify SUFFIXES in this example.

>     app_SOURCES = test.foo main.c

I'm not sure how well listing the .foo file in _SOURCES is going to
work, although someone who knows more about automake guts might be able
to tell us exactly what happens.  You should probably list the .c file
here, or in nodist_app_SOURCES if the generated source file is not meant
to be distributed.

>     .foo.c:
>       ./mytool -f $< -e $(basename $@).h -o $@
> 
> (main.c includes test.h)
> 
> The first problem I'm having is that test.h may not have been built when
> compiling main.c (especially if using parallel build). I guess a
> workaround would be to add a dependency like "main.c: test.c". Is there
> a way to handle this dependency in a more automatic way?

Again, this is covered in depth in the Automake manual.  Regardless, you
*will* need to add an explicit prerequisite on the header file, because
automatic dependency tracking does not work for generated headers.

> Secondly if test.foo is in a subdirectory and I'm doing an out-of-tree
> build the folder does not exist. Is there a way to automatically create
> the required folders or rename the file similar to object files (e.g.
> myapp-main.o)? I got it working using "@test -e $(dir $@) || mkdir -p
> $(dir $@)", which I guess would be fine but it also seems like a hack to me.

Instead of the above, you should use something like the following at the
start of your rule:

  test x"$(@D)" = x || $(MKDIR_P) "$(@D)"

$(MKDIR_P) is provided by autoconf and works even when the system mkdir
does not support -p.

Unlike $(dir $@), the $(@D) construct is defined in POSIX and works on
every make implementation I have access to.  However, contrary to POSIX,
dmake expands $(@D) to the empty string for targets in the current
working directory (which will cause the mkdir to fail), so we need to
avoid calling mkdir in that case (hence the test).

If all the relevant targets are in subdirectories, you can remove the
test as dmake will produce a useful string in this case.

> Next I'm having issues when cleaning. I would like "make clean" to
> remove the generated files. I was expecting at least test.c to be
> cleaned automatically but none of the generated files was removed.
> Manually adding to CLEANFILES or clean-local is tiresome and error-prone.

Automake does not, and cannot, generate clean rules for files it does
not know about.  So I'm afraid that's your job.  Such rules are usually
pretty easy to write, for example if you have all your .foo files in a
variable:

 FOOFILES = a.foo b.foo c.foo

and the tool generates .c and .h files from a .foo file, then something
like

 CLEANFILES = $(FOOFILES:.foo=.c) $(FOOFILES:.foo=.h)

is probably sufficient.  For more involved cases, you might add a
--clean option to your tool which deletes any files that it generates,
and then call it from clean-local.

Hope that helps,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)