Re: [Fwd: Can nobase_pkgdata_DATA take directories? Does not seam so.]

[Sander Niemeijer]

On vrijdag, maa 5, 2004, at 19:54 Europe/Amsterdam, Hans Deragon wrote:

> Sander Niemeijer wrote:
> > Hi Hans,
> > Automake does not support wildcard specification of files (e.g. whole 
> > directories). This is to make sure that:
> > - a 'make dist' only includes those files from your subdirectories 
> > that really should go in a distribution
> > - a 'make distcheck' is able to check whether any generated files in 
> > your subdirectories are properly cleaned by a 'make distclean'
> > - a 'make uninstall' does not remove any files a user may have put in 
> > one of your subdirectories
>
> Mmm... the dist target takes directories via EXTRA_DIST, which in a 
> sense is a wildcard.  As for the installation, why shouldn't in *_DATA 
> variables allow wildcards?  Its up to the author to check if any 
> unwanted files are being installed.
>
> The only place where wildcards should not be allowed is in the 
> uninstall target.  If the uninstall target uses *_DATA to determine 
> which file to uninstall, maybe the code should be changed.  A 
> dynamically created file should record what specific file has been 
> installed to record what file actually has been installed.  The 
> uninstall target would then use the content of this dynamically 
> created file to uninstall only the relevant files and yes, 
> directories. :)

There are however also some technical problems involved. If all your 
data files are already there then using some wildcard feature could be 
an option I guess. However, how would one deal with the case where the 
files you specify through wildcards need to be generated first?
This is the case we had to solve for our BEAT package. The wildcard 
files are documentation files generated by doxygen. In our package 
these files are treated similar to other generated files (such as .c 
files generated by tools such as lex and yacc) in the sense that they 
are included with a distribution (so the user does not have to create 
the documentation himself and so doesn't need to have doxygen 
installed). But if the user does have the proper documentation 
generation tools, he is always able to make changes to the sources and 
recreate the documentation if he wants.
The problem now comes when you want to create a dependency for the 
install/dist target on the generated files. We can distinguish two 
cases:

1) The generated files are not yet there and we want a dependency rule 
to create the files.
This is of course a chicken-egg problem. Because the generated files 
are not there yet, a wildcard specification would turn up empty and the 
files would never be generated. Unless you can specify some suffix 
rules to create the generated files (which we can't for our doxygen 
generated files) you are forced to enumerate each and every file 
somewhere.
In our BEAT package I have tried to create a proper solution for this. 
When the generation tool is run, I keep track of the files it generates 
and create a Makefile include for this (but the include file only gets 
updated if any changes to the list of files was found, which prevents 
unnecessary reruns of automake). This Makefile include is stored in our 
CVS and included in our distribution. There are some bootstrapping 
issues involved (which your rightly pointed out) that I'll describe 
below.

2) The generated files are already there but the sources from which 
they were generated have changed, so we want a dependency rule to 
automatically update our generated files.
In this case the wildcard specification would give the proper list of 
files, but the problems we may now face are that each file within the 
wildcard specification could have different dependencies, or, which is 
the case for our doxygen rules, that the generated files are dependent 
on a wildcard specification of source files.
Because my head was already cracking when trying to solve issue 1) and 
because 2) was not really important for us, I have not implemented any 
automatic regeneration rules for our documentation in our package :-)

> > And probably even more reasons...
> > For a project I am working on we were faced with a similar problem. I 
> > ended up writing my own script that creates a 'makefile include' 
> > containing rules for all my 'wildcard files' (in our case generated 
> > documentation files). If you want to see how we did this, just have a 
> > look at our open-source BEAT package (downloadable from 
> > http://www.science-and-technology.nl/beat/).
>
> I tried your trick and it works.  Thanks, I have two question though.  
> I have the following line:
>
>   include file.lst
>
> Now, if file.lst does not exist, make complains with an error and 
> aborts, even though I created a target to specify how to build it.  Do 
> you have a suggestion?

This is the Makefile include bootstrapping problem I mentioned. The way 
I do this is to start with an empty include file ('touch file.lst') and 
then perform the generation (this will update the file.lst and include 
rules for all generated files). Then I check in file.lst into CVS and 
I'm set for the future ;-)

> Also, when I call the "all", "install", "dist" targets, I would like 
> to have the file.lst target be called, so its be regenerated 
> automatically.  You have any advice?

IMO You should avoid regenerating the file.lst as much as possible, 
since a change to file.lst requires a rerun of automake (which users 
may not have). I recommend to treat regenerating file.lst as you would 
treat changing your Makefile.am (you are not going to regenerate the 
contents of your Makefile.am either when you run a 'make 
all/install/dist').

> Obviously, if you and I are building systems to emulate wildcards, its 
> because there is definitively a need for wildcards to be supported.  
> Should we make a feature request?

I think most people already know this would be a useful feature. But 
the trick is to find a proper solution and, as always, the question is 
whether a developer has the time/motivation to start implementing such 
a feature.

Regards,
Sander