ANN & GNU Make usage feedback: Qake, featureful build system based on GNU Make 4.0

[Michael Pankov]

Hello everyone,

I implemented Qake, build system based on GNU Make but adding some
useful features. It's currently in stage of first working prototype. I'm
happy to receive any feedback on either implementation or usage of this
build system. I also discovered some issues making it significantly
inconvenient to use GNU Make in projects like this.

Disclaimer:

I don't think I'm the first one who discovers these issues, but I still
think there are some not-very-complex features that could be implemented
to aid in implementation of projects such as this. This is not a bug
report per se, hence I post it to this list rather to bug list. In case
any of maintainers would like me to provide details on any specific
points of feedback and/or file a detailed bug report to bug submission
list, I'm ready to do that.

Please note this is not an advertisement, I provide the feature list to
refer to it later. The purpose of the email is more about providing
feedback on using GNU Make. 

Full feature list of Qake:

1. Compact syntax for definition of entire program build;
2. Prevents lengthy meaningless rebuilds - prunes it as early as
possible. Changing the Git branch doesn't cause the full rebuild
anymore!;
3. Tracks build commands - changes of build system itself are taken into
account;
4. Terse, but useful, output - short command descriptions are output
instead of full commands by default. At the same time, errors are
colorized to be easy to notice and provide the full used command when
user needs it the most;
5. Extensible to other types of built entities. Supporting builds of
static and dynamic libraries with dependencies between them is possible,
as well as adding ways to build some custom artifacts;
6. Tracks header dependencies (of course);
7. Uses GNU Make 4.0 internally and exposes it when needed - you can
still pass custom flags to Make itself for debugging or any other
purpose.

For more information, installation instructions and everything else,
head to GitHub: https://github.com/mkpankov/qake.

As for feedback:
1. I use $(eval ...) to internalize my generated build description.
There were two main problems with this for me. 
1.1. The first and foremost is that when an error occurs, GNU Make
reports filename and line number of the call site - that is, of line
where $(eval $(call ...)) occurs. But what would be infinitely more
useful is to provide line number relative to beginning of generated
sequence - that would refer to number of line in generation function
body.
1.2. The code that gets evaluated can be viewed by replacing $(eval ...)
by $(info ...). But, when there are some embedded function calls, like
(for example) `A := $$(patsubst %/a,%/b,$(SRC))`, it gets only output as
the function call (not the evaluated values). I know I can print the
Make database and find the A's value there, but it would be
significantly more convenient if there would be a way to enforce second
expansion of such expression. It is even more desirable seeing that
adding simple $$(info $$(A)) after the definition inside the $(eval ...)
does output the fully expanded value -- meaning it's already implemented
in some way.

2. There was a problem for me with unmatched parens which go completely
unnoticed and produce incomprehensible error messages. I'm not sure what
could be done with it, though.

3. During the implementation of features 2. and 3. I discovered it would
be highly useful if one could specify "proxy" timestamp files instead of
files that a target really depends on. 

What I do now is like

b: a.did_update | a

and then I specify how to determine if `a` got updated - I hash `a` and
compare it against previously saved hash value. Then I do `touch
a.did_update` in case it was really updated, so that `b` would get
rebuilt too. This seems to be general enough, and would be more easy to
do, would there be a Make primitive to define a non-timestamp dependency
of one file on another. As for syntax, I'm not sure I'm able to propose
it right now. But it's clear that currently you have to do a lot of
$(patsubst ...) to get a path of proxy file from path of real target and
then translate it back to path to target to compute, say, a hash of it.
I'd really like to be able to just mention the real target, what
"attributes" of it should be tracked, and be done with it. I think new
automatic variables would be useful.

4. With regard to feedback 3., there is even more interesting stuff. To
implement feature 3., I track commands used by build. I first evaluate
the command and store it in a file, and the target itself just reads the
file and evaluates it as shell command.

Then, the real target depends not only on proxy files *.did_update of
dependencies, but also on *.cmd.did_update "proxy" file, which
determines if target's command got updated.

So what would be really cool is ability to define multiple dependencies
of real target on "proxy" files.

This would be useful due to current implementation being limited to hash
tracking of only files under the project root. External headers, for
example, are not tracked (feature 6. tracks headers traditionally, by
time-stamps). Built-in implementation could take care of that, I
believe.

5. There is a "shell relay script", which replaces the actual shell. It
receives a command with some command line switched from Make,
preprocesses it, and passes it through to the actual shell. It's
currently not very useful. It turned out it's ill-suited to nearly all
purposes I tried to use it for - hashing, command tracking, directory
creation. But, I'm thinking whether it would be useful for displaying
the progress of rebuild of particular target file and/or tracking the
time it takes to rebuild a target.

Limitations:

I also have to note that I do understand that creating a lot of marker
files may have significant negative impact on performance. As well as
hashing of entire sources/objects makes full usage of this build system
currently infeasible. Building a test program of less than 10 objects
already takes nearly 0.8 seconds, versus 0.35 in simplest build w/o
command tracking and hashing. But I have some thoughts on how it could
potentially be sped up, should this prototype really interest anyone.

So, that's it - hope it's useful. 


Thank you for your time,
-- 
  Michael Pankov
  address@hidden