Re: directory caching anomalies in win32

[Paul Smith]

On Tue, 2007-04-24 at 09:09 +0300, Eli Zaretskii wrote:
> Paul, up until now it looks like this is a general problem with Make's
> implementation of $wildcard: it doesn't hit the disk to find out
> whether the directory got modified since it was read and cached.  I
> think that if file.o in the above example was one of the targets of
> the current Makefile, Make adds it to the cache, but what about files
> that got created as side effects of rules' commands without Make's
> knowledge?  Doesn't this mean $wildcard should always check whether
> the directory was modified?

Sorry I've been incommunicado.  I'm afraid I won't be much more
responsive until next week sometime: I have a metric ton of work to get
done in a very short amount of time for "real life".

Anyway, you're right: the disk caching, while it greatly improves
performance, is also very problematic for "marginal" makefiles where
files are created behind make's back.  Make expects that any file that
is created or modified while it is running is a target that it knows
about.  If that's not true, then you unquestionably can run into trouble
(and wildcard is not the only place).


Personally, rather than just unhook wildcard from the disk cache (which,
by the way, is not that simple because we hook the disk cache in
UNDERNEATH the GNU globbing library, IIRC), I'd rather "robust-ify" the
disk caching altogether.

I've thought about this a good bit over the years and I have some
thoughts on how to fix it, by marking the cache as stale.  This will
actually reduce performance, in some cases by a good bit, but it will
yield correct results.  Staleness can be determined by directory
timestamp (for systems with intelligent directory timestamp semantics),
or we can increase a counter every time we invoke a sub-shell, and store
the value of the counter with each directory when we read it (for
systems without intelligent directory timestamp semantics).

An alternative would be to allow the user to mark, somehow, when the
cache is stale to make sure it's re-read appropriately (marking targets
which have side-effects, for example).  This would give better
performance but would require changes to the makefiles, which I don't
like.

I guess one thing we'll need are some benchmarks from environments that
make heavy use of, and get great benefit from, the disk caching so we
can see how any changes will impact them.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <address@hidden>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist