[bug #65972] Change in dependency treatment between GNU make 4.3 and 4.4.1

[Martin Wilck]

URL:
  <https://savannah.gnu.org/bugs/?65972>

                 Summary: Change in dependency treatment between GNU make 4.3
and 4.4.1
                   Group: make
               Submitter: mwilck
               Submitted: Wed 10 Jul 2024 02:59:38 PM UTC
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.4.1
        Operating System: POSIX-Based
           Fixed Release: None
           Triage Status: None


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Wed 10 Jul 2024 02:59:38 PM UTC By: Martin Wilck <mwilck>
I noted a change between make 4.3 and 4.4.1. In short, make 4.4.1 tries to
rebuild certain targets which were considered up to date in 4.3.

In the CI for [https://github.com/openSUSE/multipath-tools/ multipath-tools],
I sometimes need to package the binary artifacts for the CI (all libraries and
the CI programs linking to them) in an archive, in order to unpack and run
them in a different environment. This is useful for building CI for non-native
architectures, for example (cross-compiling on x86_64, archiving the results,
and then running the cross-compiled test programs under qemu is much faster
then doing the entire compilation and testing under qemu).

The command I use for running the tests is "make -C tests". The Makefile has
rules to run the test programs, roughly like this (the real logic is more
complex):


all:    $(TESTS:%=%.out)

%.out:  %-test
        ./$< >$@


After unpacking the archive with the test programs and libraries, 
these rules would simply run all xxx-test programs and store the results in
the respective xxx.out files. At least this is how it worked until make 4.3.

With GNU make 4.4.1, make tries to rebuild some of the test programs, which
fails, because the environment for running the tests is lacking some of the
tools and libraries required for building.

Here is a reproducer. So far I couldn't come up with anything simpler, sorry.


> git clone -b tip --single-branch
https://github.com/openSUSE/multipath-tools
> cd multipath-tools

# Use a preconfigured container with all dependencies for building
multipath-tools
# Note that these containers have "make" as entrypoint
> podman run -it --rm  -v $PWD:/build
ghcr.io/mwilck/multipath-build-debian-sid -j8 test-progs.tar

# clean up
> mv test-progs.tar  /tmp/
> git clean -d -f -x

# Unpack the test programs. Note the "m" flag to make sure the time stamps of
the programs are newer than those of the sources / dependencies.
tar xfm /tmp/test-progs.tar

# Under Debian Sid (make 4.4.3), this just runs the test programs
# Note that these containers have "make" as entrypoint
> podman run -it --rm  -v $PWD:/build
ghcr.io/mwilck/multipath-build-debian-sid -C tests --dry-run
make: Entering directory '/build/tests'
echo == running uevent-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./uevent-test >uevent.out
2>&1 || { cat "uevent.out"; false; }
echo == running parser-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./parser-test >parser.out
2>&1 || { cat "parser.out"; false; }
echo == running util-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./util-test >util.out 2>&1 ||
{ cat "util.out"; false; }
echo == running dmevents-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./dmevents-test >dmevents.out
2>&1 || { cat "dmevents.out"; false; }
echo == running hwtable-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./hwtable-test >hwtable.out
2>&1 || { cat "hwtable.out"; false; }
echo == running blacklist-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./blacklist-test
>blacklist.out 2>&1 || { cat "blacklist.out"; false; }
echo == running unaligned-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./unaligned-test
>unaligned.out 2>&1 || { cat "unaligned.out"; false; }
...

# But under Fedora (make 4.4.1), it tries to rebuild some of the binaries:
# Note that these containers have "make" as entrypoint
> podman run -it --rm  -v $PWD:/build ghcr.io/mwilck/multipath-build-fedora-40
-C tests --dry-run
make: Entering directory '/build/tests'
echo == running uevent-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./uevent-test >uevent.out
2>&1 || { cat "uevent.out"; false; }
echo == running parser-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./parser-test >parser.out
2>&1 || { cat "parser.out"; false; }
echo == running util-test ==
LD_LIBRARY_PATH=.:../libmpathutil:../libmpathcmd ./util-test >util.out 2>&1 ||
{ cat "util.out"; false; }
echo building ../libmultipath/devmapper.o because of
../libmultipath/devmapper.c
cc -D_FORTIFY_SOURCE=3  -DURCU_VERSION=0x000e00 -D_FILE_OFFSET_BITS=64
-DBIN_DIR=\"/sbin\" -DMULTIPATH_DIR=\"/lib64/multipath\"
-DRUNTIME_DIR=\"/run\" -DCONFIG_DIR=\"/etc/multipath/conf.d\"
-DDEFAULT_CONFIGFILE=\"/etc/multipath.conf\" -DSTATE_DIR=\"/etc/multipath\"
-DEXTRAVERSION=\"\" -MMD -MP -I../libmultipath -I../libmpathutil
-I../libmpathcmd -I../multipathd -DTESTCONFDIR=\"/build/tests/conf.d\"
-std=gnu99  -O2 -g -fstack-protector-strong --param=ssp-buffer-size=4 -Werror
-Wall -Wextra -Wformat=2 -Wformat-overflow=2 -Werror=implicit-int
-Werror=implicit-function-declaration -Werror=format-security -Wno-clobbered
-Wno-error=clobbered -Werror=cast-qual -Werror=discarded-qualifiers
-Wno-type-limits -pipe -fexceptions -fPIE -DPIE -Wno-unused-parameter   -c -o
../libmultipath/devmapper.o ../libmultipath/devmapper.c
echo building dmevents.o because of dmevents.c
cc -D_FORTIFY_SOURCE=3  -DURCU_VERSION=0x000e00 -D_FILE_OFFSET_BITS=64
-DBIN_DIR=\"/sbin\" -DMULTIPATH_DIR=\"/lib64/multipath\"
-DRUNTIME_DIR=\"/run\" -DCONFIG_DIR=\"/etc/multipath/conf.d\"
-DDEFAULT_CONFIGFILE=\"/etc/multipath.conf\" -DSTATE_DIR=\"/etc/multipath\"
-DEXTRAVERSION=\"\" -MMD -MP -I../libmultipath -I../libmpathutil
-I../libmpathcmd -I../multipathd -DTESTCONFDIR=\"/build/tests/conf.d\"
-std=gnu99  -O2 -g -fstack-protector-strong --param=ssp-buffer-size=4 -Werror
-Wall -Wextra -Wformat=2 -Wformat-overflow=2 -Werror=implicit-int
-Werror=implicit-function-declaration -Werror=format-security -Wno-clobbered
-Wno-error=clobbered -Werror=cast-qual -Werror=discarded-qualifiers
-Wno-type-limits -pipe -fexceptions -fPIE -DPIE -Wno-unused-parameter   -c -o
dmevents.o dmevents.c
cc -O2 -g -fstack-protector-strong --param=ssp-buffer-size=4
-D_FORTIFY_SOURCE=3  -DURCU_VERSION=0x000e00 -D_FILE_OFFSET_BITS=64
-DBIN_DIR=\"/sbin\" -DMULTIPATH_DIR=\"/lib64/multipath\"
-DRUNTIME_DIR=\"/run\" -DCONFIG_DIR=\"/etc/multipath/conf.d\"
-DDEFAULT_CONFIGFILE=\"/etc/multipath.conf\" -DSTATE_DIR=\"/etc/multipath\"
-DEXTRAVERSION=\"\" -MMD -MP -I../libmultipath -I../libmpathutil
-I../libmpathcmd -I../multipathd -DTESTCONFDIR=\"/build/tests/conf.d\"  -E
dmevents.c | \
        sed -n 's/^.*__wrap_\([a-zA-Z0-9_]*\).*$/-Wl,--wrap=\1/p' | \
        sort -u | tr '\n' ' ' >dmevents.o.wrap
cat: dmevents.o.wrap: No such file or directory


I am attaching logs from Debian sid (sid.log, make 4.3) and Fedora 40
(f40.log, make 4.4.1).

Short excerpt for 4.3:

> grep Must /tmp/sid.log 
  Must remake target 'uevent.out'.
  Must remake target 'parser.out'.
  Must remake target 'util.out'.
  Must remake target 'dmevents.out'.
  Must remake target 'hwtable.out'.
  Must remake target 'blacklist.out'.
  Must remake target 'unaligned.out'.
  Must remake target 'vpd.out'.
  Must remake target 'pgpolicy.out'.
  Must remake target 'alias.out'.
  Must remake target 'directio.out'.
  Must remake target 'valid.out'.
  Must remake target 'devt.out'.
  Must remake target 'mpathvalid.out'.
  Must remake target 'strbuf.out'.
  Must remake target 'sysfs.out'.
  Must remake target 'features.out'.
  Must remake target 'cli.out'.
  Must remake target 'mapinfo.out'.
Must remake target 'all'.


Short excerpt for 4.4.1:

> grep Must /tmp/f40.log 
 Must remake target 'uevent.out'.
 Must remake target 'parser.out'.
 Must remake target 'util.out'.
   Must remake target '../libmultipath/devmapper.o'.
   Must remake target 'dmevents.o'.
   Must remake target 'dmevents.o.wrap'.
  Must remake target 'dmevents-test'.
 Must remake target 'dmevents.out'.
   Must remake target '../libmultipath/discovery.o'.
   Must remake target '../libmultipath/blacklist.o'.
   Must remake target '../libmultipath/structs_vec.o'.
   Must remake target '../libmultipath/structs.o'.
   Must remake target '../libmultipath/propsel.o'.
   Must remake target 'test-lib.o.wrap'.
   Must remake target 'hwtable.o'.
   Must remake target 'hwtable.o.wrap'.
   Must remake target 'test-lib.o'.
  Must remake target 'hwtable-test'.
 Must remake target 'hwtable.out'.
   Must remake target 'test-log.o.wrap'.
   Must remake target 'blacklist.o'.
   Must remake target 'blacklist.o.wrap'.
   Must remake target 'test-log.o'.
  Must remake target 'blacklist-test'.
 Must remake target 'blacklist.out'.
 Must remake target 'unaligned.out'.
   Must remake target 'vpd.o'.
   Must remake target 'vpd.o.wrap'.
  Must remake target 'vpd-test'.
 Must remake target 'vpd.out'.
 Must remake target 'pgpolicy.out'.
   Must remake target '../libmpathutil/util.o'.
   Must remake target 'alias.o'.
   Must remake target 'alias.o.wrap'.
  Must remake target 'alias-test'.
 Must remake target 'alias.out'.
 Must remake target 'directio.out'.
   Must remake target '../libmultipath/valid.o'.
   Must remake target 'valid.o'.
   Must remake target 'valid.o.wrap'.
  Must remake target 'valid-test'.
 Must remake target 'valid.out'.
 Must remake target 'devt.out'.
   Must remake target '../libmpathvalid/mpath_valid.o'.
   Must remake target 'mpathvalid.o'.
   Must remake target 'mpathvalid.o.wrap'.
  Must remake target 'mpathvalid-test'.
 Must remake target 'mpathvalid.out'.
   Must remake target '../libmpathutil/strbuf.o'.
   Must remake target 'strbuf.o'.
   Must remake target 'strbuf.o.wrap'.
  Must remake target 'strbuf-test'.
 Must remake target 'strbuf.out'.
   Must remake target '../libmultipath/sysfs.o'.
   Must remake target 'sysfs.o'.
   Must remake target 'sysfs.o.wrap'.
  Must remake target 'sysfs-test'.
 Must remake target 'sysfs.out'.
 Must remake target 'features.out'.
   Must remake target '../multipathd/cli.o'.
   Must remake target 'cli.o'.
   Must remake target 'cli.o.wrap'.
  Must remake target 'cli-test'.
 Must remake target 'cli.out'.
 Must remake target 'mapinfo.out'.
Must remake target 'all'.



The
[https://github.com/openSUSE/multipath-tools/blob/053fd3b0976604d9fcced20236e9a847baa58bf7/tests/Makefile#L125
recipe that I am using] for the test progams is arguably very complicated.
This is due to the necessity to insert "-Wl,--wrap" linker options to make the
cmocka-based tests work, and to the fact that each test program has different
dependencies (which also goes back to the way "-Wl,--wrap" works).

Anyway, the behavior between 4.3 and 4.4.1 differs, and I'm not sure if this
is a bug or a feature. You can see above that the behavior doesn't differ for
all test programs. AFAICS, only thos tests are affected list explicit
dependencies on the implicit rule for "%.-test" in the .SECONDEXPANSION part.
The others (e.g. uevent-test, mapinfo-test) are considered up-to-date by 4.4.1
as well.








    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?65972>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

signature.asc
Description: PGP signature