bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

On Thu, Aug 10, 2023 at 8:29 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> [Provide] an overview of the process by which

nt/admin/dist-build/build-dep-zips.py - it's really just that, as far
as how I collect the deps presently

To be clear, by "deps", I'm referring to DLLs that are not built with
Emacs but which we should (optionally) include when distributing
binary versions of Emacs for Windows.

Similarly pedantically, here are the files I (might) include for a
given release of Windows binaries for Emacs:

FULL ZIP: emacs-${VER}.zip - the "full zip", includes the deps DLLs
this script packages
NO DEPS ZIP: emacs-${VER}-no-deps.zip - specifically without the deps
DLLs this script packages
INSTALLER EXE: emacs-${VERION}-installer.exe - a self-installer
including the deps DLLs (and it compresses stuff even more than zip
-9. nice)
DEPS ZIP: 
emacs-${MAJOR_VERSION}-${DEPS_CREATE_DATE_IF_NOT_FIRST_AND_ONLY_FOR_MAJOR_VERSION}.zip
- contains only the deps DLLs. Created when changes are needed, more
to come on this.
DEPS SOURCES ZIP: the MSYS2/MINGW64 source code for all DLLs in DEPS
ZIP. Created only when a new DEPS ZIP is created

Finally (I swear, I'm going to answer your actual question), to place
the script in context of my present "use-case" (release and
release-similar builds specifically, here):

0. PREREQ: I have various folders setup, a working MSYS/MINGW64 that
has recently built me a working Emacs, I've been watching mailing
lists, I have ACLs going for me, etc.

1. A (pre)release tarball is pushed to GNU (alpha) FTP.

2. Skip this step if a "last good" deps and deps source file exist and
it is current (i.e. when there are not known new or updated
--including optional-- dependendencies for Emacs to include for the
first time),  Otherwise, update and run the script (more on this
below, obviously) in question to create (new) deps and deps source
archives containing, respectively, a bunch of DLLs we plan to
distribute (nominally) with Emacs (this is the deps archive), and the
MSYS2/MINGW64 sources for these DLLs and their compile time
dependencies (the deps sources archive).

3. Download and unpack the tarball manually

4. Configure && make install  to ${PREFIX}. Note, I have a tedious
pipeline command I like for this; I'm not currently using
"build-zips.sh" from the same folder as the build-dep-zips.sh script
in question.

5. Create a no-deps archive, essentially:

  zip -r9 ${PREFIX} emacs-${TARGET_VERSION_NAME}-no-deps.zip

6. Unpack the deps archive created in step #2 (or the "last good", if
that step was skipped), something like:

  unzip -d ${PREFIX}/bin ${LAST_GOOD_DEPS_ZIP_FOR_EMACS_MAJOR_VERSION}

7. Create a "full" zip of Emacs (now including those "extra" DLLs),
essentially repeating step #5 with a different archive file name  (in
fact, I copy no-deps and re-add bin to it)

8. Create the installer

9. Perform certain rights and incantations to cause files to appear on
GNU (alpha) FTP servers.  Notably, include new deps files ONLY when
they have been newly created along with the release being published.
These files have historically changed only rarely within a given Emacs
major version.

> collect(s) the list of the dependency packages for the binary distro,

Ignoring my patch for the moment, the script contains two hard-coded
lists that represent our first likely source of breakage:

DLL_REQ lists specific DLLs that should be copied into ${PREFIX}/bin,
after make install, to get a "complete" Emacs installation.  During
the Emacs 29 pre-release cycle I added sqlite3 and tree-sitter to this
list, enabling those features to work "out of the box".  In some
cases, such as these two in fact, Emacs would likely function
correctly under Windows if we chose not to distribute a particular
DLL; however, I believe this is not the case for all of DLLs included
and, moreover (in my opinion) would tend to make Emacs to less viable
as a means of drawing Windows users closer to Free Software by virtue
of Just Being Better, which would be a bit sad.

PKG_REQ lists the mingw-w64-x86_64 source package name for each item
in the first list.

These two variables are coordinated lists and so obviously could be
refactored (e.g. into an associative array) that I suspect historical
raisins (meaning, perhaps once these lists were not so coordinated; I
didn't research this so far).

In addition to these "main lists" there are a few other vars/lines to
study (now taking from my patched version):

SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
SKIP_DEP_PKGS=["mingw-w64-glib2" "mingw-w64-ca-certificates-20211016-3"]

We'll come to the logic -where all of these are used-- next.

> including the main ideas and information sources[.  ...]

Ignoring as much complexity as possible by focusing on the
release(like) use-case, we can ignore the arguments and conditions
and, finally, we can reduce the the script to:

2.1 evaluate above mentioned vars
2.2 collect all DLLs mentioned in DLL_REQ
2.3 collect the DLLs that are unique dependencies the DLLs collected
in 2.2 skipping any which appear in SKIP_DEP_PKGS
2.4 collect the source for all DLLs from 2.2 and 2.3 unless the source
package is listed in SKIP_SRC_PKGS

I hope I did cover information sources well enough, but to put a fine
point on it: I watch Emacs devel, bug reports, IRC, reddit, ..., for
information leading to updates to this script. Developers and others
using Emacs on Windows are the main "information sources", at least
speaking from developing that patch :)

>
> Thanks.
>