Re: [lmi] Should lmi use an xml catalog resolver?

[Greg Chicares]

On 10/30/20 11:24 PM, Vadim Zeitlin wrote:
> On Fri, 30 Oct 2020 20:39:16 +0000 Greg Chicares <gchicares@sbcglobal.net> 
> wrote:
> 
> GC> Kim built everything from scratch yesterday with cygwin, and her logs
> GC> show a different set of "failed to load external entity" messages,
> GC> with no chroot involved, of course.
> 
>  Sorry, it's apparently my build environment which is special and not
> yours, after all: I also see such warnings in GitHub CI builds, see e.g.
> 
> https://github.com/let-me-illustrate/lmi/runs/1333943758?check_suite_focus=true#step:10:1131

I see a check-mark in a circle, next to "Build XML libraries", and "1m 38s",
but no warnings. But "uBlock Origin" has api.github.com blocked, as well as
the more nefarious-sounding collector.github.com .

> GC> The count for Kim's log is 292. The error messages (more compact
> GC> than mine above) begin:
> GC> 
> GC> Making install in doc
> GC> make[1]: Entering directory 
> '/opt/lmi/local/gcc_x86_64-w64-mingw32/xml-ad_hoc/libxslt/doc'
> GC> Validating the HTML Web pages
> GC> Validating the HTML Web pages
> GC> warning: failed to load external entity "API.html"
> GC> warning: failed to load external entity "bugs.html"
> GC> warning: failed to load external entity "contribs.html"
> 
> It's strange that neither you nor Kim get the first error message from the
> CI build, which is:
> 
> I/O error : Attempt to load network entity 
> http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd

She doesn't, but I do.

> (could it be filtered out from the build log somehow?),

No.

> and seems to be the
> root explanation of the problem. This error seems to be due to the use of
> "--nonet" xsltproc option in tests/docbook/Makefile.am and the absence of
> the DTD on the local file system. The only strange thing is that there
> don't seem to be any Debian package providing this DTD, even though it is
> included in a few of them.

I think I reported here earlier that there was used to be a debian
package with lots of DTDs, but it was dropped after old-old-stable.

> The DTD is also present in tests/valid/dtds
> subdirectory of libxml2 sources, so we could probably just set
> XML_CATALOG_FILES environment variable to point there to avoid it. Should I
> try doing this?

It was more educational for me to do it: commit 92946c11ae.

> GC> In my most recent (chroot) log, the same section looks like this:
> GC> 
> GC> make[1]: Entering directory 
> '/opt/lmi/local/gcc_x86_64-w64-mingw32/xml-ad_hoc/libxslt/doc'
> GC> Rebuilding the HTML Web pages from xslt.html
> GC> [...the preceding line repeats about a dozen times...]
> GC> Validating the HTML Web pages
> GC> /usr/bin/xmllint: relocation error: /usr/bin/xmllint: symbol 
> xmlSchematronParse version LIBXML2_2.6.21 not defined in file libxml2.so.2 
> with link time reference
> 
>  This is annoying, we seem to be using system xmllint with our own
> libraries that are incompatible with it (e.g. because we explicitly use
> --without-schematron configure option, while system xmllint is built with
> Schematron support).
> 
>  It doesn't change much, as validation still fails, just for a different
> reason, but we still don't really care about it, but we probably should
> avoid this, probably by setting the PATH when configuring libxslt in a way
> to make sure that it finds xmllint that we've just built, instead of the
> system one which it finds by default.

That's deliberate: commit eb35fec587d refers to the explanation here:
  https://lists.nongnu.org/archive/html/lmi/2019-01/msg00034.html
| However, the full command pipeline is much slower. The demonstration
| below compares the wine vs. native timings, summarized here:
|   46.509 total [wine /path/to/lmi/xmllint]
|    0.197 total [/usr/bin/xmllint]
| which suggests that using wine in a pipeline is the problem

> GC> Does the
> GC>   symbol xmlSchematronParse version LIBXML2_2.6.21 not defined in file 
> libxml2.so.2 with link time reference
> GC> suggest a problem that might have an obvious solution?
> 
>  Yes, I think so: we should set the PATH to use the correct xmllint binary.
> Again, it's not going to change much in practice, so I'm not sure how
> important is it to do it, but we could fix this.

I no longer observe that message, so I won't spend more time on it now.

> GC> > but if there isn't, we can live with 244 extra lines in a 15000-line 
> build log.

It's more like a thousand--the 92946c11ae commit message says, for
the 'install_xml_libraries.sh' run in isolation:

    Lines and bytes of output:
      2560 171891 previously
      1671 100248 with this change

Fifty percent more lines, and seventy percent more characters--more
tedium every time I compare build logs. I'm glad to be rid of that.

BTW, the xml catalog now embedded as a here-document was generated thus:

xmlcatalog --create >throwaway_catalog
xmlcatalog --noout --add "rewriteURI" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";  
"file:///opt/lmi/src/lmi/third_party/libxml2/test/valid/dtds/xhtml1-transitional.dtd"
 throwaway_catalog
xmlcatalog --noout --add "rewriteURI" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent";  
"file:///opt/lmi/src/lmi/third_party/libxml2/test/valid/dtds/xhtml-lat1.ent" 
throwaway_catalog
xmlcatalog --noout --add "rewriteURI" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent";  
"file:///opt/lmi/src/lmi/third_party/libxml2/test/valid/dtds/xhtml-special.ent" 
throwaway_catalog
xmlcatalog --noout --add "rewriteURI" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent";  
"file:///opt/lmi/src/lmi/third_party/libxml2/test/valid/dtds/xhtml-symbol.ent" 
throwaway_catalog

but it seemed better not to require 'xmlcatalog' to have been built
before we build it. I used "rewriteURI" because that's what's used
in this script:
  http://xmlsoft.org/buildDocBookCatalog
which is referenced here:
  http://xmlsoft.org/catalog.html
  "There is a shell script to generate XML Catalogs"
and it does work, though I don't pretend to understand what all the
  {public, system, uri, rewriteSystem, rewriteURI, delegatePublic,
   delegateSystem, delegateURI, nextCatalog}
types mean.