Re: [PATCH v2 1/8] tests: Make sure LANG is set properly for iso9660_test

[Glenn Washburn]

Hi Thomas,

On Wed, 25 Aug 2021 11:34:47 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Hi,
> 
> Glenn Washburn wrote:
> > LANG must be set to something that supports international
> > characters, otherwise xorriso will refuse to include the file with
> > name having international characters, causing the test to fail.
> 
> Can you tell me the exact error message from xorriso ?
> I have some difficulties with reproducing the situation.

There is no error message from xorriso, it will do as you noticed
below, which is to default to replacing non-ASCII characters with
underscores.

The problem is that that behavior causes the tests to fail (rightly so
because non-ASCII names are not being tested). The following causes the
test to fail, but removing the LANG= causes success (because my system
has LANG set to a UTF8 encoding).

  make SUBDIRS= LANG= TESTS=iso9660_test check

> Your description is plausible, because libisofs takes its character
> set default from nl_langinfo(3) with item CODESET and uses iconv(3) to
> convert the file names to the output charset.
> 
> I would like to bring up for discussion an alternative remedy by these
> xorrisofs options:
> 
>   -input-charset UTF-8 -output-charset UTF-8
> 
> -----------------------------------------------------------------------
> 
> I am quite sure that it will work, but have problems with creating the
> situation where it shall be the remedy.
> On a system with configured LANG=en_US.UTF-8 i fail to reproduce
> the issue with LANG=C or LANG="" unless i add the explicit demand to
> convert to UTF-8.
> 
> My disappointingly unproblematic xorriso test run is
> 
>   ( LANG=  xorrisofs -o test.iso 'ÄÖÜß' )
> 
> with various attempts to set the locale charset away from UTF-8:
> LANG, LANGUAGE, LC_ALL set to empty text or to "C".
> LANG= causes nl_langinfo(CODESET) to return "ANSI_X3.4-1968" but
> iconv(3) throws no error when it shall convert
> "\303\204\303\226\303\234\303\237".
> 
> I get warning messages if i tell xorriso to aim for output charset
> UTF-8 while LANG is empty and the file name contains non-ASCII
> characters:
> 
>   (LANG= xorrisofs -o test.iso -output-charset UTF-8 'ÄÖÜß')
> 
> yields three times
> 
>   libisofs: WARNING : Charset conversion error. Cannot convert ÄÖÜß
> from ANSI_X3.4-1968 to UTF-8 libisofs: NOTE :  > Caused by: Charset
> conversion error
> 
> Inspection of test.iso shows that the file name 'ÄÖÜß' was defaulted
> to name '________'.
> 
> Adding the other charset option
> 
>   (LANG= xorrisofs -o test.iso \
>             -input-charset UTF-8 -output-charset UTF-8 'ÄÖÜß')
> 
> yields a run without warning messages resulting in Rock Ridge file
> name 'ÄÖÜß'.
> 
> So i assume that my proposal is a valid alternative to setting
> LANG=UTF-8. Whether it is preferrable over setting LANG to UTF-8
> would have to be discussed.
> At least it would be nice for me to know why LANG= causes trouble in
> the GRUB tests but not on my command line.
> (I see nothing in the xorriso runs of tests/util/grub-fs-tester.in
> which would be equivalent to the run which fails to convert 'ÄÖÜß'
> here.)

The takeaway here is that 'ÄÖÜß' is being converted and its is
precisely that which is causing the test to fail. So unless I'm
mistaken, you are confirming the issue this patch resolves.

It looks like you have a viable alternative to my proposed patch,
though I've not tested it myself. I really don't care about the
implementation as long as the test succeeds when LANG is unset or
empty. Despite having an alternative approach, do you have concerns or
objections to my patch? Also would you like to create a patch that I
would be happy to test?

Glenn