Re: [PATCH v2 1/8] tests: Make sure LANG is set properly for iso9660_test

[Glenn Washburn]

On Thu, 26 Aug 2021 19:49:37 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Hi,
> 
> now i know that i should have asked for the test-suite.log file.
> 
> The problem turned out to be in Joliet, not in ISO 9660.
> Because Joliet gets its file names and other texts encoded as UCS-2
> 16-bit characters it is indeed inavoidable to define the meaning of
> the bytes in a Unix file name.
> The UTF-8 sequence \303\251 has no meaning as ANSI_X3.4-1968
> characters (which are what we learned to love as 7-bit ASCII). So
> iconv fails.
> 
> My remedy proposal
> 
>   xorriso ... -as mkisofs -input-charset UTF-8 -output-charset UTF-8
> ...

Do there need to be any UTF-8 locales installed (or any locales for
that matter) for this to work? My guess is no.

> turned out to be a valid alternative to
> 
>   LANG=en_US.UTF-8
> 
> Using above charset options unconditionally would make obsolete the
> warning in Glenn Washburn's patch at least for the xorriso runs:
> 
>   +elif [ -n "${LANG##*UTF*}" ]; then
>   +   echo "WARNING: LANG=$LANG appears to not be unicode,
> international file test may fail."
> 
> It would still be beneficial to default LANG="" to LANG=en_US.UTF-8
> because 7-bit ASCII is really an odd character interpretation for Unix
> filenames. (HP-UX from 1986 maybe ...)

Correct me if I'm wrong, but this logic about defaulting LANG is more
general than the iso9660_test. Would you agree that this should be
applied to the tests a a whole?

If yes, then I should probably remove this patch altogether and put
this code elsewhere.

--------------------------------------------------------------------------
> 
> Noob question:
> How can i force "make check" to do something at its second run ?

I might be answering this question below.

--------------------------------------------------------------------------
> Long story:
> 
> On a Debian 10 with a freshly made GRUB git clone i see indeed a
> failure of iso9660_test with mutilated volume id and a mutilated
> exotic filename after
> 
>   make SUBDIRS= LANG= TESTS=iso9660_test check
> 
> On the same machine with the same xorriso, the same file name from the
> log and the volume id taken from the script, i get with
> 
>   LANG= xorriso -as mkisofs -o test.iso -V "...exotic.characters..."
> test_tree
> 
> a somewhat confused xterm (or libreadline) but
> 
>   xorriso -indev test.iso -find / --
> 
> reports the exotic volume id and the exotic filename.
> 
> The "make ... check" run says on the second try
>   make[3]: 'iso9660_test' is up to date.
> but i can reproduce the problem by
> 
>   LANG= ./grub-fs-tester joliet

Yes the iso9660_test is up to date in the sense that it does not need
to be generated from iso9660_test.in. However, "make check" should
still be running the test. IOW, the tests are always rerun, there is no
make rule that does not run tests based on them having been perviously
run (that I know of).

> The trigger for the file name mutilation seems to be the mkisofs
> emulation option for creating a Joliet tree
>   -J
> With default settings of mount(8) or xorriso's file name display the
> problem becomes visible only if no Rock Ridge info was added to the
> ISO 9660 tree. This is caused by xorriso command
>   --rockridge off
> 
> Nevertheless file(1) and xorriso still see the exotic volume id.
> Binary inspection shows that the ISO 9660 volume id (at offset 32768
> + 40) is originally exotic, but the Joliet volume id (at offset 34816
> + 40) was mutilated.
> 
> So obviously the problem sits in the Joliet name conversion to UCS-2.
> It is inavoidable if the file names do not match the input character
> set (defaulted from locale) because the Unix file names cannot be
> just copied to Joliet directory records.
>
> Now that i can reproduce the problem i can also confirm that my remedy
> proposal would work as expected. It preservies the exotic names in the
> Joliet tree and the exotic volume id in the Joliet supplementary
> volume descriptor.

As i understand it, your proposed change is entirely with in
grub-fs-tester. Would you submit a patch with the changes you've
outlined that would fix this issue for when LANG is empty? Based on the
work you've done, I think it makes most sense for you to author the
patch. And I'll test/review it.

Glenn