Re: [grub-fs-tester.in] zisofs test looks unsuitable

[Glenn Washburn]

On Thu, 26 Aug 2021 22:07:29 +0200
"Thomas Schmitt" <scdbackup@gmx.net> wrote:

> Hi,
> 
> i managed to get the ziso9660 test running with actual compression.
> 
>   ./grub-fs-tester ziso9660
> 
> now produces an ISO image but seems not to be happy with it.
> 
> If i get it right then it sees the compressed size 1124519 of file
> "1.img" but expects its uncompressed size 5242879.
> Linux mount sees
>   -rw-r--r-- 2 thomas thomas 5242879 Aug 26 21:52 1.img
> 
> So grub-fstest.c CMD_LS seems not yet ready for dealing with zisofs.

This confirms my suspicions. I think the changes to get the test
working are worthy of inclusion so that the tests are ready when this
feature gets implemented. I'm not familiar with that code nor the
format details, but it sounds like you might be able to determine if
its something somewhat easy to add to GRUB.

> This is the change which brought me that far, including a draft for a
> commit message:
> ============================================================================
> 
> The grub-fs-tester test "ziso9660" failed early with a xorriso error
> because it was not in the list of tests which tolerate at most 32
> bytes of FSLABEL.
> The xorriso run of ziso9660 did not actually compress the data files
> because the xorriso command for installing zisofs filters was applied
> before any files had populated the emerging ISO image.
> So add "ziso9660" where other ISO 9660 tests get a shorter FSLABEL and
> re-arrange the commands of the xorriso run to bring -add and -zisofs
> before -set_filter_r.
> 
> Signed-off-by: Thomas Schmitt <scdbackup@gmx.net>
> 
> diff --git a/tests/util/grub-fs-tester.in
> b/tests/util/grub-fs-tester.in index bfc425e1f..834c05728 100644
> --- a/tests/util/grub-fs-tester.in
> +++ b/tests/util/grub-fs-tester.in
> @@ -314,7 +314,8 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE"
> "$MAXLOGSECSIZE" 1); do FSLABEL="grub_;/testéтi
> u😁莽茝кириrewfceniuewruevrewnuuireurevueurnievrewfne";; # FS
> LIMITATION: afs and iso9660 label is at most 32 UTF-8 characters
> x"afs" | xiso9660 | xrockridge | xrockridge_joliet\
> -                     | xiso9660_1999 | xrockridge_1999 |
> xrockridge_joliet_1999)
> +                   | xiso9660_1999 | xrockridge_1999\
> +                   | xrockridge_joliet_1999 | xziso9660)
>                    FSLABEL="gr_;/é莭莽😁кирит u";;
>                   # FS LIMITATION: bfs label is at most 32 UTF-8
> characters # OS LIMITATION: bfs label can't contain ; or /
> @@ -1020,7 +1021,7 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE"
> "$MAXLOGSECSIZE" 1); do (cd "$MASTER"; find . | cpio -o -H "$(echo
> ${fs} | sed 's@^cpio_@@')" > "${FSIMAGEP}0.img" ) ;; x"ziso9660")
>                   FSUUID=$(date -u +%Y-%m-%d-%H-%M-%S-00);
> -                 xorriso -compliance rec_mtime -set_filter_r
> --zisofs -- -zisofs default -as mkisofs -iso-level 3 -graft-points -R
> -J -joliet-long -V "$FSLABEL" --modification-date=$(echo ${FSUUID} |
> sed 's/-//g;') -o "${FSIMAGEP}0.img"  -- -set_filter_r --zisofs --
> -zisofs default -add /="$MASTER" ;;
> +                 xorriso -compliance rec_mtime -as mkisofs
> -iso-level 3 -graft-points -R -J -joliet-long -V "$FSLABEL"
> --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o
> "${FSIMAGEP}0.img" -- -add /="$MASTER" -- -zisofs default
> -set_filter_r --zisofs / -- ;; x"iso9660") FSUUID=$(date -u
> +%Y-%m-%d-%H-%M-%S-00); xorriso --rockridge off -compliance rec_mtime
> -as mkisofs -iso-level 3 -graft-points -V "$FSLABEL"
> --modification-date=$(echo ${FSUUID} | sed 's/-//g;') -o
> "${FSIMAGEP}0.img" /="$MASTER"  ;;
> 
> ============================================================================
> 
> Here is a run with this change:
> ============================================================================
> 
> $ ./grub-fs-tester ziso9660
> GNU xorriso 1.5.4.pl02 : RockRidge filesystem manipulator, libburnia
> project.
> 
> Drive current: -outdev
> 'stdio:/tmp/tmp.iIXC5zIfEd/ziso9660_512_512_1_0.img' Media current:
> stdio file, overwriteable Media status : is blank
> Media summary: 0 sessions, 0 data blocks, 0 data,  400g free
> xorriso : WARNING : -volid text problematic as automatic mount point
> name xorriso : WARNING : -volid text is too long for Joliet (31 > 16)
> xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA
> 119 rules Added to ISO image: directory
> '/'='/tmp/tmp.iIXC5zIfEd/master' xorriso : UPDATE :     227 files
> added in 1 seconds xorriso : UPDATE :      10 file filters processed
> in 1 seconds libisofs: WARNING : Cannot add /longsym to Joliet tree.
> Symlinks can only be added to a Rock Ridge tree. libisofs: WARNING :
> Cannot add /psym to Joliet tree. Symlinks can only be added to a Rock
> Ridge tree. libisofs: WARNING : Cannot add /sdir/ssym to Joliet tree.
> Symlinks can only be added to a Rock Ridge tree. libisofs: WARNING :
> Cannot add /sdir/usym to Joliet tree. Symlinks can only be added to a
> Rock Ridge tree. libisofs: WARNING : Cannot add /sym to Joliet tree.
> Symlinks can only be added to a Rock Ridge tree. libisofs: WARNING :
> Filename not suitable for Joliet character set UCS-2 :
> "éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒" libisofs: WARNING : Filename
> not suitable for Joliet character set UCS-2 :
> "Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝" libisofs: WARNING : Cannot add
> /Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝 to Joliet tree. Symlinks can
> only be added to a Rock Ridge tree. xorriso : UPDATE : Writing:
> 1658s   32.2%   fifo   0%  buf  50% libisofs: WARNING : Sum of
> filenames not suitable for Joliet character set UCS-2: 2 ISO image
> produced: 4978 sectors Written to medium : 5152 sectors at LBA 32
> Writing to 'stdio:/tmp/tmp.iIXC5zIfEd/ziso9660_512_512_1_0.img'
> completed successfully.

It seems to me that we should only be testing ziso9660 here, not joliet
or Rock Ridge extensions (although perhaps a complete testing of file
compression should include these tests??). So to get rid of these
warnings the I'm thinking we should not be building the iso with Joliet
or Rock Ridge extensions.

> Device proc: Filesystem type procfs - Sector size 512B - Total size
> 0KiB Device loop0: Filesystem type iso9660 - Label `gr_;/é莭莽😁кирит
> u' - Last modifition time 2021-08-26 19:17:46 Thursday, UUID
> 2021-08-26-19-17-46-00 - Sector size 512B - Total size 10368KiB
> Device host: Filesystem type hostfs - Sector size 512B - Total size
> 0KiB
> 
> LIST FAIL
> DIR          20210826191746 ./
> DIR          20210826191746 ../
> DIR          20210826191745 0/
> 1124311      20210826191746 1.img
> 257343       20210826191746 american-english
> 1124219      20210826191746 CaSe
> 1124683      20210826191746 cAsE
> 1124311      20210826191746 hard
> 1124653      20210826191746 longsym
> 1124353      20210826191746 psym
> 1124653      20210826191746
> qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
> DIR          20210826191746 sdir/ 1124311      20210826191746 sym
> 1124813      20210826191746 .?*\!"#%@$%&'()+ ,-.:;<=>^{_}[]`|~.
> 1124638      20210826191746 éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒
> 1124638      20210826191746 Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝
> total 31692
> drwxr-xr-x 3 thomas thomas    4096 Aug 26 19:17 0
> -rw-r--r-- 2 thomas thomas 5242879 Aug 26 19:17 1.img
> -rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 CaSe
> -rw-r--r-- 1 thomas thomas  972398 Aug 26 19:17 american-english
> -rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17 cAsE
> -rw-r--r-- 2 thomas thomas 5242879 Aug 26 19:17 hard
> lrwxrwxrwx 1 thomas thomas     227 Aug 26 19:17 longsym ->
> qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
> lrwxrwxrwx 1 thomas thomas     806 Aug 26 19:17 psym ->
> /0//1/2/3//4/5/6//7/8/9//10/11/12//13/14/15//16/17/18//19/20/21//22/23/24//25/26/27//28/29/30//31/32/33//34/35/36//37/38/39//40/41/42//43/44/45//46/47/48//49/50/51//52/53/54//55/56/57//58/59/60//61/62/63//64/65/66//67/68/69//70/71/72//73/74/75//76/77/78//79/80/81//82/83/84//85/86/87//88/89/90//91/92/93//94/95/96//97/98/99//100/101/102//103/104/105//106/107/108//109/110/111//112/113/114//115/116/117//118/119/120//121/122/123//124/125/126//127/128/129//130/131/132//133/134/135//136/137/138//139/140/141//142/143/144//145/146/147//148/149/150//151/152/153//154/155/156//157/158/159//160/161/162//163/164/165//166/167/168//169/170/171//172/173/174//175/176/177//178/179/180//181/182/183//184/185/186//187/188/189//190/191/192//193/194/195//196/197/198//199/200/201//202/203/204//205/206/207//208/209/p.img
> -rw-r--r-- 1 thomas thomas 5242879 Aug 26 19:17
> qwertzuiopasdfghjklyxcvbnm1234567890qwertzuiopasdfghjklyxcvbnm1234567890oiewqfiewioqoiqoiurqruewqoiuwoieoiiuewqroreqiufieiuwrnureweriuvceoiroiewqoiricdsalkcndsakfirefoiwqeoircorejwoijfreoijojoiewjfwnfcoirenfoirefnreoifenoiwfnoi
> drwxr-xr-x 2 thomas thomas    4096 Aug 26 19:17 sdir lrwxrwxrwx 1
> thomas thomas       5 Aug 26 19:17 sym -> 1.img -rw-r--r-- 1 thomas
> thomas 5242879 Aug 26 19:17 éàèüöäëñкирилица莭莽茝Ελληνικά😁😜
> lrwxrwxrwx 1 thomas thomas      72 Aug 26 19:17
> Ελληνικάкирилица😁😜😒éàèüöäëñ莭莽茝 ->
> éàèüöäëñкирилица莭莽茝Ελληνικά😁😜😒 $ echo $? 1
> 
> ============================================================================

Glenn