Re: [PATCH v3 2/5] tests/acceptance/boot_linux_console: remove Armbian 19.11.3 bionic test for orangepi-pc machine

[Willian Rampazzo]

On Mon, Mar 22, 2021 at 2:24 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> On 3/22/21 6:12 PM, Willian Rampazzo wrote:
> > Hi Philippe,
> >
> > On Mon, Mar 22, 2021 at 1:59 PM Philippe Mathieu-Daudé <f4bug@amsat.org> 
> > wrote:
> >>
> >> On Mon, Mar 22, 2021 at 5:54 PM Philippe Mathieu-Daudé <f4bug@amsat.org> 
> >> wrote:
> >>>
> >>> Hi Willian,
> >>>
> >>> On 3/8/21 9:44 PM, Willian Rampazzo wrote:
> >>>> On Mon, Mar 8, 2021 at 5:41 PM Willian Rampazzo <wrampazz@redhat.com> 
> >>>> wrote:
> >>>>>
> >>
> >>>>>>>>> -    def do_test_arm_orangepi_uboot_armbian(self, image_path):
> >>>>>>>>> +    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
> >>>>>>>>> +                'Test artifacts fetched from unreliable 
> >>>>>>>>> apt.armbian.com')
> >>>>>>>>> +    @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage 
> >>>>>>>>> limited')
> >>>>>>>>> +    def test_arm_orangepi_bionic_20_08(self):
> >>>>>>>>> +        """
> >>>>>>>>> +        :avocado: tags=arch:arm
> >>>>>>>>> +        :avocado: tags=machine:orangepi-pc
> >>>>>>>>> +        :avocado: tags=device:sd
> >>>>>>>>> +        """
> >>>>>>>>> +
> >>>>>>>>> +        # This test download a 275 MiB compressed image and expand 
> >>>>>>>>> it
> >>>>>>>>> +        # to 1036 MiB, but the underlying filesystem is 1552 MiB...
> >>>>>>>>> +        # As we expand it to 2 GiB we are safe.
> >>>>>>>>> +
> >>>>>>>>> +        image_url = ('https://dl.armbian.com/orangepipc/archive/'
> >>>>>>>>> +                     
> >>>>>>>>> 'Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz')
> >>>>>>>>
> >>>>>>>> The Armbian 20 is not available on this link anymore. I can browse 
> >>>>>>>> just 21.
> >>>>>>>
> >>>>>>> Cat-and-mouse game *sigh*.
> >>>
> >>> 2021-03-22 17:18:10,701 download         L0067 INFO | Fetching
> >>> https://archive.armbian.com/orangepipc/archive/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz
> >>> -> /home/phil/avocado/da
> >>> ta/cache/by_location/f2eb27a12b81ce15e93f340fabbced2136af1caa/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz.nfwdzivg
> >>> 2021-03-22 17:18:11,263 asset            L0136 INFO | Temporary asset
> >>> file unavailable due to failed download attempt.
> >>> 2021-03-22 17:18:11,263 asset            L0368 ERROR| URLError: <urlopen
> >>> error [Errno 113] No route to host>
> >>>
> >>> It might be a temporary problem, but looking long term the
> >>> current setup doesn't scale IMHO.
> >>
> >> Also I just noticed I *do* have the image cached, even twice:
> >>
> >> $ find /home/phil/avocado/data/cache/by_location/ -name
> >> Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz -ls
> >>   131661 280768 -rw-rw-r--   1 phil     phil     287501560 Oct 23
> >> 12:45 
> >> /home/phil/avocado/data/cache/by_location/cac379e6a4480624b000dfde1b5d935454254ce2/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz
> >>   131761 280776 -rw-rw-r--   1 phil     phil     287501560 Feb 20
> >> 00:39 
> >> /home/phil/avocado/data/cache/by_location/aac619d9e7ea3726ebc86c159fddfb9d3f6274a5/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz
> >>
> >> But the hash is different.
> >>
> >
> > You have 2 images because you fetched them from different locations.
> > When you fetch by location, the assets go to the directory
> > representing the hash of the location, This allows, for example, the
> > fetch of multiple initrd or vmlinuz files from different locations,
> > representing different files, but with the same name. Also, this was
> > one of your requests some time ago :)
>
> My request because we were fetching different artifacts for different
> tests but all files had the same filename, i.e.:
>
> url1://path1/to/vmlinux
> url2://path2/to/vmlinux
> url3://yet/another/path/to/vmlinux
>
> I was obviously short sighted.
>
> > What Beraldo worked on in the last week was a way to remove "unused"
> > files. Your cache is a good example of files that are not necessary
> > anymore because now you are using a different location to fetch them.
>
> OK, I'll watch closely how you decide a cached file is "unused"...
>
> >
> >> Cleber, why not use the blob hash as a cache key? It is supposed to be
> >> constant...
> >>
> >
> > You can do that by setting the test to download the file by name, and
> > not by location.
>
> How so? Isn't it what I just described earlier we wanted to avoid?
>

Yep, but if you are sure there won't be another file called
`Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz` that means
another thing than what the file name describes, in this case, it can
be fetched by name. It will still check the file hash to make sure it
is the file needed, but it opens the possibility to download the same
file from different locations without duplicating it in the cache.

For this case, I think it makes sense to fetch it and add it to the
`by_name` location in the cache.

> >
> >> Cc'ing Beraldo because I guess remember he worked on improving Avocado 
> >> cache.
> >>
> >>>>>> Just to clarify here: in this patch I wanted to only make the change 
> >>>>>> to remove the test for the 19.11.3 image.
> >>>>>> And in the 3rd patch (tests/acceptance/boot_linux_console: change URL 
> >>>>>> for test_arm_orangepi_bionic_20_08)
> >>>>>> do the update for to correct the 20.08 link. So each patch fixes one 
> >>>>>> problem at a time.
> >>
> >
> >
>