[bug#52555] [RFC PATCH 0/3] Decentralized substitute distribution with ERIS

[Ludovic Courtès]

Hi!

pukkamustard <pukkamustard@posteo.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: gzip
>>> FileSize: 67363
>>> ERIS: 
>>> urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: zstd
>>> FileSize: 64917
>>> ERIS: 
>>> urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>>
>> Do we really need one URN per compression method?  Couldn’t we leave
>> compression (of individual chunks, possibly) as a “detail” handled by
>> the encoding or the transport layer?
>>
>
> I agree that it would be nice to leave this to the encoding layer as
> that would allow certain optimizations (e.g. de-duplication).
>
> Unfortunately, we haven't figured out yet what the most suitable
> compression/format would be. Something like EROSFS seems good (as it
> aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
> just an archive format and there do not seem to be any libraries that we
> could use to neatly integrate. It seems possible to block-align a Tar
> archive, but that seems a bit hackey [2]. Other things to look into
> might be Tarlz [3] and ZPAQ [4].

Yeah.  Though it may be that deduplication at the block level doesn’t
buy us much.  That was the conclusion I reached a long time ago[a], and
also seems to be supported by the recent guix-daemon deduplication
improvements[b].

[a] https://hal.inria.fr/hal-00187069/en
[b] https://issues.guix.gnu.org/24937#20-lineno0

> To get started I suggest just using one of the compressions/formats
> already in Guix. zstd seems to be a reasonable choice (for the same
> reasons why it makes sense to use zstd with `--discover` [5]).
>
> Does that sound like a plan?

Sure!

> I meant to send in a V2 that does this before going on holidays, but I'm
> afraid I won't make it. V2 will come in early January!

Alright, we’ll see!  :-)

Until then, enjoy your holidays!

Ludo’.