|
From: | Ricardo Wurmus |
Subject: | Re: Managing data files in workflows |
Date: | Mon, 03 May 2021 15:47:22 +0200 |
User-agent: | mu4e 1.4.15; emacs 27.2 |
zimoun <zimon.toutoune@gmail.com> writes:
Hi,On Mon, 03 May 2021 at 11:18, Ricardo Wurmus <rekado@elephly.net> wrote:I’d like to introduce modifiers “immutable” and “mutable”, so that you can write “immutable file "whatever" you "want"” etc. “immutable” would take care of recording hashes and checking previously recorded hashes in a local state directory.Ah, maybe it is same idea that we discussed some days ago on #guix-hpc.To me, everything should be “immutable“ and stored “somewhere”.Somehow, you do not need to hash all the contents but only hash all theinputs hashes and the ’process’ itself.
This is in fact what we do now. Every process code snippet is hashed, and we compute the hashes of the chain of processes to determine whether to re-run something or not.
However, I miss what “mutable” could be?
It’s a very generous assumption that intermediate output files are reproducible. Often that’s not actually the case. Having a “mutable” modifier would allow us to indicate that this is in fact the case and require recomputation.
-- Ricardo
[Prev in Thread] | Current Thread | [Next in Thread] |