Re: Managing data files in workflows

[Ricardo Wurmus]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Hi Ricardo,
>
> > We can fix the problem with symlinks by restoring the target of 
> > the link
> > instead of the link itself, but I feel that we need to take a 
> > step back
> > and consider what this cache is really to be used for.
>
> Indeed, and I have to admit that this isn't clear to me 
> yet. What is it
> supposed to protect against? Modification of files by other 
> processes of
> the workflow? Modification of files outside of the workflow? 
> Both?
>
> For the second situation (modification outside of the workflow), 
> I think
> it would be sufficient to store a checksum, and terminate the 
> workflow
> with an error if it detects such tampering.
>
> The first situation is more difficult. There are actually two 
> cases:
>  1. The workflow intentionally updates files as it proceeds.
>  2. The workflow modifies a file by mistake.
>
> Only the workflow author can make the distinction, so this needs 
> some
> specific input syntax. Case 2 could then again be handled by a 
> simple
> checksum test for signalling an error.
>
> This leaves case 1, for which the only good solution is to make 
> a copy
> of the file at the end of each process, and restore it in later 
> runs.

Yes, you are right.  On wip-drmaa I changed the cache to never 
symlink.  It either hardlinks or copies.  This solves the 
immediate problem.

Yes, the semantics of hardlink/copy differ, but since our 
assumption is that intermediate files are reproducible, we can 
ignore this at this point.

I want to make the cache store/restore actions configurable, 
though, so that you can implement whatever caching method you want 
(including caching by copying to AWS S3).  

I’d like to introduce modifiers “immutable” and “mutable”, so that 
you can write “immutable file "whatever" you "want"” etc. 
“immutable” would take care of recording hashes and checking 
previously recorded hashes in a local state directory.

--
Ricardo