Re: [gwl-devel] [GWL] (random) next steps?

[Ricardo Wurmus]

Hi simon,

>> > 6.
>> > The graph of dependencies between the processes/units/rules is written
>> > by hand. What should be the best strategy to capture it ? By files "à
>> > la" Snakemake ? Other ?
>>
>> The GWL currently does not use the input information provided by the
>> user in the data-inputs field.  For the content addressible store we
>> will need to change this.  The GWL will then be able of determining that
>> data-inputs are in fact the outputs of other processes.
>
> Hum? nice but how?
> I mean, the graph cannot be deduced and it needs to be written by
> hand, somehow. Isn't it?

We can connect a graph by joining the inputs of one process with the
outputs of another.

With a content addressed store we would run processes in isolation and
map the declared data inputs into the environment.  Instead of working
on the global namespace of the shared file system we can learn from Guix
and strictly control the execution environment.  After a process has run
to completion, only files that were declared as outputs end up in the
content addressed store.

A process could declare outputs like this:

    (define the-process
      (process
        (name 'foo)
        (outputs
         '((result "path/to/result.bam")
           (meta   "path/to/meta.xml")))))

Other processes can then access these files with:

    (output the-process 'result)

i.e. the file corresponding to the declared output “result” of the
process named by the variable “the-process”.

The question here is just how far we want to take the idea of “content
addressed” – is it enough to take the hash of all inputs or do we need
to compute the output hash, which could be much more expensive?

--
Ricardo