Re: Preparing for a new release

[zimoun]

Hi,

Thank you for pushing forward.


On Sun, 9 Feb 2020 at 14:01, Ricardo Wurmus <address@hidden> wrote:

> * inputs and outputs are not validated
>
>   When a process declares that it produces an output, but then doesn’t do
>   that, the next process will fail with a nasty error message.  This is
>   especially nasty when using containerization as the error is about
>   failing to map the input into the container.
>
>   Processes should automatically validate their inputs and outputs.
>   Since inputs and outputs could technically be something other than
>   files I’m not sure exactly how to do this.

>From my understanding, Snakemake uses only files as inputs/outputs.
But I do not know what happens if 'rule 1' claims to output 'file' and
'rule 2' says its input is 'file' and then 'rule 1' never produces
this file 'file'; the 'rule 2' is never processed, I guess. Hum? need
to check...


>   @Roel: can you give an example of inputs / outputs that are not files?

Since UNIX considers everything as file.. ;-)

>   I remember that you suggested that inputs might be database queries,
>   for example.  I wonder if we should mark inputs and outputs with
>   types, so that the GWL can know if something is supposed to be a file
>   or something else.  …just how would “something else” be used in a
>   process?

I am not able to imagine other thing than files. But often, I would
like that these files stay in memory instead of be written on disk and
then re-read by the next process. When the IO is slow (disk full using
RAID6), it kills the workflow performance.



> * It’s not possible to select more than one tagged item
>
>   In my test workflow I’m generating a bunch of inputs by mapping over
>   an argument list.  Now the problem is that I can’t select all of these
>   inputs easily in a code snippet.  With the syntax we have I can only
>   select the first item following a tag.
>
>   To address this I’ve extended the accessor syntax, so this works now:
>
> --8<---------------cut here---------------start------------->8---
> process frobnicate
>   packages "frobnicator"
>   inputs
>     . genome: "hg19.fa"
>     . samples: "a" "b" "c"
>   outputs
>     . "result"
>   # {
>     frobnicate -g {{inputs:genome}} --files {{inputs::samples}} > {{outputs}}
>   }
> --8<---------------cut here---------------end--------------->8---
>
>   Note how {{inputs::samples}} is substituted with “a b c”.  With just a
>   single colon it would be just “a”.  Single colon = single item; double
>   colon = more than one item.

I am confused by the syntax.
Well how to select the second element "b"?

Naively, I would tempt to write {{inputs:samples:2}} or {{inputs::samples:2}}.



All the best,
simon