Re: Use guix to distribute data & reproducible (data) science

[Amirouche Boubekki]

Hello again Ludovic,

On 2018-02-09 18:13, address@hidden wrote:
> Hi!
>
> Amirouche Boubekki <address@hidden> skribis:
>
> > tl;dr: Distribution of data and software seems similar.
> >        Data is more and more important in software and reproducible
> >        science. Data science ecosystem lakes resources sharing.
> >        I think guix can help.
>
> I think some of us especially Guix-HPC folks are convinced about the
> usefulness of Guix as one of the tools in the reproducible science
> toolchain (that was one of the themes of my FOSDEM talk).  :-)
>
> Now, whether Guix is the right tool to distribute data, I don’t know.
> Distributing large amounts of data is a job in itself, and the store
> isn’t designed for that.  It could quickly become a bottleneck.

What does it mean technically that the store “isn't designed for that”?

> That’s one of the reasons why the Guix Workflow Language (GWL)
> does not store scientific data in the store itself.

Sorry, I did not follow the engineering discussion around GWL.
Looking up the web brings me [0]. That said the question I am
asking is not answered there. In particular there is no rationale
for that in the design paper.

[0] http://lists.gnu.org/archive/html/guix-devel/2016-10/msg01248.html

> I think data should probably be stored and distributed out-of-band 
> using
> appropriate storage mechanisms.

Then, in a follow up mail, you reply to Konrad:

> > Konrad Hinsen <address@hidden> skribis:
>
> [...]
>
> > It would be nice if big datasets could conceptually be handled in the
> > same way while being stored elsewhere - a bit like git-annex does for
> > git. And for parallel computing, we could have special build daemons.
>
> Exactly.  I think we need a git-annex/git-lfs-like tool for the store.
> (It could also be useful for things like secrets, which we don’t want
> to have in the store.)


-

" The most basic of all human needs is the need to understand and be 
understood " Ralph Nichols