[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: Guix Workflow Language ?
From: |
Cook, Malcolm |
Subject: |
RE: Guix Workflow Language ? |
Date: |
Thu, 25 Jan 2018 22:23:34 +0000 |
Hi,
Watching this thread and trying to take the pulse of GWL.
Where should I look?
https://git.roelj.com/guix/gwl has little documentation - it does say " GWL has
a built-in getting-started guide. To use it, run: guix workflow
--web-interface" - but supposing we just want to read some documentation
https://www.guixwl.org/ is 503
Workflow management with GNU Guix
https://archive.fosdem.org/2017/schedule/event/guixworkflowmanagement/ is
interesting but not documentation
Can someone please catch me up?
Thx,
address@hidden
> -----Original Message-----
> From: Guix-devel [mailto:address@hidden
> On Behalf Of Roel Janssen
> Sent: Thursday, January 25, 2018 4:05 PM
> To: zimoun <address@hidden>
> Cc: address@hidden
> Subject: Re: Guix Workflow Language ?
>
>
> zimoun writes:
>
> > Dear Roel,
> >
> > Thank you for your comments.
> >
> > I was imaging your point 2. And the softwares come from Guix.
> > The added benefit was: a controlled and reproducible environment.
> > In other words, the added benefit came from the GuixWorkflow (the
> > engine of workflow), and not from the Language (lisp EDSL).
> > But maybe it is a wrong way.
>
> I get that point. Maybe it's then a better idea to write the workflow
> in CWL (like you would do), and use Guix to generate Docker containers.
>
> Then you do get the benefit of Guix's strong reproducibility and
> composability forscientific software, plus you get to keep writing the
> workflow in CWL. :-)
>
> >
> >>From my experience, the classical strategy of writing pipelines is to
> > adapt an already existing workflow for one another particular
> > question. We fetch bits here and there, do some ugly and dirty hacks
> > to have some results; then depending on them, a cleaner pipeline is
> > written (or not! :-) or other pieces are tested.
> > Again from my experience, there is (at least) 3 issues: the number of
> > tools to learn and know enough to be able to adapt; the bits/pieces
> > already available; the environment/dependencies and how they are
> > managed.
> >
> > In this context, since 'lispy' syntax is not mainstream (and will
> > never be), it appears to me as a hard position. That's why I asked if
> > a Guix-backend workflow engine for CWL specs is doable. Run CWL specs
> > workflow on the top of the GWL engine.
>
> This is a good question, but how can you describe the origin of a
> software package in CWL? In the GWL, we use the Scheme symbols, and
> the
> Guix programming interface directly, but that is unavailable in CWL.
>
> This is a real problem that I don't see we can easily solve.
>
>
> >
> > However, I got your point, I guess.
> > You mean: it is a lot of work with unclear benefits over existing engines.
>
> So, I think it's impossible to express the deployment of a software
> program in CWL. It is not as expressive as GWL in this regard.
> Translating to a precise Guix package recipe and its dependencies is
> very hard from what we can write in CWL.
>
> If I am mistaken here, please let me know. Maybe we can figure
> something out.
>
> >
> >
> > Therefore, your point 1. reverses "my issue".
> > Once the pipeline is well-established, write it with GWL! :-)
> > Next, if it is possible to convert this GWL specs pipeline to CWL one
> > [+ Docker] (with softwares coming from Guix), then we can enjoy the
> > CWL-world engine capabilities.
> > The benefit of that is from two sides: run the pipeline with different
> > engines; and produce a clean docker image.
> >
> > So , instead of working on improving the GWL engine (adding features
> > about efficiency, Grid, Amazon, etc.) which is a very tough task, the
> > doable plan would be to add an "exporter".
> > Right ?
>
> The plan is to implement back-ends, or 'process-engines' for GWL to work
> with AWS, Kubernetes, Grid (this one is already supported).
>
> These back-ends are surprisingly easy to write, because the Guix
> programming interface allows us to generate virtual machines,
> containers, or simply store items if Guix is available locally.
>
> We also implemented a Bash-engine that can generate Bash scripts for
> every step of the workflow. That in combination with the variety of
> deployment options solves most of the challenges.
>
> >
> >
> > Another question, do you think it is doable to write "importers" ?
> >
> > I am not sure that the metaphor is good enough, but do you think it is
> > a feasible goal from the existing GWL to go towards a kind of `Pandoc
> > of workflows` ? also packing the softwares.
> >
> > And a start should be:
> > - write a parser for (subset of) CWL yaml file and obtain the GWL
> > representation of the workflow
> > - write a exporter to CWL + Docker image
> >
> > What do you think ?
>
> Maybe. But in CWL we cannot describe precise software packages. So
> translating these things to Guix is hard.
>
> >
> >
> > About the parser, I haven't found yet an easy-to-use Guile lib for
> > parsing YAML-like files. Any pointer ? Adapt some Racket ones ?
>
> I don't know of one, sorry.
>
>
> > Thank you for your insights.
> >
> > All the best,
> > simon
>
> Thanks!
>
> Kind regards,
> Roel Janssen
- Guix Workflow Language ?, zimoun, 2018/01/24
- Re: Guix Workflow Language ?, Roel Janssen, 2018/01/24
- Re: Guix Workflow Language ?, zimoun, 2018/01/25
- Re: Guix Workflow Language ?, Ricardo Wurmus, 2018/01/25
- Re: Guix Workflow Language ?, Roel Janssen, 2018/01/25
- RE: Guix Workflow Language ?,
Cook, Malcolm <=
- Re: Guix Workflow Language ?, Pjotr Prins, 2018/01/26
- Re: Guix Workflow Language ?, zimoun, 2018/01/29
- Re: Guix Workflow Language ?, Ricardo Wurmus, 2018/01/29