RE: [Rd] using R as SHELL in gnu make

[Cook, Malcolm]

Simon,

>Malcom,
 >
 >On Feb 3, 2014, at 12:43 PM, Cook, Malcolm <address@hidden> wrote:
 >
 >> Simon,
 >> Reviving an old thread . and moving it to your preferred forum for such 
 >> discussion ..
 >> I am re-invigorated to improve my old approach to allowing GNU-make's 
 >> recipes to be written in R language and evaluated by a
 >running and configured Rserve.
 >> Having unix local sockets working and the new RS.eval makes this make more 
 >> reasonable.
 >> I can now for instance start Rserve using a socket path in the current 
 >> directory, like this:
 >> (R_DEFAULT_PACKAGES=MASS;R CMD Rserve --vanilla --silent --RS-socket 
 >> ${PWD}/.rs-socket --RS-workdir ${PWD})
 >> Rserv started in daemon mode.
 >> ...and then query it from the command line like this:
 >> Rscript --default-packages=RSclient -e 
 >> 'RS.eval(RS.connect(port=0,host=".rs-socket"),sessionInfo())'
 >> R version 3.0.2 (2013-09-25)
 >> Platform: x86_64-unknown-linux-gnu (64-bit)
 >>
 >> locale:
 >> [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               
 >> LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 >LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8    
 >   LC_NAME=C                  LC_ADDRESS=C
 >LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
 >>
 >> attached base packages:
 >> [1] utils     stats     graphics  grDevices base
 >>
 >> other attached packages:
 >> [1] MASS_7.3-29
 >> This works great and as documented.
 >> However, in order to make the Rserve process evaluate a parsed stream of 
 >> expressions, as needed to have it be the back-end for
 >GNU-make recipes, I feel I am going through too many contortions. The 
 >following example works, but "feels wrong" (tm)
 >> echo $'print("hello")\nprint("world")\nsetwd("..")\ngetwd()' | Rscript 
 >> --default-packages=RSclient -
 >e'lines=readLines("stdin");expr.list=parse(text=lines);RS.eval(RS.connect(port=0,host=".rs-
 >socket"),as.call(list(quote(eval),expr.list)),lazy=FALSE)'
 >> [1] "hello"
 >> [1] "world"
 >> [1] "/home/mec/local/mecsrc/mkr"
 >> Can you advise if there is a simpler, more direct approach than the above.
 >
 >I'm not sure what would constitute a more direct approach. You could push the 
 >parsing to the remote side as well if that helps, but it
 >will make the code more ugly since you have to use eval(paste(text=...)).

OK.  Got it.  I'll stay with what I've rolled....

 >
 >Note, however, that the output of the print() statements goes out of the 
 >Rserve process, not the calling process, so if you want that,
 >you probably have to use capture.output() as well.

Yeah, I know, thanks, the 'print' statements were just proof of idea.... if I 
in fact _use_ this approach the make recipes will be building reports and 
plotting to disk and/or serializing objects with saveRDS and the like.

>> Also, my recipes will want to generate output files which of course are 
>> being placed in the working directory, which is named
 >'connNNNNN' subordinate to the -RS-workdir. Other than explicitly calling 
 >`setwd("..")`, is there perhaps a value in having a flag to
 >RServe which avoids creating connection specific directories?
 >
 >If you set the workdir setting to '' then Rserve won't manage the working 
 >directory.

Perfect!

>There is, however, one additional issue - the daemon version of Rserve changes 
>the working directory to / as part of the
 >daemonization process, so you won't get to pwd in that case. 
[Cook, Malcolm] 

Oh, wait.   Not perfect...

> One way to deal with that is to use in-R Rserve process, e.g. start it via
 >
 >Rscript -e 'Rserve::run.Rserve(workdir="",socket=".rs-socket")'
 >
 >then you'll and up in the working directory.

I think rather I'll just prepend `parse(text=setwd(${CURDIR}))` to my array of  
 for each connection.... 

... It works a treat....

Thanks much for the consult,

~Malcolm 
        
 >
 >Cheers,
 >Simon
 >
 >
 >
 >
 >> Thanks for your consideration, and, thanks for Rserve!
 >> ~Malcolm Cook
 >>
 >> >-----Original Message-----
 >> >From: Cook, Malcolm
 >> >Sent: Thursday, September 22, 2011 9:14 AM
 >> >To: 'Simon Urbanek'
 >> >Subject: RE: [Rd] using R as SHELL in gnu make
 >> >
 >> >Simon,
 >> >
 >> >sorry about the old link.  I am using the latest from sourceforge, but 
 >> >when I composed the email I took top hit from Google.  My
 >> >mistake.  Won't happen again.
 >> >
 >> >I use Rscript all the time to write stand alone scripts, but don't see the 
 >> >advantage here.  Can you elaborate what you see it to be?
 >> >Make does not pass recipes to its SHELL via stdin, rather, as options 
 >> >which must be preceeded by '-e'.
 >> >
 >> >Cheers,
 >> >
 >> >~Malcolm
 >> >
 >> >
 >> >-----Original Message-----
 >> >From: Simon Urbanek [mailto:address@hidden
 >> >Sent: Thursday, September 22, 2011 9:01 AM
 >> >To: Cook, Malcolm
 >> >Subject: Re: [Rd] using R as SHELL in gnu make
 >> >
 >> >
 >> >On Sep 22, 2011, at 9:29 AM, Cook, Malcolm wrote:
 >> >
 >> >> OK,
 >> >>
 >> >> I now have a working version of setting SHELL=R in a GNU make script, 
 >> >> allowing make's recipes to be written in R, that works out
 >> >most of the complications.
 >> >>
 >> >
 >> >Shouldn't that be SHELL=Rscript?
 >> >
 >> >
 >> >> I plan to clean it up a little more and blog about it soon here: 
 >> >> http://malcook-gedanken.blogspot.com/ - I'll follow up here  when I
 >> >do.
 >> >>
 >> >> The approach optionally allows evaluating the recipes using a running 
 >> >> Rserve (http://rosuda.org/Rserve/), avoiding initialization
 >time
 >> >and allowing pre-loading of R libraries common to multiple recipes.
 >> >>
 >> >
 >> >Thank you for referencing Rserve, but can you, please, use links that are 
 >> >not 7 years out of date? I have left Ausgburg (rosuda) in
 >2004
 >> >...
 >> >
 >> >Thanks,
 >> >Simon
 >> >
 >> >
 >> >> The approach however does NOT provide any special mechanism to preserve 
 >> >> state between recipes,
 >> >> Rather, recipes may create the make rule's target as a state dump by 
 >> >> setting with `file='$@'`  in a call to `save` (or `save.image`,
 >> >`dump`, as desired).
 >> >> Other make rules may then call `load('$<')` when the previously saved 
 >> >> dump is a pre-requisite to the rule.
 >> >>
 >> >> I'm still not sure if it is not more of an amusement and am interested 
 >> >> in all thoughts on this, and welcome any suggestions for
 >> >example applications that I might include when I go to blog it up...
 >> >>
 >> >> Cheers,
 >> >>
 >> >> ~Malcolm
 >> >>
 >> >>
 >> >> -----Original Message-----
 >> >> From: Paul Gilbert [mailto:address@hidden
 >> >> Sent: Tuesday, September 20, 2011 8:32 AM
 >> >> To: Cook, Malcolm; 'address@hidden'; 'address@hidden'
 >> >> Subject: RE: using R as SHELL in gnu make
 >> >>
 >> >> Other than the RServe part, I do this all the time. It works well. 
 >> >> Perhaps we can put together some notes off-line and then bring it
 >> >back to the list.
 >> >>
 >> >> Paul
 >> >>
 >> >>> -----Original Message-----
 >> >>> From: address@hidden [mailto:address@hidden
 >> >>> project.org] On Behalf Of Cook, Malcolm
 >> >>> Sent: September 19, 2011 6:35 PM
 >> >>> To: 'address@hidden'; 'address@hidden'
 >> >>> Subject: [Rd] using R as SHELL in gnu make
 >> >>>
 >> >>> I am intrigued by the possibility of using R as the SHELL in a (Gnu)
 >> >>> makefile (instead of /bin/sh).  (c.f.
 >> >>> http://www.gnu.org/software/make/manual/make.html#Choosing-the-Shell)
 >> >>>
 >> >>> Well, rather, I would like the makefile's SHELL to be a command which
 >> >>> communicated with an R process.
 >> >>>
 >> >>> The makefile targets/prerequistes would, as always, be OS files, which
 >> >>> would be written/read using standard R file IO.
 >> >>>
 >> >>> The makefile's "recipe"s would be written in R (instead of the usual
 >> >>> shell).
 >> >>>
 >> >>> The R process would be able to be initiated by `load`ing one or more R
 >> >>> datasets, libraries or entire images.
 >> >>>
 >> >>> The R process would be able to accumulate state as the makefile
 >> >>> progressed.  The recipe's would be able to refer to that state,
 >> >>> allowing conditional execution.
 >> >>>
 >> >>> The R process would optionally be saved as an image of on job
 >> >>> termination/completion.
 >> >>>
 >> >>> The R process might be managed using the RServe package, and would need
 >> >>> to be initiated once only, when the makefile was first invoked.
 >> >>>
 >> >>> I would appreciate learning if anyone had any success, informative
 >> >>> failures, or other lore that may help in (or dissuade me from)
 >> >>> embarking on attempt this.
 >> >>>
 >> >>> Thanks,
 >> >>>
 >> >>> Malcolm Cook
 >> >>> Stowers Institute for Medical Research
 >> >>>
 >> >>> ______________________________________________
 >> >>> address@hidden mailing list
 >> >>> https://stat.ethz.ch/mailman/listinfo/r-devel
 >> >> ====================================================================================
 >> >>
 >> >> La version française suit le texte anglais.
 >> >>
 >> >> ------------------------------------------------------------------------------------
 >> >>
 >> >> This email may contain privileged and/or confidential information, and 
 >> >> the Bank of
 >> >> Canada does not waive any related rights. Any distribution, use, or 
 >> >> copying of this
 >> >> email or the information it contains by other than the intended 
 >> >> recipient is
 >> >> unauthorized. If you received this email in error please delete it 
 >> >> immediately from
 >> >> your system and notify the sender promptly by email that you have done 
 >> >> so.
 >> >>
 >> >> ------------------------------------------------------------------------------------
 >> >>
 >> >> Le présent courriel peut contenir de l'information privilégiée ou 
 >> >> confidentielle.
 >> >> La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute 
 >> >> diffusion,
 >> >> utilisation ou copie de ce courriel ou des renseignements qu'il contient 
 >> >> par une
 >> >> personne autre que le ou les destinataires désignés est interdite. Si 
 >> >> vous recevez
 >> >> ce courriel par erreur, veuillez le supprimer immédiatement et envoyer 
 >> >> sans délai à
 >> >> l'expéditeur un message électronique pour l'aviser que vous avez éliminé 
 >> >> de votre
 >> >> ordinateur toute copie du courriel reçu.
 >> >> ______________________________________________
 >> >> address@hidden mailing list
 >> >> https://stat.ethz.ch/mailman/listinfo/r-devel