Re: [O] Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Charles C. Berry]

On Wed, 17 Jun 2015, William Denton wrote:

> On 17 June 2015, Xebar Saram wrote:
>
> > I do alot of modeling work that involves using huge datasets and run
> > process intensive R processes (such as complex mixed models, Gamms etc). in
> > R studio all works well yet when i use the orgmode eval on R code blocks it
> > works well for small simple process but 90% of the time when dealing with
> > complex models and bug data (up to 256GB) it will just freeze emacs/ess.
> > sometimes i can C-c or C-g it and other times i need to physically kill
> > emacs.
>
> I've been having the same problem for a while, but wasn't able to isolate it 
> any more than large data sets, lack of memory, and heavy CPU usage. 
> Sometimes everything hangs and I need to power cycle the computer. :(

And you (both) have `ess-eval-visibly' set to nil, right?

I do statistical genomics, which can be compute intensive. Sometimes 
processes need to run for a while, and I get impatient having to wait.

I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in 
org-mode.

Basically, ravel will export Org mode to a format that knitr (and the 
like) can run - turning src blocks into `code chunks'. That allows me to 
set the cache=TRUE chunk option, etc. I run knitr on the exported document 
to initialize objects for long running computations or to produce a 
finished report.

When I start a session, I run knitr in the R session, then all the cached 
objects are loaded in and ready to use.

If I write a src block I know will take a long time to export, I export 
from org mode to update the knitr document and re-knit it to refresh the 
cache.

Mostly, I work in org-mode adding src blocks, revising existing ones, or 
editing text and graphics.

If you decide to try ravel I recommend the `ravel-lang' branch[2] as that 
will soon replace master.

HTH,

Chuck


[1] https://github.com/chasberry/orgmode-accessories
[2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang