Re: [O] Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Xebar Saram]

Thx Chuck

this sounds great. could you perhaps point us to some documentation on this, or perhpas consider sharing a detailed overview of your workflow? this seems it could really fit my needs.

thx so much in advance

Z

On Thu, Jun 18, 2015 at 6:17 AM, Charles C. Berry <address@hidden> wrote:

On Wed, 17 Jun 2015, William Denton wrote:

On 17 June 2015, Xebar Saram wrote:

I do alot of modeling work that involves using huge datasets and run
process intensive R processes (such as complex mixed models, Gamms etc). in
R studio all works well yet when i use the orgmode eval on R code blocks it
works well for small simple process but 90% of the time when dealing with
complex models and bug data (up to 256GB) it will just freeze emacs/ess.
sometimes i can C-c or C-g it and other times i need to physically kill
emacs.

I've been having the same problem for a while, but wasn't able to isolate it any more than large data sets, lack of memory, and heavy CPU usage. Sometimes everything hangs and I need to power cycle the computer. :(

And you (both) have `ess-eval-visibly' set to nil, right?

I do statistical genomics, which can be compute intensive. Sometimes processes need to run for a while, and I get impatient having to wait.

I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in org-mode.

Basically, ravel will export Org mode to a format that knitr (and the like) can run - turning src blocks into `code chunks'. That allows me to set the cache=TRUE chunk option, etc. I run knitr on the exported document to initialize objects for long running computations or to produce a finished report.

When I start a session, I run knitr in the R session, then all the cached objects are loaded in and ready to use.

If I write a src block I know will take a long time to export, I export from org mode to update the knitr document and re-knit it to refresh the cache.

Mostly, I work in org-mode adding src blocks, revising existing ones, or editing text and graphics.

If you decide to try ravel I recommend the `ravel-lang' branch[2] as that will soon replace master.

HTH,

Chuck


[1] https://github.com/chasberry/orgmode-accessories
[2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang