Re: Shared memory between parallel threads

[Olaf Till]

On Tue, Apr 22, 2014 at 02:58:15PM +0000, Anton Flugge wrote:
> Hi,
> 
> I am trying to run calls to the kmeans function in parallel. In
> principal that works well with parcellfun in the parallel toolbox,
> however, the matrix that I want to use kmeans on is quite large (a
> couple of GB), and parcellfun creates separate copies of the
> workspace/variables for each parallel thread, which means I run out
> of memory quicker than I run out of processor cores. Now kmeans only
> needs read access, and no write access, to the input data, which
> means in principal it shouldn't be a problem to share the same
> matrix/memory between all threads, however, I could not find a
> solution anywhere on how to do that in Octave... is it possible to
> have global variables accessed by multiple parallel threads?
> 
> Thanks,
> Anton

AFAIK Linux processes use copy-on-write, so that if Octave really only
reads data it does not generate copies of that data in the child
processes. In fact, on my system:

octave:4> a = ones (500000000, 1);

'a' occupies now half of my systems RAM and 1/6 of total systems
memory including swap space. Nevertheless

octave:5> r = parcellfun (10, @ (x) sum (a), cell (10, 1))
parcellfun: 10/10 jobs done
r =

   500000000
   500000000
   500000000
   500000000
   500000000
   500000000
   500000000
   500000000
   500000000
   500000000

with 10 child processes works, although

octave:6> repmat (a, 1, 10);
error: out of memory or dimension too large for Octave's index type

indicates that 10 copies of 'a' would be too much, as expected.

Olaf

-- 
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc
Description: Digital signature