Re: Shared memory between parallel threads

[c.]

On 27 Apr 2014, at 20:28, Olaf Till <address@hidden> wrote:

> On Tue, Apr 22, 2014 at 02:58:15PM +0000, Anton Flugge wrote:
>> Hi,
>> 
>> I am trying to run calls to the kmeans function in parallel. In
>> principal that works well with parcellfun in the parallel toolbox,
>> however, the matrix that I want to use kmeans on is quite large (a
>> couple of GB), and parcellfun creates separate copies of the
>> workspace/variables for each parallel thread, which means I run out
>> of memory quicker than I run out of processor cores. Now kmeans only
>> needs read access, and no write access, to the input data, which
>> means in principal it shouldn't be a problem to share the same
>> matrix/memory between all threads, however, I could not find a
>> solution anywhere on how to do that in Octave... is it possible to
>> have global variables accessed by multiple parallel threads?
>> 
>> Thanks,
>> Anton
> 
> AFAIK Linux processes use copy-on-write, so that if Octave really only
> reads data it does not generate copies of that data in the child
> processes. In fact, on my system:
> 
> octave:4> a = ones (500000000, 1);
> 
> 'a' occupies now half of my systems RAM and 1/6 of total systems
> memory including swap space. Nevertheless
> 
> octave:5> r = parcellfun (10, @ (x) sum (a), cell (10, 1))
> parcellfun: 10/10 jobs done
> r =
> 
>   500000000
>   500000000
>   500000000
>   500000000
>   500000000
>   500000000
>   500000000
>   500000000
>   500000000
>   500000000
> 
> with 10 child processes works, although
> 
> octave:6> repmat (a, 1, 10);
> error: out of memory or dimension too large for Octave's index type
> 
> indicates that 10 copies of 'a' would be too much, as expected.
> 
> Olaf
> 
> -- 

Olaf, 

I think you might be misinterpreting the results of your example,
look at this:

>> a = ones (500000000, 1);
>> for ii = 1:10
eval (sprintf ("b_%d = a; b_%d(end) = 2;", ii, ii))
endfor

>> clear all
>> a = ones (500000000, 1);
>> b = repmat (a, 1, 10);
error: out of memory or dimension too large for Octave's index type
error: called from:
error:   /opt/local/share/octave/3.8.0/m/general/repmat.m at line 105, column 9
>> 

the reason for the error is not that the memory has been filled
but that you are trying to create an array that is larger than what
can be indexed by and index of type "int".

So creating ten copies of the vector a is fine, but creating ONE vector
with size ten times that of a is not possible.

HTH,
c.