On 04/22/2014 04:58 PM, Anton Flugge wrote:
Hi,
I am trying to run calls to the kmeans function in parallel. In
principal that works well with parcellfun in the parallel toolbox,
however, the matrix that I want to use kmeans on is quite large (a
couple of GB), and parcellfun creates separate copies of the
workspace/variables for each parallel thread, which means I run out of
memory quicker than I run out of processor cores. Now kmeans only needs
read access, and no write access, to the input data, which means in
principal it shouldn't be a problem to share the same matrix/memory
between all threads, however, I could not find a solution anywhere on
how to do that in Octave... is it possible to have global variables
accessed by multiple parallel threads?
Thanks,
Anton
Hi Anton,
parcellfun uses the fork command to create child processes. I am not an
expert in this, but from what I find is that fork should use a
copy-on-write strategy for the process memory and not blindly copy
everything. I ran a small experiment on my Ubuntu 12.04 with the Octave
development version:
% create a ~800MB matrix
A=rand(1e4);
% calculate the sum 32 times using cellfun - no change in memory usage!
cellfun(@(a) sum(a(:)) , {A}(ones(1,32)) )
% calculate the sum 32 times using parcellfun
% still no change in memory usage!
parcellfun(8, @(a) sum(a(:)) , {A}(ones(1,32)) )
When executing the line with parcellfun, 8 processes are created and it
takes some seconds on my poor old laptop, but I see no change in memory
usage, so the copy-on-write is definitely working for me.
So either this is not working for you, or it is not your input matrix
but stuff created in the kmeans function (I am not familiar with this
particular function) which eats up your memory.
Hope it helps,
Markus