Re: Parallel and Large Results

[Alberto Simões]

Hello,

On Fri, Feb 21, 2014 at 7:29 PM, Olaf Till <address@hidden> wrote:

On Thu, Feb 20, 2014 at 08:01:01PM +0000, Alberto Simões wrote:

> I was able to make it work, but given that each iteration result is a huge
> cell with matrices (about 20 MB each), my machine gets out of memory before
> all processes finish their work.
>
> What I need is a way to "reduce" results during the parallelism (and not
> just at the end). These matrices should be all summed up later, so the
> reduce algorithm is associative, and therefore, I can do it whenever I like.

I don't understand. Could you explain in more detail?

Sure. Basically, each separate process computes a big matrix. This is the time/CPU consuming task, and the one I want to make parallel.

I have 2000 of these processes, therefore, parcellfun will try to create a cell array with 2000 of those matrices.

What I want at the end, if to sum up all those matrices together. I can't wait until they are all computed, or the memory gets exausted. That's why I would like to go "summing up" the resulting matrices in some "global" variable.

At the moment, the only idea I have is to compute, say 100 of these matrices, sum them all, compute more 100, and so on.
But was expecting a more effective solution

> The problem is that I do not see a way to do that. I tried to do something
> stupid, creating a global cell, and making each process change it. But as
> you might guess, the global cell is not copied to the forked octaves.

"For the protocol", globals defined before forking should of course be
present in each forked octave, but are separate copies and not a
variable shared by the processes.

Yes, sorry. I wasn't clear. Yes, it is created in all processes, but yes, they are all separated variables :)

THanks,
Alberto

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

--

Alberto Simões