Re: Pararrayfun

[Clinton Winant]

Many thanks for your helpful reply.  I understand that with "Vectorized", the output can be a 2d matrix, but I am not quite there yet.  I modified your script as:

clear all;
nproc=4;

function N = single_thread (arg)
  Array2d=arg*[1,i;-i,1];
  N=Array2d(:);
endfunction

b = pararrayfun (nproc, @single_thread, [1:4],'Vectorized',true,'ChunksPerProc',1)

b = pararrayfun (nproc, @single_thread, [1:5],'Vectorized',true,'ChunksPerProc',2)

b = pararrayfun (nproc, @single_thread, [1:5],'Vectorized',true,'ChunksPerProc',1)

the output:

octave:1> test_pararray
parcellfun: 4/4 jobs done
b =

   1 + 0i   2 + 0i   3 + 0i   4 + 0i
  -0 - 1i  -0 - 2i  -0 - 3i  -0 - 4i
   0 + 1i   0 + 2i   0 + 3i   0 + 4i
   1 + 0i   2 + 0i   3 + 0i   4 + 0i

parcellfun: 5/5 jobs done
b =

   1 + 0i   2 + 0i   3 + 0i   4 + 0i   5 + 0i
  -0 - 1i  -0 - 2i  -0 - 3i  -0 - 4i  -0 - 5i
   0 + 1i   0 + 2i   0 + 3i   0 + 4i   0 + 5i
   1 + 0i   2 + 0i   3 + 0i   4 + 0i   5 + 0i

parcellfun: 2/4 jobs done
warning: parcellfun: unhandled error in subprocess 1
warning: called from
    parcellfun at line 291 column 9
    chunk_parcellfun at line 47 column 23
    pararrayfun at line 73 column 26
    test_pararray at line 13 column 3
parcellfun: 3/4 jobs done
error: reshape: can't reshape 12x1 array to 1x5 array
error: called from
    chunk_parcellfun>@<anonymous> at line 55 column 33
    chunk_parcellfun at line 55 column 15
    pararrayfun at line 73 column 26
    test_pararray at line 13 column 3

The first two calls work fine, but the third does not, leading me to conclude I dont undestand what ChunksPerProc means.  For instance if the input vector had 256 values, and nproc=16, the should ChunksPerProc need to be 16?  Clearly not, if it was that simple, the program would it compute it by itself.

Sorry to be so dense.  The possibilty of breaking large calculations into "chunks" and using multiple processors is such a giant step forward, Perhaps I could contribute in some way? 

On Sun, Mar 12, 2017 at 10:21 AM, Olaf Till <address@hidden> wrote:

On Sat, Mar 11, 2017 at 04:06:26PM -0800, Clinton Winant wrote:
> I want to use the parallel package to furnish a large 4 dimensional array.
> The following:
>
> clear;
> function a=single_thread(ky);
>   a(:,:,1)=ky*[1,2;4,5];
>   a(:,:,2)=[1,2;4,5];
>   a(:,:,3)=-ky*[1,2;4,5];
> endfunction
>
> b=pararrayfun(4,@single_thread,[1:4],"ChunksPerProc", 1)
>
> works

by some chance. As arrayfun, pararrayfun with "vectorized" not set to
true by default expects the user function to return a single
element. With "vectorized" set to true, pararrayfun expects the user
function an array of the same dimensions as the user functions input
argument(s).

You can set "UniformOutput" to false and concatenate the the results
in the returned cell array manually.

If you want to go in for some hassle to optimize your processing, read on.

Currently pararrayfun is written in a way making the following work:

octave:1> function a = fmultdim (arg)
> a = cat (4, cat (3, [1; 2], [3; 4]), cat (3, arg * [1; 2], arg * [3; 4]), cat (3, [1; 2], [3; 4]));
> endfunction

octave:2> b = pararrayfun (4, @fmultdim, [1:5])
parcellfun: 5/5 jobs done
b =

ans(:,:,1,1) =

   1   1   1   1   1
   2   2   2   2   2

...

But if you give "ChunksPerProc" with "vectorized" false, cellfun is
called internally, and doesn't accept this input. If you set
"vectorized" to true, it should work if you change the user function
to accept row vectors and return the corresponding number of slices
along dimension 2.

Olaf

--
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net