Re: data structure for simulation

[Jaroslav Hajek]

On Wed, Mar 18, 2009 at 3:37 PM, Francesco Potorti` <address@hidden> wrote:
>>> The general problem is that I am trying to find a good and simple way to
>>> use a cluster for Montecarlo simulation.  Ideally, I'd like to do that
>>> from inside Octave, without resorting to an external program like a bash
>>> script.
>>
>>Maybe I can help with that. I have a function of the form
>># montecarlo.m: generates a specified number of replications of a function's
>>output and
>># writes them to a user-specified output file.
>>#
>># USAGE: montecarlo(f, f_args, reps, outfile, nslaves, n_pooled,  n_returns,
>>debug)
>>#
>># IMPORTANT: f should return a row vector of output from feval(f,f_args)
>>#
>># For normal evaluation on one machine, only the first 4 arguments are
>>required.
>># * Arg 1: (required) the function that generates a row vector of output
>># * Arg 2: (required) the arguments of the function, in a cell
>># * Arg 3: (required) the number of replications to generate
>># * Arg 4: (required) the output file name
>># * Arg 5: (optional) number of compute nodes for parallel runs
>># * Arg 6 (optional) number of replications to be pooled together before
>>they are sent
>># back to the master node.
>># * Arg 7 (optional) the number of columns of the output returned by f. When
>>f is very costly
>># to evaluate, use this to get a parallel run started immediately. Otherwise
>>single run must
>># be finished before the size is determined. This is bad, since many nodes
>>will be idle.
>># * Arg 8 (optional) verbose: 1 for on, 0 for off
>># * Arg 9 (optional) 0/1 switch controlling debug mode, off by default.
>>
>>
>>This will Monte Carlo any function that takes arguments as a cell, and
>>writes output as a row vector. I think that in most cases a wrapper can be
>>written to satisfy this requirement. To do this in parallel, you must have
>>MPITB installed. I can provide more details if you're interested.
>
> That would be very good, more than I expected :)
>
> To be true, when I had written "simple" I meant that no recompilation of
> Octave was needed, and no fiddling with external libraries :(
>
> The fact is, Montecarlo is a very particular and simple case of parallel
> execution, which could be done by using Octave's popen command, fread,
> fwrite and little else, all in a .m file without the help of external
> libraries, taking Juroslav's parcellfun as a starting point.  If I am
> not oversimplifying things...
>

Parcellfun is easy to use - it should just work out of the box in
standard Octave instalation. But since it relies on fork, it is really
for multicore machines with shared memory. I have heard of systems
where processes could be distributed amongst machines, but never seen
any such thing. If you want to exploit a heterogeneous cluster, MPITB
is probably the way to go.

Pipes, even FIFOs, don't generally work across NFS filesystems and the
like. It is possible to use regular files for communication, but you
need some NFS-safe locking mechanism. For instance, our cluster has
"lockfile".
But using shared files will be surely slower than using MPI, so it
should only be recommended if your jobs are very expensive, so that
even the file communication becomes negligible.

regards

-- 
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz