Re: parallel pkg unexplainable error

[Juan Pablo Carbajal]

On Mon, Sep 15, 2014 at 5:54 PM, Moritz Keuthen <address@hidden> wrote:
> Am 15.09.2014 um 17:36 schrieb Juan Pablo Carbajal:
>
>> On Mon, Sep 15, 2014 at 4:53 PM, Moritz Keuthen <address@hidden> wrote:
>>>
>>> Am 15.09.2014 um 16:37 schrieb Juan Pablo Carbajal:
>>>>
>>>> On Mon, Sep 15, 2014 at 10:22 AM, Moritz Keuthen <address@hidden>
>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I am using the octave package parallel and have had increasingly
>>>>> problems. Quite a lot of my last test runs aborted with the following
>>>>> error:
>>>>>
>>>>> parcellfun: 9/13 jobs doneerror: invalid conversion from real matrix to
>>>>> real scalar
>>>>> error: file id must be a file object, std::string, or integer value
>>>>> warning: broken pipe
>>>>> warning: broken pipe
>>>>> warning: broken pipe
>>>>> warning: broken pipe
>>>>> error: called from:
>>>>> error: /home/hpc/t1313/di34koy/octave/parallel-2.2.0/parcellfun.m at
>>>>> line
>>>>> 334, column 11
>>>>>
>>>>> The error is not replicable (running the same program again won't
>>>>> necessarily produce the error at the same iteration) and has appeared
>>>>> in
>>>>> different pipes, e.g.
>>>>>
>>>>> parcellfun: 12/13 jobs doneerror: invalid conversion from real matrix
>>>>> to
>>>>> real scalar
>>>>> error: file id must be a file object, std::string, or integer value
>>>>> warning: broken pipe
>>>>> error: called from:
>>>>> error: /home/hpc/t1313/di34koy/octave/parallel-2.2.0/parcellfun.m at
>>>>> line
>>>>> 334, column 11
>>>>>
>>>>> I am using octave 3.8.1 and parallel 2.2.0. It seems to me that the
>>>>> problem lies somewhere in the parallel code, but I am not an expert (;
>>>>>
>>>>> I would be deeply grateful if someone could help me with the issue.
>>>>>
>>>>> Regards
>>>>> Moritz Keuthen
>>>>> _______________________________________________
>>>>> Help-octave mailing list
>>>>> address@hidden
>>>>> https://lists.gnu.org/mailman/listinfo/help-octave
>>>>
>>>> We need to see the code that is generating the "file id". It might be
>>>> that you are doing this automatically and you have a bug there.
>>>
>>>
>>> Hey,
>>>
>>> I am not sure I understand your question correctly. This is how I employ
>>> the
>>> pararrayfun:
>>>
>>> p.ArmijoBeta = 0.5;
>>> p.nArmijoProc = 13;
>>> ...
>>>      parTs = deltaT * p.ArmijoBeta.^(0 : p.nArmijoProc - 1);
>>>      errorHandler = @(s, curT) deal (NA, s);
>>>      handler = @(curT) getCostForStep (curT, input);
>>>
>>>        [output] = pararrayfun (p.nArmijoProc, handler, parTs, ...
>>>                                       "UniformOutput", false, ...
>>>                                       "ErrorHandler", errorHandler);
>>>
>>> The rest is handled by the parallel package.
>>>
>>> Regards
>>> Moritz Keuthen
>>>
>>>
>>>
>>>
>>>
>> The error you are getting is
>>
>> error: file id must be a file object, std::string, or integer value
>>
>> so somewhere in your function "handler" you are creating a file id
>> that seems to be wrong. Does "getCostForStep" use any file id?
>>
>> Also, you have 13 processors, right?
>
>
> No it does not. As far as I understand the parallel code the file id in
> question is generated by the code.
>
> Line 334 of parcellfun reads "fwrite (cmdw(isubp), 0, "double");"
>
> cmdw is created in line 115 and filled in line 119.
>
> The handler function only gets called with different input in each
> subprocess but is not aware of the parallel structure around it.
>
>
> I have 42 processors, and parallel is also able to handle jobs with more
> threads than processors.
>
> Best
> Moritz Keuthen

Please keep the mailing list always in CC. I am not your personal consultant.

Does your program works when the number of processors is 1?
If yes, what is the smallest number that causes the problem?
You can modify Line 334 of parcellfun to print the cmdw pipe id.
It seems that the pipes disappear (or never got opened) while running
your program.
What OS are you using?