Re: [Duplicity-talk] idea on gpg zombies

[edgar . soldin]

On 29.06.2011 13:16, Kenneth Loafman wrote:
> On Tue, Jun 28, 2011 at 4:56 PM, <address@hidden
> <mailto:address@hidden>> wrote:
> 
> On 28.06.2011 23:39, Kenneth Loafman wrote:
>> We make it in a thread so it can proceed.
> 
> proceed for what. we can as well wait for gpg to finish up and go on
> with the next file. can't we?
> 
> 
> No, it would not work that way.  Think of it as a list of files being
> iterated in sync, kind of a sideways iteration, like a column moving
> across a spreadsheet. When that iteration is complete, all files are
> closed, i.e. the waitpid() happens.  By starting threaded_waitpid()
> early, we're able to harvest the return code and close the processes
> as soon as they complete, not when the iteration is complete.  Then
> when duplicity finally closes the process, the return code is
> returned and all is normal.

this is how it is done now. but couldn't we modify the routine so that it is 
done one after the other, the files are downloaded one after the other anyways? 

> 
>> At the end of the waitpid all of the closed file handles are
>> harvested (except on a Mac) and the task is allowed to close.
> 
> i can't see code that harvests it, so you speak of waitpid doing that
> on it's own, do you? if so: why not close them manually before
> calling waitpid? can't hurt, might help?
> 
> 
> Yes, the harvesting is done by waitpid itself, when the task is
> allowed to close.  The gpg task closes the files itself.
> 

but what if the gpg process refuses to die by its own, because it still expects 
data from a pipe or so? we could exclude that by manually closing all these.

 
>> Without the threading, you get a pile of gpg processes that don't
>> go away until the very end.
> 
> well e.g. ncftp calls don't stick until the end and are done totally
> without threading. simple system calls.
> 
> 
> That's because ncftp is wait'ed for.  The gpg tasks are not.

so let's make gpg also waited for (just an idea, i'd rather solve it by keeping 
threading, but hey).

> 
>> The parent has to issue the waitpid to get this to happen, and on
>> its own, duplicity waits until the very end of the iteration to
>> take care of this task.
> 
> i already understood what waitpid should accomplish. sadly it does
> not under certain, not well known, circumstances. i try to think of
> solutions here, or at least approaches that might resolve this issue
> once and for all.
> 
> 
> I appreciate the work.  It is a hard problem.
> 
> Then again, the real fix is to not do long strings of incrementals.
> We're trying to provide a fix for what is really an operator error.
> 

that's one way to look at it.

but i can't agree with that. having endless incrementals is unsecure and bad 
style, but totally possible and within the scope of duplicity. if it would be a 
user error, it'd happen on all platforms _and_ duplicity would try to prevent 
it. like the other comment said e.g. automatically keeping chains short, by 
starting fulls after a certain amount of incrementals.

this is certainly a bug somewhere in python, or better in the python 
implementations of different platforms, which would be our duty to workaround - 
because if we don't users end up with unusable backups.


ede/duply.net