Re: [Duplicity-talk] Always reads manifest from remote

[dcottingham00]

I was not aware that truncated remote manifests are a thing to look out for.  My immediate reaction is that the current state of the code is inconsistent with solving that problem. As it is, the code silently accepts having the manifest only on the remote, but throws an assertion failure if the manifest is only local. That seems odd if the local manifest is the more trusted one. It also, as I mentioned, silently ignores the absence of the key needed to compare the manifests.

Seems to me that if you want to solve the problem of the remote manifest being truncated, you should read it back and check it right after writing it -- not wait until the next backup, when the local copy might be missing. It should also insist on the key being available. What do you think about this change?

I agree that making the secret key available during inc backups is no big deal.  The docs should probably make clear that this is necessary.  I'll review them and try to make a more specific suggestion.

About Glacier, I just want to comment that there is no need for a backend. I'm using the S3 backend and using lifecycle transition rules to send things to Glacier, and I'm not the only one. (And, incidentally, checking the manifest right after writing it would be Glacier-friendly -- not that I'm suggesting that should be a big consideration.)

Thanks,

Dave Cottingham

---

From: "Kenneth Loafman" <address@hidden>
To: "Discussion of the backup program duplicity" <address@hidden>
Sent: Tuesday, December 10, 2013 11:59:45 AM
Subject: Re: [Duplicity-talk] Always reads manifest from remote

It's needed since we sometimes have truncated manifests on the remote and this one is critical.  We should probably extend the check for truncation to all the files we write.

If you have encrypted backups, you will need your key to increment those backups.  I don't see a problem here.

I think we need to keep the check, but condition it on the Glacier backend, when/if that comes along.

...Ken

On Fri, Dec 6, 2013 at 12:06 PM, <address@hidden> wrote:

I've figured out why we have bug 1170113, which is that on incremental backups duplicity always reads the latest manifest from the remote even if there is a copy in the local cache. This is usually innocuous, but if the remote is Amazon S3 transitioning to Glacier, the remote can't be read (without a previous arrangements) so it's fatal.

The question is what would be the preferred fix. I'll just copy here my latest entry from the bug report page, and I'd be interested in what you guys think:

I have identified why this bug occurs. For incremental backups duplicity always calls BackupSet.check_manifests(), described as "Make sure remote manifest is equal to local one." It reads the manifest from the remote and checks that it is identical to the local copy. I believe this is the only file that duplicity checks in this way.

Of course, the usual case is that the remote files are encrypted and duplicity is not in possession of the secret key. After reading the file from remote, if duplicity finds this is the case then it just proceeds as if the comparison had succeeded.

As I mentioned above, this behavior is fatal if the remote is Amazon S3 transitioned to Glacier. But there are several ways one could imagine patching it:

1. Get rid of check_manifests() altogether. I don't see why in this one case we should be paranoid about a remote file not matching the one in the local cache.

2. Instead of ignoring only the exception that the secret key is not available, also ignore the exception of not being able to read from the remote.

3. Add a command line flag to disable check_manifests().

 - Dave Cottingham

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk