Re: [Duplicity-talk] Restoring a file deleted before the latest incremental

[duplicity]

Thank you for the additional comments. See inline comments.

On Tue, Aug 22, 2023, at 12:31, Nate Eldredge wrote:

On Mon, 21 Aug 2023, duplicity--- via Duplicity-talk wrote:

> Hi all, I am executing a somewhat complex test plan which I will share if you need it to give advice, but here is the short version first. I generally leave my partitions mounted and have a partition by partition backup that runs nightly.

>

> What happens if the following timeline occurs?

>

> 1) Full backup, some number of incrementals, each with a given file changed.

> 2) LUKS partition is inadvertently dismounted or system boots w/o me manually mounting partitions.

> 3) Scheduled backup occurs on raw mount point.

> 4) LUKS partition is mounted on raw mount point.

> 5) Changed file is deleted.

> 6) Next scheduled backup is run.

What I would expect is that the #3 backup is empty, as if all the files 

had been deleted, and that #6 behaves as if all the files had been newly 

created again.  (Unless there is something in the mount point directory 

that gets hidden when the partition is mounted.)

I'm not even certain that I did a --list-current after #3 because I was  banking on --time taking me back to the incrementals that did include my test file, however the --file-to-restore answered that my file was not in the catalogue. As I revamp my test plan, I'll include the extra --list-current.

> At this point the changed file from #1 is no longer in the catalogue 

> because of #3. I have tried restoring by timestamp previous to #2, but 

> that older file isn't available. I see two options:

That shouldn't happen.  In my tests, using --time and --path-to-restore, 

it works fine to restore a file that existed at some previous time, even 

if that file does not exist in the most recent backup.

Very encouraging, it means the problem is in my planning and execution, not in the features of duplicity.

Are you certain that you specified a timestamp between #1 and #3? 

Sometimes time zones can lead to mistakes.

Can you reproduce this with a smaller test case, showing the exact 

commands you ran to do the backups and restores?

I am confident I used the correct time and timezone of the timestamp on the file I edited so that it would be backed up in the incremental. I'm also now thinking maybe I should set --time to some seconds before the file timestamp as I don't know how time is calculated.

> 1) Run a backup when I mount the partition in #4, but since some of my 

> partitions are large and my jobs run 4-7 hours I would like to find 

> another way. Also, #5 could happen prior to that backup completing.

Having occasional backups where the directory is empty shouldn't hurt 

their integrity, but is inefficient if you are doing incrementals.  In 

your example, if backup #6 is an incremental, it will have to contain a 

complete copy of all the files, since in the previous backup #3, none of 

them existed.  If backup #3 had not taken place, then #6 would only have 

to contain the diffs since #1.

#6 would typically be incremental, but the command in the scripts is --full-if-older-than 1M. Without realizing it, I was testing whether #6 would act like a differential instead. Obviously, it won't.

One idea is to have your automated backup job test that the partition is 

mounted before doing the backup.  A simple way is to choose a file that 

should always be present in the mounted partition (or create a dummy file 

for this purpose), and test that it exists.  E.g. a cron job like

0 0 * * * [ -e /mnt/foo/myfile ] && duplicity ...

I follow that if [ -e /mnt/foo/myfile ] works, the duplicity script will run. I have to understand what [ -e /mnt/foo/myfile ] does but will work on it.

> 2) I found the following post, but don't understand "... If you had the

> exact time of deletion and moved any incremental backups out of the way

> that had occurred since then, it would be possible to do it manually."

> https://lists.gnu.org/archive/html/duplicity-talk/2008-05/msg00018.html

I don't think that post is relevant to your use case.  There, the 

user wanted to recover a file *without* knowing the timestamp when it 

existed, which isn't supported.  In your case, it sounds like you do know 

the timestamp.

Agreed.  Please give me some time to redesign my test plan with the input from you and Ed. I'll report back with results on a new box and different dummy data. I had been working on my prod box with dummy data on a spare partition, but now I would be more comfortable if I don't expose my prod to all these changes.

-- 

Nate Eldredge

nate@thatsmathematics.com