[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremen
From: |
edgar . soldin |
Subject: |
Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups? |
Date: |
Sun, 31 May 2015 18:18:18 +0200 |
User-agent: |
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 |
On 31.05.2015 17:51, Remy van Elst wrote:
>
>
> On 05/31/2015 02:22 PM, address@hidden wrote:
>> [SNIP]
>>>
>>>> weird, can you send a list of the files on your backend? looks
>>>> like not all were uploaded successfully.
>>>
>>>
>>> Here's a list of all the files in the backend:
>>>
>>> https://3335ae642c4a42549b7a4489adf98d7c.objectstore.eu/duplicity/dup
> lic
>>>
>>>
>>>
> ity-3.txt
>>>
>
>> ok, comparing this with your first duplicity.txt output (line 325)
>> shows that duplicity only lists ~9900 files while there are ~74000
>> online.
>
>> this partial "blindness" let's it find only some backups,
>> seemingly the oldest ones.
>
>> this reminds me of
>> https://answers.launchpad.net/duplicity/+question/241395 where a
>> paramiko bug shortened the list of files on the backend.
>
>> another possibility would be a bug within python, i seem to
>> loosely remember that this was the case at one time in past on
>> MacOS though.
>
>
>> your leads from here (please report your progress) 1. update to
>> latest swift-client. doublecheck that the python based swift cmd
>> line client does indeed list _all_ files. 2. update python to the
>> latest stable 3. research problems with python, swift and
>> truncated file listings
>
>> ..ede/duply.net
>
>> _______________________________________________ Duplicity-talk
>> mailing list address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
>
>
> I might have located a bug.
>
> The duplicity swift module for listing a backend uses the following code
> :
>
> def _list(self):
> headers, objs = self.conn.get_container(self.container)
> return [o['name'] for o in objs]
>
>
> The Swift client documentation states the following for this function:
>
> http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#
> swiftclient.client.get_container
>
> swiftclient.client.get_container(url, token, container, marker=None,
> limit=None, prefix=None, delimiter=None, end_marker=None, path=None,
> http_conn=None, full_listing=False)
>
> full_listing – if True, return a full listing, else returns a max of
> 10000 listings
>
> This small addition to the swiftbackend seems to fix this issue. I now
> get the full contents of the container, here's the new collection-status
> :
>
>
>
>
> # Hostname: vps1.example.org
> # Storage used: 1.9T
> # Full backups to keep: 4
> # Create full backup if last full backup is older than: 14D
> -----------------------------------------
> # Duplicity collection status:
> # Local and Remote metadata are synchronized, no sync needed.
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
> 34927Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
> 34914Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
> 34917Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
> 34932Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
> 34922Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
> 35000Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
> 34952Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150417T0
> 34949Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
> 34902Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
> 34926Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
> 34928Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
> 34916Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
> 34959Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
> 35017Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150501T0
> 34946Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
> 34953Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
> 34927Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
> 35120Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
> 34941Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
> 34921Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
> 35029Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
> 35145Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
> 34948Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
> 34933Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
> 35016Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
> 34914Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
> 35000Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
> 34952Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
> 34917Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
> 34928Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
> 34926Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
> 34932Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
> 34922Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
> 34916Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
> 34927Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
> 34902Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150429T0
> 34952Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
> 34959Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
> 34948Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
> 35029Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
> 34933Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
> 34921Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
> 35120Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
> 35145Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
> 35017Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150510T0
> 35016Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
> 35016Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
> 34953Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
> 34941Z.sigtar.gz
> #
> vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
> 34927Z.sigtar.gz
> # Last full backup date: Fri May 29 13:49:11 2015
> # Collection Status
> # -----------------
> # Connecting with backend: BackendWrapper
> # Archive dir: /root/.cache/duplicity/vps1.example.org.
> # Found 17 secondary backup chains.
> # Secondary chain 1 of 17:
> # -------------------------
> # Chain start time: Tue Mar 24 14:42:44 2015
> # Chain end time: Tue Apr 7 05:49:27 2015
> # Number of contained backup sets: 14
> # Total number of contained volumes: 5532
> # Type of backup set: Time: Num
> volumes:
> # Full Tue Mar 24 14:42:44 2015
> 3009
> # Incremental Wed Mar 25 05:49:32 2015
> 170
> # Incremental Fri Mar 27 05:49:24 2015
> 198
> # Incremental Sat Mar 28 05:49:28 2015
> 199
> # Incremental Sun Mar 29 05:49:12 2015
> 188
> # Incremental Mon Mar 30 05:49:27 2015
> 194
> # Incremental Tue Mar 31 05:49:37 2015
> 194
> # Incremental Wed Apr 1 05:49:27 2015
> 195
> # Incremental Thu Apr 2 05:51:05 2015
> 198
> # Incremental Fri Apr 3 05:50:07 2015
> 196
> # Incremental Sat Apr 4 05:50:10 2015
> 218
> # Incremental Sun Apr 5 05:49:59 2015
> 187
> # Incremental Mon Apr 6 05:49:42 2015
> 193
> # Incremental Tue Apr 7 05:49:27 2015
> 193
> # -------------------------
> # Secondary chain 2 of 17:
> # -------------------------
> # Chain start time: Wed Apr 8 05:49:18 2015
> # Chain end time: Wed Apr 15 05:52:25 2015
> # Number of contained backup sets: 8
> # Total number of contained volumes: 4514
> # Type of backup set: Time: Num
> volumes:
> # Full Wed Apr 8 05:49:18 2015
> 3104
> # Incremental Thu Apr 9 05:49:18 2015
> 241
> # Incremental Fri Apr 10 05:49:23 2015
> 195
> # Incremental Sat Apr 11 05:49:03 2015
> 194
> # Incremental Sun Apr 12 05:49:03 2015
> 194
> # Incremental Mon Apr 13 05:49:08 2015
> 193
> # Incremental Tue Apr 14 05:49:09 2015
> 196
> # Incremental Wed Apr 15 05:52:25 2015
> 197
> # -------------------------
> # Secondary chain 3 of 17:
> # -------------------------
> # Chain start time: Thu Apr 16 05:49:37 2015
> # Chain end time: Wed Apr 29 05:49:52 2015
> # Number of contained backup sets: 2
> # Total number of contained volumes: 3401
> # Type of backup set: Time: Num
> volumes:
> # Full Thu Apr 16 05:49:37 2015
> 3128
> # Incremental Wed Apr 29 05:49:52 2015
> 273
> # -------------------------
> # Secondary chain 4 of 17:
> # -------------------------
> # Chain start time: Thu Apr 30 05:49:39 2015
> # Chain end time: Thu May 14 05:49:27 2015
> # Number of contained backup sets: 2
> # Total number of contained volumes: 3467
> # Type of backup set: Time: Num
> volumes:
> # Full Thu Apr 30 05:49:39 2015
> 3206
> # Incremental Thu May 14 05:49:27 2015
> 261
> # -------------------------
> # Secondary chain 5 of 17:
> # -------------------------
> # Chain start time: Fri May 15 05:50:13 2015
> # Chain end time: Fri May 15 05:50:13 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3270
> # Type of backup set: Time: Num
> volumes:
> # Full Fri May 15 05:50:13 2015
> 3270
> # -------------------------
> # Secondary chain 6 of 17:
> # -------------------------
> # Chain start time: Sat May 16 05:51:23 2015
> # Chain end time: Sat May 16 05:51:23 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3324
> # Type of backup set: Time: Num
> volumes:
> # Full Sat May 16 05:51:23 2015
> 3324
> # -------------------------
> # Secondary chain 7 of 17:
> # -------------------------
> # Chain start time: Sun May 17 05:52:07 2015
> # Chain end time: Sun May 17 05:52:07 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3378
> # Type of backup set: Time: Num
> volumes:
> # Full Sun May 17 05:52:07 2015
> 3378
> # -------------------------
> # Secondary chain 8 of 17:
> # -------------------------
> # Chain start time: Mon May 18 05:52:13 2015
> # Chain end time: Mon May 18 05:52:13 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3432
> # Type of backup set: Time: Num
> volumes:
> # Full Mon May 18 05:52:13 2015
> 3432
> # -------------------------
> # Secondary chain 9 of 17:
> # -------------------------
> # Chain start time: Tue May 19 05:50:27 2015
> # Chain end time: Tue May 19 05:50:27 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3488
> # Type of backup set: Time: Num
> volumes:
> # Full Tue May 19 05:50:27 2015
> 3488
> # -------------------------
> # Secondary chain 10 of 17:
> # -------------------------
> # Chain start time: Wed May 20 05:50:34 2015
> # Chain end time: Wed May 20 05:50:34 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3544
> # Type of backup set: Time: Num
> volumes:
> # Full Wed May 20 05:50:34 2015
> 3544
> # -------------------------
> # Secondary chain 11 of 17:
> # -------------------------
> # Chain start time: Thu May 21 05:49:33 2015
> # Chain end time: Thu May 21 05:49:33 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3607
> # Type of backup set: Time: Num
> volumes:
> # Full Thu May 21 05:49:33 2015
> 3607
> # -------------------------
> # Secondary chain 12 of 17:
> # -------------------------
> # Chain start time: Fri May 22 05:51:50 2015
> # Chain end time: Fri May 22 05:51:50 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3664
> # Type of backup set: Time: Num
> volumes:
> # Full Fri May 22 05:51:50 2015
> 3664
> # -------------------------
> # Secondary chain 13 of 17:
> # -------------------------
> # Chain start time: Sat May 23 05:50:52 2015
> # Chain end time: Sat May 23 05:50:52 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3720
> # Type of backup set: Time: Num
> volumes:
> # Full Sat May 23 05:50:52 2015
> 3720
> # -------------------------
> # Secondary chain 14 of 17:
> # -------------------------
> # Chain start time: Sun May 24 05:49:21 2015
> # Chain end time: Sun May 24 05:49:21 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3776
> # Type of backup set: Time: Num
> volumes:
> # Full Sun May 24 05:49:21 2015
> 3776
> # -------------------------
> # Secondary chain 15 of 17:
> # -------------------------
> # Chain start time: Mon May 25 05:49:42 2015
> # Chain end time: Mon May 25 05:49:42 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3833
> # Type of backup set: Time: Num
> volumes:
> # Full Mon May 25 05:49:42 2015
> 3833
> # -------------------------
> # Secondary chain 16 of 17:
> # -------------------------
> # Chain start time: Tue May 26 05:49:16 2015
> # Chain end time: Tue May 26 05:49:16 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3890
> # Type of backup set: Time: Num
> volumes:
> # Full Tue May 26 05:49:16 2015
> 3890
> # -------------------------
> # Secondary chain 17 of 17:
> # -------------------------
> # Chain start time: Thu May 28 15:59:52 2015
> # Chain end time: Thu May 28 15:59:52 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 3953
> # Type of backup set: Time: Num
> volumes:
> # Full Thu May 28 15:59:52 2015
> 3953
> # -------------------------
> # Found primary backup chain with matching signature chain:
> # -------------------------
> # Chain start time: Fri May 29 13:49:11 2015
> # Chain end time: Fri May 29 13:49:11 2015
> # Number of contained backup sets: 1
> # Total number of contained volumes: 4009
> # Type of backup set: Time: Num
> volumes:
> # Full Fri May 29 13:49:11 2015
> 4009
> # -------------------------
> # Also found 0 backup sets not part of any chain,
> # and 3 incomplete backup sets.
> # These may be deleted by running duplicity with the "cleanup"
> command.
>
>
>
> Restoring also seems to work just fine.
>
> Pull/merge request:
> https://code.launchpad.net/~raymii/duplicity/fix-swiftbackend-max-10000-
> files-in-list/+merge/260660
>
>
btw. 74k files is quite a lot.. you might want to consider raising your volume
size.
..ede/duply.net
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, (continued)
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?,
edgar . soldin <=
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Zach Adams, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Kai Stian Olstad, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28