[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremen
From: |
Remy van Elst |
Subject: |
Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups? |
Date: |
Sun, 31 May 2015 17:51:31 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 05/31/2015 02:22 PM, address@hidden wrote:
> [SNIP]
>>
>>> weird, can you send a list of the files on your backend? looks
>>> like not all were uploaded successfully.
>>
>>
>> Here's a list of all the files in the backend:
>>
>> https://3335ae642c4a42549b7a4489adf98d7c.objectstore.eu/duplicity/dup
lic
>>
>>
>>
ity-3.txt
>>
>
> ok, comparing this with your first duplicity.txt output (line 325)
> shows that duplicity only lists ~9900 files while there are ~74000
> online.
>
> this partial "blindness" let's it find only some backups,
> seemingly the oldest ones.
>
> this reminds me of
> https://answers.launchpad.net/duplicity/+question/241395 where a
> paramiko bug shortened the list of files on the backend.
>
> another possibility would be a bug within python, i seem to
> loosely remember that this was the case at one time in past on
> MacOS though.
>
>
> your leads from here (please report your progress) 1. update to
> latest swift-client. doublecheck that the python based swift cmd
> line client does indeed list _all_ files. 2. update python to the
> latest stable 3. research problems with python, swift and
> truncated file listings
>
> ..ede/duply.net
>
> _______________________________________________ Duplicity-talk
> mailing list address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
I might have located a bug.
The duplicity swift module for listing a backend uses the following code
:
def _list(self):
headers, objs = self.conn.get_container(self.container)
return [o['name'] for o in objs]
The Swift client documentation states the following for this function:
http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#
swiftclient.client.get_container
swiftclient.client.get_container(url, token, container, marker=None,
limit=None, prefix=None, delimiter=None, end_marker=None, path=None,
http_conn=None, full_listing=False)
full_listing – if True, return a full listing, else returns a max of
10000 listings
This small addition to the swiftbackend seems to fix this issue. I now
get the full contents of the container, here's the new collection-status
:
# Hostname: vps1.example.org
# Storage used: 1.9T
# Full backups to keep: 4
# Create full backup if last full backup is older than: 14D
-----------------------------------------
# Duplicity collection status:
# Local and Remote metadata are synchronized, no sync needed.
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
34927Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
34914Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
34917Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
34932Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
34922Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
35000Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
34952Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150417T0
34949Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
34902Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
34926Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
34928Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
34916Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
34959Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
35017Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150501T0
34946Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
34953Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
34927Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
35120Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
34941Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
34921Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
35029Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
35145Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
34948Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
34933Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
35016Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150418T0
34914Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150419T0
35000Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150420T0
34952Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150421T0
34917Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150422T0
34928Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150423T0
34926Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150424T0
34932Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150425T0
34922Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150426T0
34916Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150427T0
34927Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150428T0
34902Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150416T034937Z.to.20150429T0
34952Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150502T0
34959Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150503T0
34948Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150504T0
35029Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150505T0
34933Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150506T0
34921Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150507T0
35120Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150508T0
35145Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150509T0
35017Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150510T0
35016Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150511T0
35016Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150512T0
34953Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150513T0
34941Z.sigtar.gz
#
vps1.example.org.duplicity-new-signatures.20150430T034939Z.to.20150514T0
34927Z.sigtar.gz
# Last full backup date: Fri May 29 13:49:11 2015
# Collection Status
# -----------------
# Connecting with backend: BackendWrapper
# Archive dir: /root/.cache/duplicity/vps1.example.org.
# Found 17 secondary backup chains.
# Secondary chain 1 of 17:
# -------------------------
# Chain start time: Tue Mar 24 14:42:44 2015
# Chain end time: Tue Apr 7 05:49:27 2015
# Number of contained backup sets: 14
# Total number of contained volumes: 5532
# Type of backup set: Time: Num
volumes:
# Full Tue Mar 24 14:42:44 2015
3009
# Incremental Wed Mar 25 05:49:32 2015
170
# Incremental Fri Mar 27 05:49:24 2015
198
# Incremental Sat Mar 28 05:49:28 2015
199
# Incremental Sun Mar 29 05:49:12 2015
188
# Incremental Mon Mar 30 05:49:27 2015
194
# Incremental Tue Mar 31 05:49:37 2015
194
# Incremental Wed Apr 1 05:49:27 2015
195
# Incremental Thu Apr 2 05:51:05 2015
198
# Incremental Fri Apr 3 05:50:07 2015
196
# Incremental Sat Apr 4 05:50:10 2015
218
# Incremental Sun Apr 5 05:49:59 2015
187
# Incremental Mon Apr 6 05:49:42 2015
193
# Incremental Tue Apr 7 05:49:27 2015
193
# -------------------------
# Secondary chain 2 of 17:
# -------------------------
# Chain start time: Wed Apr 8 05:49:18 2015
# Chain end time: Wed Apr 15 05:52:25 2015
# Number of contained backup sets: 8
# Total number of contained volumes: 4514
# Type of backup set: Time: Num
volumes:
# Full Wed Apr 8 05:49:18 2015
3104
# Incremental Thu Apr 9 05:49:18 2015
241
# Incremental Fri Apr 10 05:49:23 2015
195
# Incremental Sat Apr 11 05:49:03 2015
194
# Incremental Sun Apr 12 05:49:03 2015
194
# Incremental Mon Apr 13 05:49:08 2015
193
# Incremental Tue Apr 14 05:49:09 2015
196
# Incremental Wed Apr 15 05:52:25 2015
197
# -------------------------
# Secondary chain 3 of 17:
# -------------------------
# Chain start time: Thu Apr 16 05:49:37 2015
# Chain end time: Wed Apr 29 05:49:52 2015
# Number of contained backup sets: 2
# Total number of contained volumes: 3401
# Type of backup set: Time: Num
volumes:
# Full Thu Apr 16 05:49:37 2015
3128
# Incremental Wed Apr 29 05:49:52 2015
273
# -------------------------
# Secondary chain 4 of 17:
# -------------------------
# Chain start time: Thu Apr 30 05:49:39 2015
# Chain end time: Thu May 14 05:49:27 2015
# Number of contained backup sets: 2
# Total number of contained volumes: 3467
# Type of backup set: Time: Num
volumes:
# Full Thu Apr 30 05:49:39 2015
3206
# Incremental Thu May 14 05:49:27 2015
261
# -------------------------
# Secondary chain 5 of 17:
# -------------------------
# Chain start time: Fri May 15 05:50:13 2015
# Chain end time: Fri May 15 05:50:13 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3270
# Type of backup set: Time: Num
volumes:
# Full Fri May 15 05:50:13 2015
3270
# -------------------------
# Secondary chain 6 of 17:
# -------------------------
# Chain start time: Sat May 16 05:51:23 2015
# Chain end time: Sat May 16 05:51:23 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3324
# Type of backup set: Time: Num
volumes:
# Full Sat May 16 05:51:23 2015
3324
# -------------------------
# Secondary chain 7 of 17:
# -------------------------
# Chain start time: Sun May 17 05:52:07 2015
# Chain end time: Sun May 17 05:52:07 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3378
# Type of backup set: Time: Num
volumes:
# Full Sun May 17 05:52:07 2015
3378
# -------------------------
# Secondary chain 8 of 17:
# -------------------------
# Chain start time: Mon May 18 05:52:13 2015
# Chain end time: Mon May 18 05:52:13 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3432
# Type of backup set: Time: Num
volumes:
# Full Mon May 18 05:52:13 2015
3432
# -------------------------
# Secondary chain 9 of 17:
# -------------------------
# Chain start time: Tue May 19 05:50:27 2015
# Chain end time: Tue May 19 05:50:27 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3488
# Type of backup set: Time: Num
volumes:
# Full Tue May 19 05:50:27 2015
3488
# -------------------------
# Secondary chain 10 of 17:
# -------------------------
# Chain start time: Wed May 20 05:50:34 2015
# Chain end time: Wed May 20 05:50:34 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3544
# Type of backup set: Time: Num
volumes:
# Full Wed May 20 05:50:34 2015
3544
# -------------------------
# Secondary chain 11 of 17:
# -------------------------
# Chain start time: Thu May 21 05:49:33 2015
# Chain end time: Thu May 21 05:49:33 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3607
# Type of backup set: Time: Num
volumes:
# Full Thu May 21 05:49:33 2015
3607
# -------------------------
# Secondary chain 12 of 17:
# -------------------------
# Chain start time: Fri May 22 05:51:50 2015
# Chain end time: Fri May 22 05:51:50 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3664
# Type of backup set: Time: Num
volumes:
# Full Fri May 22 05:51:50 2015
3664
# -------------------------
# Secondary chain 13 of 17:
# -------------------------
# Chain start time: Sat May 23 05:50:52 2015
# Chain end time: Sat May 23 05:50:52 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3720
# Type of backup set: Time: Num
volumes:
# Full Sat May 23 05:50:52 2015
3720
# -------------------------
# Secondary chain 14 of 17:
# -------------------------
# Chain start time: Sun May 24 05:49:21 2015
# Chain end time: Sun May 24 05:49:21 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3776
# Type of backup set: Time: Num
volumes:
# Full Sun May 24 05:49:21 2015
3776
# -------------------------
# Secondary chain 15 of 17:
# -------------------------
# Chain start time: Mon May 25 05:49:42 2015
# Chain end time: Mon May 25 05:49:42 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3833
# Type of backup set: Time: Num
volumes:
# Full Mon May 25 05:49:42 2015
3833
# -------------------------
# Secondary chain 16 of 17:
# -------------------------
# Chain start time: Tue May 26 05:49:16 2015
# Chain end time: Tue May 26 05:49:16 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3890
# Type of backup set: Time: Num
volumes:
# Full Tue May 26 05:49:16 2015
3890
# -------------------------
# Secondary chain 17 of 17:
# -------------------------
# Chain start time: Thu May 28 15:59:52 2015
# Chain end time: Thu May 28 15:59:52 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 3953
# Type of backup set: Time: Num
volumes:
# Full Thu May 28 15:59:52 2015
3953
# -------------------------
# Found primary backup chain with matching signature chain:
# -------------------------
# Chain start time: Fri May 29 13:49:11 2015
# Chain end time: Fri May 29 13:49:11 2015
# Number of contained backup sets: 1
# Total number of contained volumes: 4009
# Type of backup set: Time: Num
volumes:
# Full Fri May 29 13:49:11 2015
4009
# -------------------------
# Also found 0 backup sets not part of any chain,
# and 3 incomplete backup sets.
# These may be deleted by running duplicity with the "cleanup"
command.
Restoring also seems to work just fine.
Pull/merge request:
https://code.launchpad.net/~raymii/duplicity/fix-swiftbackend-max-10000-
files-in-list/+merge/260660
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBCAAGBQJVay4DAAoJECtnVb0bf4jchVAH/R8RdgOyaztk3VpDnYMvR1Cx
klhYIoTscrYPWyYiFyi8acDlFRzkRgv6wYxOaRZNk5JOFZ3/HhKMmSl0THDAK7iR
EZkyFIOmEQZwOE7RuN47QvmjiAA04BPAecHEsl5h0P7RvcZenne8RRDz/14lR3/e
y0gmYwthbqAxlmz3uMPD1/a/KMRJjAJ93xlgTABZp3aKoWM+TUQ5dIcIbBjRzVSI
y+sxZVLEJJAlG6/dRwlnEgIOO+MKHYC5eRpHdZLGbMkzVbB5jx9PyY2kWPqqAzqf
LJSmKCW4AzqRmdKoKQxlSqyshsspAAtzTxtkqx7rW1cJ3gctH7/aRL5tNKs7mo8=
=x1sF
-----END PGP SIGNATURE-----
0x1B7F88DC.asc
Description: application/pgp-keys
0x1B7F88DC.asc.sig
Description: PGP signature
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, (continued)
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?,
Remy van Elst <=
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, edgar . soldin, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Zach Adams, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Kai Stian Olstad, 2015/05/31
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/29
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28
- Re: [Duplicity-talk] Duplicity using 1.5 TB storage and loosing incremental backups?, Remy van Elst, 2015/05/28