[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity process killed
From: |
Chris Vanden Berghe |
Subject: |
Re: [Duplicity-talk] Duplicity process killed |
Date: |
Wed, 13 Jul 2016 14:54:51 +0800 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 |
But what about the root cause? Version 0.6.x has worked without issue
for many months (years?).
Is there a particular reason why memory usage has increased
significantly with version 0.7.x? Is Duplicity 0.7.x expected to use
more than 1.4GB for a backup?
Chris.
---
On 13/07/2016 03:07, Nate Eldredge via Duplicity-talk wrote:
> Look up "overcommit". Linux by default will let malloc (really sbrk or
> mmap) succeed even when the request exceeds the amount of available
> virtual memory, in hopes that not all of it will actually be used, or
> that more memory will be available by the time it is. You get a
> pointer to unmapped memory, and memory is allocated by the page fault
> handler when you actually write to the page. If no memory is
> available, the OOM killer fires up in an attempt to get some. But if
> your process is the chosen victim, it gets killed. This happens
> "asynchronously" so there isn't really any good way for the process to
> handle it more gracefully.
>
> On Jul 12, 2016, at 14:41, Mark Grandi via Duplicity-talk
> <address@hidden <mailto:address@hidden>> wrote:
>
>> I guess i'm just confused on the behavior here, if the system was out
>> of memory, wouldn't python throw a MemoryError when a malloc() fails?
>> why would the linux OOM killer terminate python/duplicity before it
>> terminates itself due to an uncaught MemoryError?
>>
>> ~Mark
>>
>>
>>> On Jul 12, 2016, at 02:49, Chris Vanden Berghe via Duplicity-talk
>>> <address@hidden <mailto:address@hidden>> wrote:
>>>
>>> So, indeed, looks like an out-of-memory issue... slightly surprising
>>> with 1.4GB of free memory, no?
>>>
>>> # free -h
>>> total used free shared buffers cached
>>> Mem: 1.9G 737M 1.2G 117M 19M
>>> 169M
>>> -/+ buffers/cache: 549M 1.4G
>>> Swap: 0B 0B 0B
>>>
>>> Did memory usage increase from 0.6.x to 0.7.x?
>>>
>>> Thanks,
>>> Chris.
>>>
>>> ---
>>> On 12/07/2016 11:37, Remy van Elst via Duplicity-talk wrote:
>>>> You might want to check your syslog (/var/log/messages or
>>>> /var/log/syslog) and dmesg output to see if you are having memory
>>>> issues (RAM/out of memory killer)
>>>>
>>>>
>>>>
>>>> https://raymii.org <https://raymii.org/>
>>>>
>>>> On Tue, Jul 12, 2016 at 3:41 AM, Chris Vanden Berghe via
>>>> Duplicity-talk <address@hidden
>>>> <mailto:address@hidden>
>>>> <mailto:address@hidden>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Since the recent upgrade to Duplicity 0.7.08 I'm having some
>>>> problems of
>>>> the process being killed spontaneously.
>>>>
>>>> I start Duplicity as follows:
>>>>
>>>> nice /usr/bin/duplicity --verbosity 9 --full-if-older-than 1M
>>>> --archive-dir=/root/Backup/cache/ --include-globbing-filelist
>>>> /root/Backup/backup.list --exclude '**' / gs://bobo-backup/
>>>>
>>>> The output is as follows:
>>>>
>>>> Using archive dir:
>>>> /root/Backup/cache/01f81df3eb37282f76c93fb89a3708f5
>>>> Using backup name: 01f81df3eb37282f76c93fb89a3708f5
>>>> Import of duplicity.backends.acdclibackend Succeeded
>>>> Import of duplicity.backends.azurebackend Succeeded
>>>> Import of duplicity.backends.b2backend Succeeded
>>>> Import of duplicity.backends.botobackend Succeeded
>>>> Import of duplicity.backends.cfbackend Succeeded
>>>> Import of duplicity.backends.copycombackend Succeeded
>>>> Import of duplicity.backends.dpbxbackend Failed: No module named
>>>> dropbox
>>>> Import of duplicity.backends.gdocsbackend Succeeded
>>>> Import of duplicity.backends.giobackend Succeeded
>>>> Import of duplicity.backends.hsibackend Succeeded
>>>> Import of duplicity.backends.hubicbackend Succeeded
>>>> Import of duplicity.backends.imapbackend Succeeded
>>>> Import of duplicity.backends.lftpbackend Succeeded
>>>> Import of duplicity.backends.localbackend Succeeded
>>>> Import of duplicity.backends.mediafirebackend Succeeded
>>>> Import of duplicity.backends.megabackend Succeeded
>>>> Import of duplicity.backends.multibackend Succeeded
>>>> Import of duplicity.backends.ncftpbackend Succeeded
>>>> Import of duplicity.backends.onedrivebackend Succeeded
>>>> Import of duplicity.backends.par2backend Succeeded
>>>> Import of duplicity.backends.pydrivebackend Succeeded
>>>> Import of duplicity.backends.rsyncbackend Succeeded
>>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>>> Import of duplicity.backends.swiftbackend Succeeded
>>>> Import of duplicity.backends.sxbackend Succeeded
>>>> Import of duplicity.backends.tahoebackend Succeeded
>>>> Import of duplicity.backends.webdavbackend Succeeded
>>>> Reading globbing filelist /root/Backup/backup.list
>>>> Main action: inc
>>>>
>>>> ================================================================================
>>>> duplicity 0.7.08 ($reldate)
>>>> Args: /usr/bin/duplicity --verbosity 9 --full-if-older-than 1M
>>>> --archive-dir=/root/Backup/cache/ --include-globbing-filelist
>>>> /root/Backup/backup.list --exclude ** / gs://bobo-backup/
>>>> Linux bobodioulasso 3.16.0-44-generic #59-Ubuntu SMP Tue Jul 7
>>>> 02:07:39
>>>> UTC 2015 x86_64 x86_64
>>>> /usr/bin/python 2.7.9 (default, Apr 2 2015, 15:33:21)
>>>> [GCC 4.9.2]
>>>>
>>>> ================================================================================
>>>> Using temporary directory /tmp/duplicity-2G5OwC-tempdir
>>>> Registering (mkstemp) temporary file
>>>> /tmp/duplicity-2G5OwC-tempdir/mkstemp-9wghI1-1
>>>> Temp has 280009195520 available, backup will use approx 272629760.
>>>> Listed
>>>> gs://bobo-backup//duplicity-full-signatures.20160704T151139Z.sigtar.gpg
>>>> Listed gs://bobo-backup//duplicity-full.20160704T151139Z.manifest.gpg
>>>>
>>>> <SNIP>
>>>>
>>>> Listed
>>>> gs://bobo-backup//duplicity-full.20160704T151139Z.vol97.difftar.gpg
>>>> Listed
>>>> gs://bobo-backup//duplicity-full.20160704T151139Z.vol98.difftar.gpg
>>>> Listed
>>>> gs://bobo-backup//duplicity-full.20160704T151139Z.vol99.difftar.gpg
>>>> Local and Remote metadata are synchronized, no sync needed.
>>>> Listed
>>>> gs://bobo-backup//duplicity-full-signatures.20160704T151139Z.sigtar.gpg
>>>> Listed gs://bobo-backup//duplicity-full.20160704T151139Z.manifest.gpg
>>>>
>>>> <SNIP>
>>>>
>>>> Listed
>>>> gs://bobo-backup//duplicity-full.20160704T151139Z.vol97.difftar.gpg
>>>> Listed
>>>> gs://bobo-backup//duplicity-full.20160704T151139Z.vol98.difftar.gpg
>>>> Listed
>>>> gs://bobo-backup//duplicity-full.20160704T151139Z.vol99.difftar.gpg
>>>> 256 files exist on backend
>>>> 11 files exist in cache
>>>> Extracting backup chains from list of files:
>>>> [u'duplicity-full-signatures.20160704T151139Z.sigtar.gpg',
>>>> u'duplicity-full.20160704T151139Z.manifest.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol1.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol10.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol100.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol101.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol102.difftar.gpg',
>>>>
>>>> <SNIP>
>>>>
>>>> u'duplicity-full.20160704T151139Z.vol92.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol93.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol94.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol95.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol96.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol97.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol98.difftar.gpg',
>>>> u'duplicity-full.20160704T151139Z.vol99.difftar.gpg']
>>>> File duplicity-full-signatures.20160704T151139Z.sigtar.gpg is
>>>> not part
>>>> of a known set; creating new set
>>>> Ignoring file (rejected by backup set)
>>>> 'duplicity-full-signatures.20160704T151139Z.sigtar.gpg'
>>>> File duplicity-full.20160704T151139Z.manifest.gpg is not part of a
>>>> known
>>>> set; creating new set
>>>> Processing local manifest
>>>>
>>>> /root/Backup/cache/01f81df3eb37282f76c93fb89a3708f5/duplicity-full.20160704T151139Z.manifest
>>>> (161677527)
>>>> Found manifest volume 1
>>>> Found manifest volume 2
>>>> Found manifest volume 3
>>>>
>>>> <SNIP>
>>>>
>>>> Found manifest volume 252
>>>> Found manifest volume 253
>>>> Found manifest volume 254
>>>> Found 254 volumes in manifest
>>>> File duplicity-full.20160704T151139Z.vol1.difftar.gpg is part of
>>>> known set
>>>> File duplicity-full.20160704T151139Z.vol10.difftar.gpg is part of
>>>> known set
>>>> File duplicity-full.20160704T151139Z.vol100.difftar.gpg is part of
>>>> known set
>>>>
>>>> <SNIP>
>>>>
>>>> File duplicity-full.20160704T151139Z.vol97.difftar.gpg is part of
>>>> known set
>>>> File duplicity-full.20160704T151139Z.vol98.difftar.gpg is part of
>>>> known set
>>>> File duplicity-full.20160704T151139Z.vol99.difftar.gpg is part of
>>>> known set
>>>> Found backup chain [Mon Jul 4 17:11:39 2016]-[Mon Jul 4 17:11:39
>>>> 2016]
>>>> Last full backup date: Mon Jul 4 17:11:39 2016
>>>> Collection Status
>>>> -----------------
>>>> Connecting with backend: BackendWrapper
>>>> Archive dir: /root/Backup/cache/01f81df3eb37282f76c93fb89a3708f5
>>>>
>>>> Found 0 secondary backup chains.
>>>>
>>>> Found primary backup chain with matching signature chain:
>>>> -------------------------
>>>> Chain start time: Mon Jul 4 17:11:39 2016
>>>> Chain end time: Mon Jul 4 17:11:39 2016
>>>> Number of contained backup sets: 1
>>>> Total number of contained volumes: 254
>>>> Type of backup set: Time: Num
>>>> volumes:
>>>> Full Mon Jul 4 17:11:39 2016
>>>> 254
>>>> -------------------------
>>>> No orphaned or incomplete backup sets found.
>>>> Registering (mktemp) temporary file
>>>> /tmp/duplicity-2G5OwC-tempdir/mktemp-nXSXjQ-2
>>>> Deleting /tmp/duplicity-2G5OwC-tempdir/mktemp-nXSXjQ-2
>>>> Forgetting temporary file
>>>> /tmp/duplicity-2G5OwC-tempdir/mktemp-nXSXjQ-2
>>>> Processing remote manifest
>>>> duplicity-full.20160704T151139Z.manifest.gpg
>>>> (161677527)
>>>> Found manifest volume 1
>>>> Found manifest volume 2
>>>> Found manifest volume 3
>>>>
>>>> <SNIP>
>>>>
>>>> Found manifest volume 251
>>>> Found manifest volume 252
>>>> Found manifest volume 253
>>>> Killed
>>>>
>>>> Any idea what could cause this? If not, what is the best way to
>>>> debug?
>>>>
>>>> Thanks,
>>>> Chris.
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> <mailto:address@hidden> <mailto:address@hidden>
>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden <mailto:address@hidden>
>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden <mailto:address@hidden>
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden <mailto:address@hidden>
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
- [Duplicity-talk] Duplicity process killed, Chris Vanden Berghe, 2016/07/11
- Re: [Duplicity-talk] Duplicity process killed, Remy van Elst, 2016/07/11
- Re: [Duplicity-talk] Duplicity process killed, Chris Vanden Berghe, 2016/07/12
- Re: [Duplicity-talk] Duplicity process killed, Mark Grandi, 2016/07/12
- Re: [Duplicity-talk] Duplicity process killed, Nate Eldredge, 2016/07/12
- Re: [Duplicity-talk] Duplicity process killed,
Chris Vanden Berghe <=
- Re: [Duplicity-talk] Duplicity process killed, edgar . soldin, 2016/07/13
- Re: [Duplicity-talk] Duplicity process killed, Chris Vanden Berghe, 2016/07/13
- Re: [Duplicity-talk] Duplicity process killed, edgar . soldin, 2016/07/14
- Re: [Duplicity-talk] Duplicity process killed, Chris Vanden Berghe, 2016/07/14
- Re: [Duplicity-talk] Duplicity process killed, edgar . soldin, 2016/07/15
- [Duplicity-talk] Fwd: Re: Duplicity process killed, edgar . soldin, 2016/07/15
- Re: [Duplicity-talk] Fwd: Re: Duplicity process killed, Chris Vanden Berghe, 2016/07/15
- Re: [Duplicity-talk] Fwd: Re: Duplicity process killed, edgar . soldin, 2016/07/15
- Re: [Duplicity-talk] Fwd: Re: Duplicity process killed, edgar . soldin, 2016/07/15
- Re: [Duplicity-talk] Fwd: Re: Duplicity process killed, Kenneth Loafman, 2016/07/15