Re: [Duplicity-talk] full backup killed

On Thu, Apr 20, 2017 at 8:31 AM, Russell Clemings via Duplicity-talk <address@hidden> wrote:

Possibly. I hope not though because this is a fairly new VPS from a well-known provider. I guess we'll find out if it happens again.

On Thu, Apr 20, 2017 at 3:42 AM, edgar.soldin--- via Duplicity-talk <address@hidden> wrote:
maybe something has a memory leak and filled up your memory?.. ede/duply.net

On 20.04.2017 07:15, Russell Clemings via Duplicity-talk wrote:
> Well, what do you know? The reboot seemed to have fixed the problem, whatever it was.
>
> I was already using the current version.
>
> thx
>
> rac
>
> On Wed, Apr 19, 2017 at 2:34 AM, edgar.soldin--- via Duplicity-talk <address@hidden <mailto:address@hiddenorg>> wrote:
>
> Russell,
>
> 1.
> do you use the latest duplicity 0.7.12 ? if not, try it.
>
> 2.
> if the problem persists, can you reboot the box and run a full and after it was killed, zip the complete 'dmesg' output and send it to me personally?
>
> ..ede&/duply.net <http://duply.net>
>
>
> On 19.04.2017 00:11, Nate Eldredge via Duplicity-talk wrote:
> > That's weird. My next suggestion would be strace -vvv to try to determine if the process is SIGKILLing itself, or if some other process is doing it, or if it's coming from the kernel (though I'd expect a log message in that case).
> >
> > On Tue, 18 Apr 2017, Russell Clemings via Duplicity-talk wrote:
> >
> >> This is CentOS 7. I'm not seeing anything in the logs about a memory issue
> >> in any case.
> >>
> >> On Tue, Apr 18, 2017 at 2:30 PM, Nate Eldredge <address@hidden <mailto:address@hiddencom>>
> >> wrote:
> >>
> >>> This happens on Linux machines (maybe others) when the machine runs out of
> >>> memory. If that's the case, you should see an entry in your syslog talking
> >>> about oom-killer. Check your syslog in any case.
> >>>
> >>> I was encountering this recently on an Ubuntu 16.04 machine. There was a
> >>> kernel bug where the kernel would think it was out of memory, and start
> >>> killing processes, even though there was plenty of swap. See
> >>> https://bugs.launchpad.net/bugs/1655842 <https://bugs.launchpad.net/bugs/1655842>. It is fixed in recent updates
> >>> (you will see people there claiming it is not fixed, but I think they are
> >>> either finding different bugs or they really are out of memory). At least,
> >>> since updating it no longer happens for me.
> >>>
> >>>
> >>> On Tue, 18 Apr 2017, Russell Clemings via Duplicity-talk wrote:
> >>>
> >>> This looks like the same problem discussed last summer in
> >>>> https://lists.nongnu.org/archive/html/duplicity-talk/2016- <https://lists.nongnu.org/archive/html/duplicity-talk/2016->

> >>>> 07/msg00005.html
> >>>> but I can't see that there was any resolution.
> >>>>
> >>>> Full (but not incremental) backups are (apparently at least) failing with
> >>>> no traceback, just the word "Killed," and the lockfile is not removed.
> >>>>
> >>>> I've pasted the last lines below from the most recent attempt.
> >>>>
> >>>> It seems to be failing at same place in the backup every time (after
> >>>> var/spool/mail/wcsj2017). But the "Processing local manifest" filename is
> >>>> different each time.
> >>>>
> >>>> It worked fine until a few days ago. And as I said, incremental backups
> >>>> are
> >>>> fine (although the full backup failure leaves a lock file that has to be
> >>>> removed; same filename each time). Running collection-status yields the
> >>>> results I expect -- a list of full backups and associated incrementals,
> >>>> and
> >>>> "No orphaned or incomplete backup sets found." So maybe the backup is
> >>>> succeeding despite the "Failed" message and lock problem.
> >>>>
> >>>> I'm not seeing anything that would suggest a memory or load issue. Just in
> >>>> case, I've run it at different times of day as well as in the middle of
> >>>> the
> >>>> night, with the same results.
> >>>>
> >>>> Suggestions? Should I maybe clean out
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d on the theory
> >>>> that
> >>>> the backend is fine but the local cache is corrupted?
> >>>>
> >>>> A var/spool/mail/swnet
> >>>> A var/spool/mail/wcsj17dv
> >>>> A var/spool/mail/wcsj2017
> >>>> AsyncScheduler: running task synchronously (asynchronicity disabled)
> >>>> Writing duplicity-full.20170418T140701Z.vol115.difftar.gpg
> >>>> Uploading s3://

> >>>> s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z <http://s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z>.
> >>>> vol115.difftar.gpg
> >>>> to STANDARD Storage
> >>>> Deleting
> >>>> /home/newton/duplicity-0.7.12/s3backuptmp/duplicity-7y1aYN-t
> >>>> empdir/mktemp-d8rFt2-116
> >>>> AsyncScheduler: task completed successfully
> >>>> Processed volume 115
> >>>> Writing duplicity-full-signatures.20170418T140701Z.sigtar.gpg
> >>>> Uploading s3://
> >>>> s3.amazonaws.com/BUCKETNAME/duplicity-full-signatures.201704 <http://s3.amazonaws.com/BUCKETNAME/duplicity-full-signatures.201704>
> >>>> 18T140701Z.sigtar.gpg
> >>>> to STANDARD Storage
> >>>> Deleting
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full-signatures.20170418T140701Z.sigtar.gpg
> >>>> Writing duplicity-full.20170418T140701Z.manifest.gpg
> >>>> Uploading s3://
> >>>> s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z.manifest.gpg <http://s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z.manifest.gpg>

> >>>> to
> >>>> STANDARD Storage
> >>>> Deleting
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170418T140701Z.manifest.gpg
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170128T070703Z.manifest
> >>>> (28635198)
> >>>> Found 97 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170204T070703Z.manifest
> >>>> (28771709)
> >>>> Found 99 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170211T070707Z.manifest
> >>>> (28798317)
> >>>> Found 100 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170218T070703Z.manifest
> >>>> (28785755)
> >>>> Found 102 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170225T070705Z.manifest
> >>>> (29127918)
> >>>> Found 104 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170304T070703Z.manifest
> >>>> (29236665)
> >>>> Found 106 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170311T070712Z.manifest
> >>>> (29239486)
> >>>> Found 107 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170318T060703Z.manifest
> >>>> (29246628)
> >>>> Found 109 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170325T060703Z.manifest
> >>>> (29266901)
> >>>> Found 111 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170401T060704Z.manifest
> >>>> (29375862)
> >>>> Found 112 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170401T151402Z.manifest
> >>>> (29378174)
> >>>> Found 112 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170408T060702Z.manifest
> >>>> (29085454)
> >>>> Found 113 volumes in manifest
> >>>> Processing local manifest
> >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
> >>>> duplicity-full.20170415T060703Z.manifest
> >>>> (28559242)
> >>>> /home/newton/duplicity-0.7.10/s3backup/backup_full.sh: line 97: 9126
> >>>> Killed /home/newton/duplicity-0.7.12/bin/duplicity full
> >>>> --verbosity 8 --timeout=30 --num-retries=5 --volsize=250
> >>>> --tempdir=/home/newton/duplicity-0.7.12/s3backuptmp
> >>>> --encrypt-key=${GPG_KEY} --sign-key=${GPG_KEY}
> >>>> --exclude=/home/newton/duplicity-0.7.10/s3backup
> >>>> --exclude=/home/newton/duplicity-0.7.12/s3backup --exclude=/home/.cpan
> >>>> --exclude=/home/.cpcpan --exclude=/home/virtfs --exclude=/root/.cache
> >>>> --exclude=/proc --exclude=/dev --exclude=/home/scie2010/public_html/media
> >>>> --exclude=/home/mailarch-nasw/archives --exclude=/etc/csf/csf.deny
> >>>> --exclude=/etc/udev/devices/ploop54946p1
> >>>> --exclude=/etc/valiases/nasw_aliases
> >>>> --exclude=/home/cybraria/.spamassassinenable
> >>>> --exclude=/home/sciencew/public_html/stats/logs
> >>>> --exclude=/root/.cpanel/caches/dynamicui --exclude=/root/.MirrorSearch/

> >>>> myip.cpanel.net <http://myip.cpanel.net> --exclude=/root/tmp --exclude=/usr/local/apache/domlogs
> >>>> --exclude=/usr/local/apache/logs --exclude=/usr/local/cpanel/3r
> >>>> dparty/share
> >>>> --exclude=/usr/local/cpanel/base/frontend
> >>>> --exclude=/usr/local/cpanel/img-sys --exclude=/usr/local/cpanel/logs
> >>>> --exclude=/usr/local/cpanel/src --exclude=/usr/local/cpanel/src/nsswitch
> >>>> --exclude=/usr/local/cpanel/src/redhat_pathtools
> >>>> --exclude=/usr/local/cpanel/src/remove_sendmail_rpm --include=/etc
> >>>> --include=/home/clamav --include=/home/cybraria --include=/home/devnasw
> >>>> --include=/home/mailarch-nasw --include=/home/newton
> >>>> --include=/home/scie2008 --include=/home/scie2009 --include=/home/scie2010
> >>>> --include=/home/scie2011 --include=/home/scie2012 --include=/home/scie2013
> >>>> --include=/home/scie2014 --include=/home/scie2015 --include=/home/scie2016
> >>>> --include=/home/scie2017 --include=/home/scie2018 --include=/home/scie2019
> >>>> --include=/home/scie2020 --include=/home/scie2021 --include=/home/scie2022
> >>>> --include=/home/sciencew --include=/home/solr --include=/home/sw2016dv
> >>>> --include=/home/swmeeting --include=/home/swnet --include=/home/wcsj17dv
> >>>> --include=/home/wcsj2017 --include=/root --include=/s3backup
> >>>> --include=/scripts --include=/usr/local --include=/var/lib/mysql
> >>>> --include=/var/mail --include=/var/named --include=/var/spool
> >>>> --exclude=/**
> >>>> ${SOURCE} ${DEST}
> >>>>
> >>>>
> >>> --
> >>> Nate Eldredge
> >>> address@hidden <mailto:address@hiddencom>
> >>>
> >>>
> >>
> >>
> >> --
> >> ===============================================
> >> Russell Clemings
> >> <address@hidden <mailto:address@hidden>>
> >> ===============================================
> >>
> >
>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden <mailto:address@hiddenorg>
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
>
>
>
>
> --
> ===============================================
> Russell Clemings
> <address@hidden <mailto:address@hidden>>

> ===============================================
>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

--
===============================================
Russell Clemings
<address@hidden>
===============================================

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

From:	Kenneth Loafman
Subject:	Re: [Duplicity-talk] full backup killed
Date:	Fri, 21 Apr 2017 06:31:09 -0500