Re: [Duplicity-talk] full backup killed

[edgar . soldin]

maybe something has a memory leak and filled up your memory?.. ede/duply.net

On 20.04.2017 07:15, Russell Clemings via Duplicity-talk wrote:
> Well, what do you know? The reboot seemed to have fixed the problem, whatever 
> it was.
> 
> I was already using the current version.
> 
> thx
> 
> rac
> 
> On Wed, Apr 19, 2017 at 2:34 AM, edgar.soldin--- via Duplicity-talk 
> <address@hidden <mailto:address@hidden>> wrote:
> 
>     Russell,
> 
>     1.
>     do you use the latest duplicity 0.7.12 ? if not, try it.
> 
>     2.
>     if the problem persists, can you reboot the box and run a full and after 
> it was killed, zip the complete 'dmesg' output and send it to me personally?
> 
>     ..ede&/duply.net <http://duply.net>
> 
> 
>     On 19.04.2017 00:11, Nate Eldredge via Duplicity-talk wrote:
>     > That's weird.  My next suggestion would be strace -vvv to try to 
> determine if the process is SIGKILLing itself, or if some other process is 
> doing it, or if it's coming from the kernel (though I'd expect a log message 
> in that case).
>     >
>     > On Tue, 18 Apr 2017, Russell Clemings via Duplicity-talk wrote:
>     >
>     >> This is CentOS 7. I'm not seeing anything in the logs about a memory 
> issue
>     >> in any case.
>     >>
>     >> On Tue, Apr 18, 2017 at 2:30 PM, Nate Eldredge <address@hidden 
> <mailto:address@hidden>>
>     >> wrote:
>     >>
>     >>> This happens on Linux machines (maybe others) when the machine runs 
> out of
>     >>> memory.  If that's the case, you should see an entry in your syslog 
> talking
>     >>> about oom-killer.  Check your syslog in any case.
>     >>>
>     >>> I was encountering this recently on an Ubuntu 16.04 machine.  There 
> was a
>     >>> kernel bug where the kernel would think it was out of memory, and 
> start
>     >>> killing processes, even though there was plenty of swap.  See
>     >>> https://bugs.launchpad.net/bugs/1655842 
> <https://bugs.launchpad.net/bugs/1655842>.  It is fixed in recent updates
>     >>> (you will see people there claiming it is not fixed, but I think they 
> are
>     >>> either finding different bugs or they really are out of memory).  At 
> least,
>     >>> since updating it no longer happens for me.
>     >>>
>     >>>
>     >>> On Tue, 18 Apr 2017, Russell Clemings via Duplicity-talk wrote:
>     >>>
>     >>> This looks like the same problem discussed last summer in
>     >>>> https://lists.nongnu.org/archive/html/duplicity-talk/2016- 
> <https://lists.nongnu.org/archive/html/duplicity-talk/2016->
>     >>>> 07/msg00005.html
>     >>>> but I can't see that there was any resolution.
>     >>>>
>     >>>> Full (but not incremental) backups are (apparently at least) failing 
> with
>     >>>> no traceback, just the word "Killed," and the lockfile is not 
> removed.
>     >>>>
>     >>>> I've pasted the last lines below from the most recent attempt.
>     >>>>
>     >>>> It seems to be failing at same place in the backup every time (after
>     >>>> var/spool/mail/wcsj2017). But the "Processing local manifest" 
> filename is
>     >>>> different each time.
>     >>>>
>     >>>> It worked fine until a few days ago. And as I said, incremental 
> backups
>     >>>> are
>     >>>> fine (although the full backup failure leaves a lock file that has 
> to be
>     >>>> removed; same filename each time). Running collection-status yields 
> the
>     >>>> results I expect -- a list of full backups and associated 
> incrementals,
>     >>>> and
>     >>>> "No orphaned or incomplete backup sets found." So maybe the backup is
>     >>>> succeeding despite the "Failed" message and lock problem.
>     >>>>
>     >>>> I'm not seeing anything that would suggest a memory or load issue. 
> Just in
>     >>>> case, I've run it at different times of day as well as in the middle 
> of
>     >>>> the
>     >>>> night, with the same results.
>     >>>>
>     >>>> Suggestions? Should I maybe clean out
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d on the theory
>     >>>> that
>     >>>> the backend is fine but the local cache is corrupted?
>     >>>>
>     >>>> A var/spool/mail/swnet
>     >>>> A var/spool/mail/wcsj17dv
>     >>>> A var/spool/mail/wcsj2017
>     >>>> AsyncScheduler: running task synchronously (asynchronicity disabled)
>     >>>> Writing duplicity-full.20170418T140701Z.vol115.difftar.gpg
>     >>>> Uploading s3://
>     >>>> s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z 
> <http://s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z>.
>     >>>> vol115.difftar.gpg
>     >>>> to STANDARD Storage
>     >>>> Deleting
>     >>>> /home/newton/duplicity-0.7.12/s3backuptmp/duplicity-7y1aYN-t
>     >>>> empdir/mktemp-d8rFt2-116
>     >>>> AsyncScheduler: task completed successfully
>     >>>> Processed volume 115
>     >>>> Writing duplicity-full-signatures.20170418T140701Z.sigtar.gpg
>     >>>> Uploading s3://
>     >>>> s3.amazonaws.com/BUCKETNAME/duplicity-full-signatures.201704 
> <http://s3.amazonaws.com/BUCKETNAME/duplicity-full-signatures.201704>
>     >>>> 18T140701Z.sigtar.gpg
>     >>>> to STANDARD Storage
>     >>>> Deleting
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full-signatures.20170418T140701Z.sigtar.gpg
>     >>>> Writing duplicity-full.20170418T140701Z.manifest.gpg
>     >>>> Uploading s3://
>     >>>> 
> s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z.manifest.gpg 
> <http://s3.amazonaws.com/BUCKETNAME/duplicity-full.20170418T140701Z.manifest.gpg>
>     >>>> to
>     >>>> STANDARD Storage
>     >>>> Deleting
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170418T140701Z.manifest.gpg
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170128T070703Z.manifest
>     >>>> (28635198)
>     >>>> Found 97 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170204T070703Z.manifest
>     >>>> (28771709)
>     >>>> Found 99 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170211T070707Z.manifest
>     >>>> (28798317)
>     >>>> Found 100 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170218T070703Z.manifest
>     >>>> (28785755)
>     >>>> Found 102 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170225T070705Z.manifest
>     >>>> (29127918)
>     >>>> Found 104 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170304T070703Z.manifest
>     >>>> (29236665)
>     >>>> Found 106 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170311T070712Z.manifest
>     >>>> (29239486)
>     >>>> Found 107 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170318T060703Z.manifest
>     >>>> (29246628)
>     >>>> Found 109 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170325T060703Z.manifest
>     >>>> (29266901)
>     >>>> Found 111 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170401T060704Z.manifest
>     >>>> (29375862)
>     >>>> Found 112 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170401T151402Z.manifest
>     >>>> (29378174)
>     >>>> Found 112 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170408T060702Z.manifest
>     >>>> (29085454)
>     >>>> Found 113 volumes in manifest
>     >>>> Processing local manifest
>     >>>> /root/.cache/duplicity/5c439fc8a10be3c159ce21e965eef31d/
>     >>>> duplicity-full.20170415T060703Z.manifest
>     >>>> (28559242)
>     >>>> /home/newton/duplicity-0.7.10/s3backup/backup_full.sh: line 97:  9126
>     >>>> Killed                  /home/newton/duplicity-0.7.12/bin/duplicity 
> full
>     >>>> --verbosity 8 --timeout=30 --num-retries=5 --volsize=250
>     >>>> --tempdir=/home/newton/duplicity-0.7.12/s3backuptmp
>     >>>> --encrypt-key=${GPG_KEY} --sign-key=${GPG_KEY}
>     >>>> --exclude=/home/newton/duplicity-0.7.10/s3backup
>     >>>> --exclude=/home/newton/duplicity-0.7.12/s3backup 
> --exclude=/home/.cpan
>     >>>> --exclude=/home/.cpcpan --exclude=/home/virtfs --exclude=/root/.cache
>     >>>> --exclude=/proc --exclude=/dev 
> --exclude=/home/scie2010/public_html/media
>     >>>> --exclude=/home/mailarch-nasw/archives --exclude=/etc/csf/csf.deny
>     >>>> --exclude=/etc/udev/devices/ploop54946p1
>     >>>> --exclude=/etc/valiases/nasw_aliases
>     >>>> --exclude=/home/cybraria/.spamassassinenable
>     >>>> --exclude=/home/sciencew/public_html/stats/logs
>     >>>> --exclude=/root/.cpanel/caches/dynamicui 
> --exclude=/root/.MirrorSearch/
>     >>>> myip.cpanel.net <http://myip.cpanel.net> --exclude=/root/tmp 
> --exclude=/usr/local/apache/domlogs
>     >>>> --exclude=/usr/local/apache/logs --exclude=/usr/local/cpanel/3r
>     >>>> dparty/share
>     >>>> --exclude=/usr/local/cpanel/base/frontend
>     >>>> --exclude=/usr/local/cpanel/img-sys --exclude=/usr/local/cpanel/logs
>     >>>> --exclude=/usr/local/cpanel/src 
> --exclude=/usr/local/cpanel/src/nsswitch
>     >>>> --exclude=/usr/local/cpanel/src/redhat_pathtools
>     >>>> --exclude=/usr/local/cpanel/src/remove_sendmail_rpm --include=/etc
>     >>>> --include=/home/clamav --include=/home/cybraria 
> --include=/home/devnasw
>     >>>> --include=/home/mailarch-nasw --include=/home/newton
>     >>>> --include=/home/scie2008 --include=/home/scie2009 
> --include=/home/scie2010
>     >>>> --include=/home/scie2011 --include=/home/scie2012 
> --include=/home/scie2013
>     >>>> --include=/home/scie2014 --include=/home/scie2015 
> --include=/home/scie2016
>     >>>> --include=/home/scie2017 --include=/home/scie2018 
> --include=/home/scie2019
>     >>>> --include=/home/scie2020 --include=/home/scie2021 
> --include=/home/scie2022
>     >>>> --include=/home/sciencew --include=/home/solr 
> --include=/home/sw2016dv
>     >>>> --include=/home/swmeeting --include=/home/swnet 
> --include=/home/wcsj17dv
>     >>>> --include=/home/wcsj2017 --include=/root --include=/s3backup
>     >>>> --include=/scripts --include=/usr/local --include=/var/lib/mysql
>     >>>> --include=/var/mail --include=/var/named --include=/var/spool
>     >>>> --exclude=/**
>     >>>> ${SOURCE} ${DEST}
>     >>>>
>     >>>>
>     >>> --
>     >>> Nate Eldredge
>     >>> address@hidden <mailto:address@hidden>
>     >>>
>     >>>
>     >>
>     >>
>     >> --
>     >> ===============================================
>     >> Russell Clemings
>     >> <address@hidden <mailto:address@hidden>>
>     >> ===============================================
>     >>
>     >
> 
> 
>     _______________________________________________
>     Duplicity-talk mailing list
>     address@hidden <mailto:address@hidden>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk 
> <https://lists.nongnu.org/mailman/listinfo/duplicity-talk>
> 
> 
> 
> 
> -- 
> ===============================================
> Russell Clemings
> <address@hidden <mailto:address@hidden>>
> ===============================================
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>