Re: [Duplicity-talk] cache problems

[Scott Classen]

Hello

OK. I think the root (pun intended) of the problem is that I have both the distro python and my custom python installed.

The cron env is:

SHELL=/bin/sh
USER=root
PATH=/usr/bin:/bin
PWD=/root
LANG=en_US.UTF-8
SHLVL=1
HOME=/root
LOGNAME=root

The duplicity script uses:

#!/usr/bin/env python

which, in the limited cron environment  is picking up python 2.6 (the default system version).

I've changed this to:

#!/usr/bin/env /usr/local/bin/python

which now uses my custom installed python 2.7.13

I'm hoping that this will fix the cache issue.

On Apr 10, 2017, at 10:31 AM, edgar.soldin--- via Duplicity-talk <address@hidden> wrote:

Scott,

1.
i need the _complete_ output of the command.

This is the full output of the command that I run in crontab.

sh-4.1# /usr/bin/duply /etc/duply/home_a pre_full_purgeFull_post --force --preview
Start duply v2.0.1, time is 2017-04-10 11:20:25.
Using profile '/etc/duply/home_a'.
Using installed duplicity version 0.7.12, python 2.7.13, gpg 2.0.14 (Home: ~/.gnupg), awk 'GNU Awk 3.1.7', grep 'grep (GNU grep) 2.20', bash '4.1.2(2)-release (x86_64-redhat-linux-gnu)'.
Autoset found secret key of first GPG_KEY entry 'ABCDEFG' for signing.
-- Run cmd -- Checking TEMP_DIR '/duplicity' is a folder and writable --
test -d '/duplicity' && test -w '/duplicity' 2>&1
-- Run cmd -- Test - Encrypt to 'ABCDEFG' & Sign with 'ABCDEFG' --
echo password123 | gpg --sign --default-key ABCDEFG --passphrase-fd 0 --batch -r ABCDEFG --status-fd 1 -o '/duplicity/duply.23401.1491848425_ENC' -e '/usr/bin/duply' 2>&1
-- Run cmd -- Test - Decrypt --
echo password123 | gpg --passphrase-fd 0 --batch -o '/duplicity/duply.23401.1491848425_DEC' -d '/duplicity/duply.23401.1491848425_ENC' 2>&1
-- Run cmd -- Test - Compare --
test "$(cat '/usr/bin/duply')" = "$(cat '/duplicity/duply.23401.1491848425_DEC')" 2>&1
Cleanup - Delete '/duplicity/duply.23401.1491848425_*'(FAILED)

--- Start running command PRE at 11:20:25.775 ---
/etc/duply/home_a/pre
--- Finished state OK at 11:20:25.789 - Runtime 00:00:00.013 ---

--- Start running command FULL at 11:20:25.799 ---
TMPDIR='/duplicity'  PASSPHRASE=password123 FTP_PASSWORD='PASSWORD'  '/usr/local/bin/python' '/usr/bin/duplicity'  full --archive-dir '/duplicity/.duply-cache' --name duply_home_a --encrypt-key ABCDEFG --sign-key ABCDEFG --verbosity '4'    --volsize 500  --s3-use-rrs --asynchronous-upload     --force --exclude-filelist '/etc/duply/home_a/exclude' '/home' 's3://amazonaws.com/swq/a'
--- Finished state OK at 11:20:25.847 - Runtime 00:00:00.048 ---

--- Start running command PURGEFULL at 11:20:25.857 ---
TMPDIR='/home-old/duplicity'  PASSPHRASE=password123 FTP_PASSWORD='PASSWORD'  '/usr/local/bin/python' '/usr/bin/duplicity'  remove-all-but-n-full 4 --archive-dir '/home-old/duplicity/.duply-cache' --name duply_home_a --encrypt-key ABCDEFG --sign-key ABCDEFG --verbosity '4'    --volsize 500  --s3-use-rrs --asynchronous-upload     --force 's3://amazonaws.com/swq/a'
--- Finished state OK at 11:20:25.885 - Runtime 00:00:00.028 ---

--- Start running command POST at 11:20:25.896 ---
/etc/duply/home_a/post
--- Finished state OK at 11:20:25.910 - Runtime 00:00:00.014 ---

2.
additionally duply complains that it cannot parse your version

INFO:

duplicity version check failed (please report, this is a bug)

Using profile '/etc/duply/home_a'.
Using installed duplicity version MISSING,

that is unexpected.. ede


On 10.04.2017 18:36, Scott Classen wrote:

All my backup jobs are executed as cron jobs.
I just had a look at one of the log files from last night and I'm seeing the following at the top of the log file:

# cat /data/log/duply/home_a_incr_2017-04-10.log
Start duply v2.0.1, time is 2017-04-10 01:00:01.

INFO:

duplicity version check failed (please report, this is a bug)

Using profile '/etc/duply/home_a'.
Using installed duplicity version MISSING, python 2.7.13, gpg 2.0.14 (Home: ~/.gnupg), awk 'GNU Awk 3.1.7', grep 'grep (GNU grep) 2.20', bash '4.1.2(2)-release (x86_64-redhat-linux-gnu)'.
Autoset found secret key of first GPG_KEY entry '78ABCDEF' for signing.
Checking TEMP_DIR '/bup/duplicity' is a folder and writable (OK)
Test - Encrypt to '78ABCDEF' & Sign with '78ABCDEF' (OK)
Test - Decrypt (OK)
Test - Compare (OK)
Cleanup - Delete '/bup/duplicity/duply.30881.1491811201_*'(OK)

Perhaps there is something wrong with the environment variables for duply jobs run in a cron setting? The backup jobs are completing successfully. The main problem at the moment is that the cache files aren't going where they are supposed to.

Cheers,
Scott

On Apr 9, 2017, at 6:09 PM, Scott Classen <address@hidden <mailto:address@hidden>> wrote:

On Apr 9, 2017, at 11:09 AM, edgar.soldin--- via Duplicity-talk <address@hidden <mailto:address@hidden>> wrote:

On 09.04.2017 19:52, Scott Classen via Duplicity-talk wrote:

Hello,

I have a dozen or so duply profiles. All of them have the cache directory specified as a separate volume on my main backup machine.

e.g.

# default '~/.cache/duplicity/duply_<profile>/'
# if set  '${ARCH_DIR}/<profile>'
#ARCH_DIR=/some/space/safe/.duply-cache
ARCH_DIR=/bup/duplicity/.duply-cache

However, it appears that duply/duplicity is not honoring this configuration directive and all of the cache files are are going into /root/.cache/duplicity/
This is causing me severe headaches as the root partition is getting critically full.

hi Scott,

1.
what's your duply version?

2.0.1

2.
please run one backup w/ the '--preview' parameter and post it's result. check for and obfuscate private strings beforehand.

[root]# duply project status --preview
Start duply v2.0.1, time is 2017-04-09 18:04:36.
Using profile '/etc/duply/project'.
Using installed duplicity version 0.7.12, python 2.7.13, gpg 2.0.14 (Home: ~/.gnupg), awk 'GNU Awk 3.1.7', grep 'grep (GNU grep) 2.20', bash '4.1.2(2)-release (x86_64-redhat-linux-gnu)'.
Autoset found secret key of first GPG_KEY entry '78BD589D' for signing.
-- Run cmd -- Checking TEMP_DIR ‘/bup/duplicity' is a folder and writable --
test -d ‘/bup/duplicity' && test -w ‘/bup/duplicity' 2>&1
-- Run cmd -- Test - Encrypt to '78BD589D' & Sign with '78BD589D' --
echo password123 | gpg --sign --default-key 78BD589D --passphrase-fd 0 --batch -r 78BD589D --status-fd 1 -o ‘/bup/duplicity/duply.16873.1491786277_ENC' -e '/usr/bin/duply' 2>&1
-- Run cmd -- Test - Decrypt --
echo password123 | gpg --passphrase-fd 0 --batch -o ‘/bup/duplicity/duply.16873.1491786277_DEC' -d ‘/bup/duplicity/duply.16873.1491786277_ENC' 2>&1
-- Run cmd -- Test - Compare --
test "$(cat '/usr/bin/duply')" = "$(cat ‘/bup/duplicity/duply.16873.1491786277_DEC')" 2>&1
Cleanup - Delete ‘/bup/duplicity/duply.16873.1491786277_*'(FAILED)

--- Start running command STATUS at 18:04:37.283 ---
TMPDIR=‘/bup/duplicity'  PASSPHRASE=password123 FTP_PASSWORD=‘PASSWORD'  '/usr/local/bin/python' '/usr/local/bin/duplicity'  collection-status --archive-dir ‘/bup/duplicity/.duply-cache' --name duply_project --encrypt-key 78BD589D --sign-key 78BD589D --verbosity '4'    --s3-use-rrs --asynchronous-upload     's3://address@hidden/backup-project'
--- Finished state OK at 18:04:37.380 - Runtime 00:00:00.097 ---

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk