Re: [Duplicity-talk] cache problems

[edgar . soldin]

why didn't you simply set PATH=/usr/local/bin:/usr/bin:/bin in cron?

can you send me the output from 'duplicity --version', when run with the old 
python 2.6 please?

..ede/duply.net

On 10.04.2017 20:42, Scott Classen wrote:
> Hello
> 
> OK. I think the root (pun intended) of the problem is that I have both the 
> distro python and my custom python installed.
> 
> The cron env is:
> 
> SHELL=/bin/sh
> USER=root
> PATH=/usr/bin:/bin
> PWD=/root
> LANG=en_US.UTF-8
> SHLVL=1
> HOME=/root
> LOGNAME=root
> 
> 
> The duplicity script uses:
> 
> #!/usr/bin/env python
> 
> which, in the limited cron environment  is picking up python 2.6 (the default 
> system version).
> 
> I've changed this to:
> 
> #!/usr/bin/env /usr/local/bin/python
> 
> which now uses my custom installed python 2.7.13
> 
> I'm hoping that this will fix the cache issue.
> 
>> On Apr 10, 2017, at 10:31 AM, edgar.soldin--- via Duplicity-talk 
>> <address@hidden <mailto:address@hidden>> wrote:
>>
>> Scott,
>>
>> 1.
>> i need the _complete_ output of the command.
> 
> 
> This is the full output of the command that I run in crontab.
> 
> 
> sh-4.1# /usr/bin/duply /etc/duply/home_a pre_full_purgeFull_post --force 
> --preview
> Start duply v2.0.1, time is 2017-04-10 11:20:25.
> Using profile '/etc/duply/home_a'.
> Using installed duplicity version 0.7.12, python 2.7.13, gpg 2.0.14 (Home: 
> ~/.gnupg), awk 'GNU Awk 3.1.7', grep 'grep (GNU grep) 2.20', bash 
> '4.1.2(2)-release (x86_64-redhat-linux-gnu)'.
> Autoset found secret key of first GPG_KEY entry 'ABCDEFG' for signing.
> -- Run cmd -- Checking TEMP_DIR '/duplicity' is a folder and writable --
> test -d '/duplicity' && test -w '/duplicity' 2>&1
> -- Run cmd -- Test - Encrypt to 'ABCDEFG' & Sign with 'ABCDEFG' --
> echo password123 | gpg --sign --default-key ABCDEFG --passphrase-fd 0 --batch 
> -r ABCDEFG --status-fd 1 -o '/duplicity/duply.23401.1491848425_ENC' -e 
> '/usr/bin/duply' 2>&1
> -- Run cmd -- Test - Decrypt --
> echo password123 | gpg --passphrase-fd 0 --batch -o 
> '/duplicity/duply.23401.1491848425_DEC' -d 
> '/duplicity/duply.23401.1491848425_ENC' 2>&1
> -- Run cmd -- Test - Compare --
> test "$(cat '/usr/bin/duply')" = "$(cat 
> '/duplicity/duply.23401.1491848425_DEC')" 2>&1
> Cleanup - Delete '/duplicity/duply.23401.1491848425_*'(FAILED)
> 
> --- Start running command PRE at 11:20:25.775 ---
> /etc/duply/home_a/pre
> --- Finished state OK at 11:20:25.789 - Runtime 00:00:00.013 ---
> 
> --- Start running command FULL at 11:20:25.799 ---
> TMPDIR='/duplicity'  PASSPHRASE=password123 FTP_PASSWORD='PASSWORD'  
> '/usr/local/bin/python' '/usr/bin/duplicity'  full --archive-dir 
> '/duplicity/.duply-cache' --name duply_home_a --encrypt-key ABCDEFG 
> --sign-key ABCDEFG --verbosity '4'    --volsize 500  --s3-use-rrs 
> --asynchronous-upload     --force --exclude-filelist 
> '/etc/duply/home_a/exclude' '/home' 's3://amazonaws.com/swq/a'
> --- Finished state OK at 11:20:25.847 - Runtime 00:00:00.048 ---
> 
> --- Start running command PURGEFULL at 11:20:25.857 ---
> TMPDIR='/home-old/duplicity'  PASSPHRASE=password123 FTP_PASSWORD='PASSWORD'  
> '/usr/local/bin/python' '/usr/bin/duplicity'  remove-all-but-n-full 4 
> --archive-dir '/home-old/duplicity/.duply-cache' --name duply_home_a 
> --encrypt-key ABCDEFG --sign-key ABCDEFG --verbosity '4'    --volsize 500  
> --s3-use-rrs --asynchronous-upload     --force 's3://amazonaws.com/swq/a'
> --- Finished state OK at 11:20:25.885 - Runtime 00:00:00.028 ---
> 
> --- Start running command POST at 11:20:25.896 ---
> /etc/duply/home_a/post
> --- Finished state OK at 11:20:25.910 - Runtime 00:00:00.014 ---
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>>
>> 2.
>> additionally duply complains that it cannot parse your version
>>
>>> INFO:
>>>
>>> duplicity version check failed (please report, this is a bug)
>>>
>>> Using profile '/etc/duply/home_a'.
>>> Using installed duplicity version MISSING,
>>
>> that is unexpected.. ede
>>
>>
>> On 10.04.2017 18:36, Scott Classen wrote:
>>> All my backup jobs are executed as cron jobs.
>>> I just had a look at one of the log files from last night and I'm seeing 
>>> the following at the top of the log file:
>>>
>>> # cat /data/log/duply/home_a_incr_2017-04-10.log
>>> Start duply v2.0.1, time is 2017-04-10 01:00:01.
>>>
>>> INFO:
>>>
>>> duplicity version check failed (please report, this is a bug)
>>>
>>> Using profile '/etc/duply/home_a'.
>>> Using installed duplicity version MISSING, python 2.7.13, gpg 2.0.14 (Home: 
>>> ~/.gnupg), awk 'GNU Awk 3.1.7', grep 'grep (GNU grep) 2.20', bash 
>>> '4.1.2(2)-release (x86_64-redhat-linux-gnu)'.
>>> Autoset found secret key of first GPG_KEY entry '78ABCDEF' for signing.
>>> Checking TEMP_DIR '/bup/duplicity' is a folder and writable (OK)
>>> Test - Encrypt to '78ABCDEF' & Sign with '78ABCDEF' (OK)
>>> Test - Decrypt (OK)
>>> Test - Compare (OK)
>>> Cleanup - Delete '/bup/duplicity/duply.30881.1491811201_*'(OK)
>>>
>>> Perhaps there is something wrong with the environment variables for duply 
>>> jobs run in a cron setting? The backup jobs are completing successfully. 
>>> The main problem at the moment is that the cache files aren't going where 
>>> they are supposed to.
>>>
>>> Cheers,
>>> Scott
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Apr 9, 2017, at 6:09 PM, Scott Classen <address@hidden 
>>>> <mailto:address@hidden> <mailto:address@hidden>> wrote:
>>>>
>>>>>
>>>>> On Apr 9, 2017, at 11:09 AM, edgar.soldin--- via Duplicity-talk 
>>>>> <address@hidden <mailto:address@hidden> <mailto:address@hidden>> wrote:
>>>>>
>>>>> On 09.04.2017 19:52, Scott Classen via Duplicity-talk wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I have a dozen or so duply profiles. All of them have the cache 
>>>>>> directory specified as a separate volume on my main backup machine.
>>>>>>
>>>>>> e.g.
>>>>>>
>>>>>> # default '~/.cache/duplicity/duply_<profile>/'
>>>>>> # if set  '${ARCH_DIR}/<profile>'
>>>>>> #ARCH_DIR=/some/space/safe/.duply-cache
>>>>>> ARCH_DIR=/bup/duplicity/.duply-cache
>>>>>>
>>>>>> However, it appears that duply/duplicity is not honoring this 
>>>>>> configuration directive and all of the cache files are are going into 
>>>>>> /root/.cache/duplicity/
>>>>>> This is causing me severe headaches as the root partition is getting 
>>>>>> critically full.
>>>>>>
>>>>>
>>>>> hi Scott,
>>>>>
>>>>> 1.
>>>>> what's your duply version?
>>>>
>>>> 2.0.1
>>>>
>>>>
>>>>
>>>>>
>>>>> 2.
>>>>> please run one backup w/ the '--preview' parameter and post it's result. 
>>>>> check for and obfuscate private strings beforehand.
>>>>
>>>>
>>>> [root]# duply project status --preview
>>>> Start duply v2.0.1, time is 2017-04-09 18:04:36.
>>>> Using profile '/etc/duply/project'.
>>>> Using installed duplicity version 0.7.12, python 2.7.13, gpg 2.0.14 (Home: 
>>>> ~/.gnupg), awk 'GNU Awk 3.1.7', grep 'grep (GNU grep) 2.20', bash 
>>>> '4.1.2(2)-release (x86_64-redhat-linux-gnu)'.
>>>> Autoset found secret key of first GPG_KEY entry '78BD589D' for signing.
>>>> -- Run cmd -- Checking TEMP_DIR ‘/bup/duplicity' is a folder and writable 
>>>> --
>>>> test -d ‘/bup/duplicity' && test -w ‘/bup/duplicity' 2>&1
>>>> -- Run cmd -- Test - Encrypt to '78BD589D' & Sign with '78BD589D' --
>>>> echo password123 | gpg --sign --default-key 78BD589D --passphrase-fd 0 
>>>> --batch -r 78BD589D --status-fd 1 -o 
>>>> ‘/bup/duplicity/duply.16873.1491786277_ENC' -e '/usr/bin/duply' 2>&1
>>>> -- Run cmd -- Test - Decrypt --
>>>> echo password123 | gpg --passphrase-fd 0 --batch -o 
>>>> ‘/bup/duplicity/duply.16873.1491786277_DEC' -d 
>>>> ‘/bup/duplicity/duply.16873.1491786277_ENC' 2>&1
>>>> -- Run cmd -- Test - Compare --
>>>> test "$(cat '/usr/bin/duply')" = "$(cat 
>>>> ‘/bup/duplicity/duply.16873.1491786277_DEC')" 2>&1
>>>> Cleanup - Delete ‘/bup/duplicity/duply.16873.1491786277_*'(FAILED)
>>>>
>>>> --- Start running command STATUS at 18:04:37.283 ---
>>>> TMPDIR=‘/bup/duplicity'  PASSPHRASE=password123 FTP_PASSWORD=‘PASSWORD'  
>>>> '/usr/local/bin/python' '/usr/local/bin/duplicity'  collection-status 
>>>> --archive-dir ‘/bup/duplicity/.duply-cache' --name duply_project 
>>>> --encrypt-key 78BD589D --sign-key 78BD589D --verbosity '4'    --s3-use-rrs 
>>>> --asynchronous-upload     's3://address@hidden/backup-project'
>>>> --- Finished state OK at 18:04:37.380 - Runtime 00:00:00.097 ---
>>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden <mailto:address@hidden>
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>