[Duplicity-talk] Management of the temporary storage

[Pascal Levasseur]

Hello all,

I have an issue with the duplicity verify command, all other duplicity
commands seems OK.

The duplicity verify command process the first volumes perfectly, then
abort with this message (sorry for the paranoid xxxx) :

Running 'ncftpget
-f /home/zoe/tmp/duplicity-EYYkrt-tempdir/mkstemp-SrTlr3-1 -F -t 30 -o
useCLNT=0,useHELP_SITE=0  -V -C 'ftp.xxxxx.net'
'Duplicity/duplicity-full.20090729T153503Z.vol5.difftar.gpg'
'/home/zoe/tmp/duplicity-EYYkrt-tempdir/mktemp-5PTaAJ-15'' (attempt #1)
Deleting /home/zoe/tmp/duplicity-EYYkrt-tempdir/mktemp-hqSU5Q-9
Processed volume 6 of 214
Running 'ncftpget
-f /home/zoe/tmp/duplicity-EYYkrt-tempdir/mkstemp-SrTlr3-1 -F -t 30 -o
useCLNT=0,useHELP_SITE=0  -V -C 'ftp.xxxxx.net'
'Duplicity/duplicity-inc.20090801T110014Z.to.20090810T064752Z.vol2.difftar.gpg' 
'/home/zoe/tmp/duplicity-EYYkrt-tempdir/mktemp-ha9PCR-16'' (attempt #1)
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 825, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 818, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 777, in main
    verify(col_stats)
  File "/usr/bin/duplicity", line 554, in verify
    for backup_ropath, current_path in collated:
  File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line
256, in collate2iters
    relem1 = riter1.next()
  File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py", line
492, in integrate_patch_iters
    final_ropath = patch_seq2ropath(normalize_ps(patch_seq))
  File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py", line
472, in patch_seq2ropath
    misc.copyfileobj(current_file, tempfp)
  File "/usr/lib/python2.5/site-packages/duplicity/misc.py", line 170,
in copyfileobj
    outfp.write(buf)
IOError: [Errno 28] No space left on device


This error is not transient, it fails every time I execute the command.

The /tmp volume (350 Mb) shows a steady increase up to 97% of used space
until duplicity crash, then goes back to a very low value (df /tmp).


Filesystem           1K-blocks      Used Available Use% Mounted
on /dev/mapper/loadstone-tmp
                        376807    344394     12957  97% /tmp

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/loadstone-tmp
                        376807     10311    347040   3% /tmp

I was not able to find in the /tmp hierarchy the file(s) related to this
increase of used space.

The duplicity temporary directory (--tempdir) increase up to 60 Mb
 (du --si /home/zoe/tmp) then is cleared of any files by duplicity after
duplicity abort.

60M     /home/zoe/tmp/duplicity-EYYkrt-tempdir


My configuration is :

- Debian Linux 2.6.26-2-62-686 version 5 with luks crypto file system,
- /tmp is a separate volume with a fixed size. I dont want to increase
its size, as the actual size seems ok for all the applications except
Duplicity and I dont want to mess with the crypto file system. 
- Duplicity version 0.5.18,
- Backend is ftp with ncfftp 3.2.1,
- The Duplicity backup size is more than 200 volumes of 25 MBytes.   
- The shell script used to start the duplicity verify command is :

FTP_PASSWORD=xxxxxxx
export FTP_PASSWORD
#
PASSPHRASE=xxxxxxx
export PASSPHRASE
#
TMPDIR=/home/zoe/tmp/
export TMPDIR
#
TEMP=/home/zoe/tmp/
export TEMP
#
TMP=/home/zoe/tmp/
export TMP
#
duplicity verify -v5 --num-retries 10 --ftp-passive \
--tempdir /home/zoe/tmp/
--exclude-globbing-filelist /home/zoe/duplicity_xxxxx/duplicity_exclude_files \
ftp://address@hidden/Duplicity/ /home/zoe/Sauvegarde/


I think I have used all the "by the book" options to redirect temporary
files outside of /tmp, but it seems that I fail to find the complete
solution !


Thanks in advance for your help.

Pascal Levasseur