Re: [Duplicity-talk] Fwd: GPG taking many ressources

[Henrik Olsen]

On Aug 14, 2010, at 9:32 PM, Kenneth Loafman wrote:

> Henrik Olsen wrote:
>> On Aug 12, 2010, at 2:02 PM, address@hidden wrote:
>> 
>>> On 12.08.2010 13:21, Kenneth Loafman wrote:
>>>> Jacob Godserv wrote:
>>>>> On Tue, Aug 10, 2010 at 17:25, Henrik
>>>>> Olsen<address@hidden>  wrote:
>>>>>> As I'm still having the issue, I was wondering if GPG 1.4.10
>>>>>> could be the issue, and I should try 2.x instead?
>>>> There are still some interface changes that need to be made for
>>>> 2.x, but it might work.
>>> I recall having used duplicity together with gpg2 flawlessly.
>>> Worked for me.
>> 
>> Trying with gpg2 now. Took a bit to get it to compile with the many
>> dependencies, but now duplicity can use it, as long as I change
>> self.call from gpg to gpg2 in GnuPGInterface.py.
>> 
>> But... Unfortunately I still see a heavy ressource usage, exactly
>> like with gpg. Now with gpg2 as well, and as I noticed yesterday with
>> gpg, also even if only one process is running:
>> 
>> ps aux | grep gpg root     98278  95.1  0.0  2436668    804 s000  RN+
>> 8:42PM   2:28.35 gpg2 --status-fd 5 --passphrase-fd 10 --logger-fd 3
>> --batch --no-tty --no-secmem-warning --decrypt holsen   98287   0.8
>> 0.0  2435036    528 s002  S+    8:44PM   0:00.00 grep gpg
>> 
>> On non-OSX usage, do anyone ever see gpg usage like this?
> 
> Did gpg2 solve the issue of multiple gpg processes running?  From your
> example, it looks like it did.  If so, that was the issue that I was
> trying to solve.  Every gpg process that is still open is tying up 3
> file descriptors and a very small amount of memory, but no CPU.  With
> long chains of incrementals some people were running out of open files.

My duplicity backup seems to get stock. It has now used over 10 hours of CPU 
time on the single running gpg2 process and still no apparent progress. Last 
log message was still about copying the last signature file to local cache. 

I tried doing a stack sample of the process from OS X's Acticity Monitor, and 
got this - perhaps someone can get some clue as to what is, or not, going on:

Analysis of sampling gpg2 (pid 98435) every 1 millisecond
Process:         gpg2 [98435]
Path:            /usr/local/duplicity/bin/gpg2
Load Address:    0x100000000
Identifier:      gpg2
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  Python [98360]

Date/Time:       2010-08-15 07:38:52.303 +0200
OS Version:      Mac OS X 10.6.4 (10F569)
Report Version:  6

Call graph:
    2473 Thread_441772   DispatchQueue_1: com.apple.main-thread  (serial)
      2473 start
        2473 main
          2473 decrypt_message
            2473 proc_encryption_packets
              2473 do_proc_packets
                2473 proc_encrypted
                  2473 decrypt_data
                    2473 proc_packets
                      2473 do_proc_packets
                        2473 proc_compressed
                          2473 handle_compressed
                            2473 proc_packets
                              2473 do_proc_packets
                                2473 proc_compressed
                                  2473 free_packet
                                    2473 free_compressed
                                      2473 iobuf_read
                                        2473 underflow
                                          2472 compress_filter
                                            2446 do_uncompress
                                              1660 inflate
                                                1312 inflate
                                                334 inflateCopy
                                                  179 inflateCopy
                                                  144 __memcpy
                                                  11 memcpy
                                                14 dyld_stub_memcpy
                                              434 do_uncompress
                                              352 iobuf_read
                                                213 iobuf_read
                                                139 underflow
                                            24 dyld_stub_inflate
                                            2 compress_filter
                                          1 underflow

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        inflate        1312
        do_uncompress        434
        iobuf_read        213
        inflateCopy        179
        __memcpy        144
        underflow        140
        dyld_stub_inflate        24
        dyld_stub_memcpy        14
        memcpy        11

Binary Images:
       0x100000000 -        0x1000d0fef +gpg2 ??? (???) 
<D5FE0363-D19C-5D7D-A023-15810C617AC9> /usr/local/duplicity/bin/gpg2
       0x1000f0000 -        0x1001a0ff7 +libgcrypt.11.dylib 18.0.0 
(compatibility 18.0.0) <21CBE8FD-3206-7E27-E23F-3A271B08C160> 
/tmp/duplicity-build/lib/libgcrypt.11.dylib
       0x1001c0000 -        0x1001c2ff7 +libgpg-error.0.dylib 8.0.0 
(compatibility 8.0.0) <53EE8593-3089-0F2F-EC04-37764ED37656> 
/tmp/duplicity-build/lib/libgpg-error.0.dylib
       0x1001c5000 -        0x1001d4ff7 +libassuan.0.dylib 2.0.0 (compatibility 
2.0.0) <3F40FF14-FB00-249F-863D-736A809D8300> 
/tmp/duplicity-build/lib/libassuan.0.dylib
    0x7fff5fc00000 -     0x7fff5fc3bdef  dyld 132.1 (???) 
<B536F2F1-9DF1-3B6C-1C2C-9075EA219A06> /usr/lib/dyld
    0x7fff816de000 -     0x7fff8189efef  libSystem.B.dylib 125.2.0 
(compatibility 1.0.0) <95E02DD0-ADEA-745B-E7FA-ABA064E4658C> 
/usr/lib/libSystem.B.dylib
    0x7fff81dac000 -     0x7fff81dbdff7  libz.1.dylib 1.2.3 (compatibility 
1.0.0) <FB5EE53A-0534-0FFA-B2ED-486609433717> /usr/lib/libz.1.dylib
    0x7fff82181000 -     0x7fff8218dfff  libbz2.1.0.dylib 1.0.5 (compatibility 
1.0.0) <5C876577-ACB7-020C-F7DB-EE0135C3AB8D> /usr/lib/libbz2.1.0.dylib
    0x7fff821ba000 -     0x7fff821beff7  libmathCommon.A.dylib 315.0.0 
(compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5> 
/usr/lib/system/libmathCommon.A.dylib
    0x7fff841db000 -     0x7fff842d3ff7  libiconv.2.dylib 7.0.0 (compatibility 
7.0.0) <7E4ADB5A-CC77-DCFD-3E54-2F35A2C8D95A> /usr/lib/libiconv.2.dylib
    0x7fffffe00000 -     0x7fffffe01fff  libSystem.B.dylib ??? (???) 
<95E02DD0-ADEA-745B-E7FA-ABA064E4658C> /usr/lib/libSystem.B.dylib

/Henrik