Re: [rdiff-backup-users] OverflowError reading get_sigs() response in very large tree

[dean gaudet]

i wonder why i don't have this problem on million+ inode systems... i've 
had it working with python2.3 before too (although i'm using 2.4 now)... 
i'm still on 1.0.x.

is there a 32/64-bit mismatch between your hosts?  any chance there's some 
bug in that?

-dean

On Sat, 13 Jan 2007, Charles Duffy wrote:

> Running a backup with a very large number of files (about 150,000) on both
> ends, one of the RPC requests returns a response larger than PY_SSIZE_T_MAX:
> 
> Traceback (most recent call last):
>  File "/usr/bin/rdiff-backup", line 23, in ?
>    rdiff_backup.Main.Main(sys.argv[1:])
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/Main.py", line 286, in
> Main
>    take_action(rps)
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/Main.py", line 256, in
> take_action
>    elif action == "backup": Backup(rps[0], rps[1])
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/Main.py", line 306, in
> Backup
>    backup.Mirror_and_increment(rpin, rpout, incdir)
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/backup.py", line 49, in
> Mirror_and_increment
>    dest_sigiter = DestS.get_sigs(dest_rpath)
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", line
> 445, in __call__
>    return apply(self.connection.reval, (self.name,) + args)
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", line
> 365, in reval
>    result = self.get_response(req_num)
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", line
> 314, in get_response
>    try: req_num, object = self._get()
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", line
> 239, in _get
>    data = self._read(length)
>  File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", line
> 208, in _read
>    return self.inpipe.read(length)
> OverflowError: requested number of bytes is more than a Python string can hold
> 
> This is, needless to say, a Bad Thing. There are certainly a few fixes
> possible (such as chunking the signatures -- perhaps by directory)... but if
> anyone can think of something better (or is familiar with upstream Python
> changes which might impact this issue, or otherwise has something to
> add/suggest), I'm very much all ears.
> 
> 
> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>