Re: [rdiff-backup-users] rdiff-backup dies with ENOMEM due to requesting enormous amounts of memory

[Patrik Dufresne]

Hello Jeff,

You problem is interesting to me ! Rdiff-backup doesn't have the tendency to use alot of memory (it rarely reach 100MB on my systems). It is surprising to see an out or memory error like this in rdiff-backup.

May you provide some information revargin your current environment:

1. How many memory free you have

2. May you describes how big it the stuff you are trying to backup (nb files, average file size, etc.)

3. x86 or x86_64 ?

4. What is you python version ?

Can you get your hand on the server logs too ? Causes it fail when requesting stuff from remote server...

--

Patrik Dufresne Service Logiciel inc.

http://www.patrikdufresne.com/

514-971-6442

1-114 rue des Hautbois,

St-Colomban, QC J5K 2H6

On Thu, Jun 9, 2016 at 6:54 PM, Jeff White <address@hidden> wrote:

I'm trying to get rdiff-backup working for the first time across SSH (with sudo) on two CentOS 7 hosts.  I'm getting the following crash:

$ sudo rdiff-backup --restrict-read-only / -v5 --remote-schema 'ssh -l rdiff -i /home/rdiff/.ssh/id_rsa -t -t %s "sudo /usr/bin/rdiff-backup --server"' bacula-p1n01::/ /dumps/bacula-p1n01/
Thu Jun  9 15:34:42 2016  Using rdiff-backup version 1.2.8
Thu Jun  9 15:34:42 2016  Executing ssh -l rdiff -i /home/rdiff/.ssh/id_rsa -t -t bacula-p1n01 "sudo /usr/bin/rdiff-backup --server"
Thu Jun  9 15:34:42 2016  Client sending (0): ConnectionRequest: Globals.get with 1 arguments
Thu Jun  9 15:34:42 2016  Client sending (0): 'version'
Connection to bacula-p1n01 closed.
Thu Jun  9 15:34:43 2016  Exception '' raised of class '<type 'exceptions.MemoryError'>':
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 450, in __call__
    return apply(self.connection.reval, (self.name,) + args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 368, in reval
    result = self.get_response(req_num)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 315, in get_response
    try: req_num, object = self._get()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 240, in _get
    data = "">   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 210, in _read
    try: return self.inpipe.read(length)

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 450, in __call__
    return apply(self.connection.reval, (self.name,) + args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 368, in reval
    result = self.get_response(req_num)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 315, in get_response
    try: req_num, object = self._get()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 240, in _get
    data = "">   File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 210, in _read
    try: return self.inpipe.read(length)
MemoryError


On that host when I strace the process I see:

[pid 11602] mmap(NULL, 18118029061677056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
...
[pid 11602] <... mmap resumed> )        = -1 ENOMEM (Cannot allocate memory)


... am I mistaken or is rdiff-backup requesting a ridiculous amount of memory?  Why would it do that?

--
Jeff White
HPC Systems Engineer
Information Technology Services - WSU


_______________________________________________
rdiff-backup-users mailing list at address@hidden
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki