[Mldonkey-bugs] [bugs #7967] memory leak and lockup

[spiralvoice]

This mail is an automated notification from the bugs tracker
 of the project: mldonkey, a multi-networks file-sharing client.

/**************************************************************************/
[bugs #7967] Latest Modifications:

Changes by: 
                spiralvoice <address@hidden>
'Date: 
                Fri 07/23/04 at 00:23 (Europe/Berlin)

------------------ Additional Follow-up Comments ----------------------------
You are not the first one mentioning problems on a SMP system. What is the 
situation now?






/**************************************************************************/
[bugs #7967] Full Item Snapshot:

URL: <http://savannah.nongnu.org/bugs/?func=detailitem&item_id=7967>
Project: mldonkey, a multi-networks file-sharing client
Submitted by: Mark-Oliver Wolter
On: Tue 03/02/04 at 00:37

Category:  Core
Severity:  5 - Average
Item Group:  Memory leak
Resolution:  None
Assigned to:  None
Status:  Open
Release:  2-5-12
Release:  2-5-12
Platform Version:  Linux
Binaries Origin:  CVS / Self compiled
CPU type:  None


Summary:  memory leak and lockup

Original Submission:  Moin,

I'm seeing something not completely unlike bugs #7222 in that my self-compiled 
2-5-12 eats a lot of memory.
However, the mldonkey-2.5-4.static.i686-Linux.tar.bz2 I used before didn't.

So I think the problem lies maybe not in the mldonkey itself but in the way we 
compile it, ie gcc & libs ...

What happened here is:
After running since Jan 2, on Feb 25 I shut down the old mlnet 2.5-4:
  637 mow       14   0 39704  38M  2088 R     0  2.1 31.1  2278m mlnet

As you can see, memory usage is quite low. Download was about 3.4GB during that 
time, with about 12GB upload.

Then I installed the newly-built 2-5-12 (by the way, thankyouverymuch for 
rm-rf'ing the ocaml tree after copying the libs to the safe place failed due to 
insufficient disk space, thus having to compile the stuff a second time).
The new mldonkey quickly went to download and even appeared to perform better 
than the older version, and initially even the memory footprint looked better:
 8602 mow       12   0 15156  14M  2824 R     0 25.5 11.8   2:01 mlnet
However, it didn't stay like that.
Due to a power outage, I had to boot up the machine on Saturday.
This morning, I found mlnet was dead.
A straight line in mldonkeywatch stats.
Memory footprint now didn't look so good:
  892 mow        7   0  170M 106M  2472 D     1  3.5 85.9 228:33 mlnet
strace -p 892 showed ... nothing.
lsof showed 125 open thingies.
telnet to 4000 also hung.
I then murdered mlnet by sending an -IOT/ABRT.

Then tried to restart:
[...]
File ./fasttrack.ini.tmp exists
An error may have occurred during previous configuration save.
Please, check your configurations files, and rename/remove this file
before restarting
SAVING SHARED FILES AND SOURCES
SAVED
Network.save_complex_options not implemented by BitTorrent
Options correctly saved

What mlnet did, is:
a) remove said fasttrack.ini.tmp itself
b) move servers.ini, files.ini, file_sources.ini, friends.ini, 
shared_files_new.ini to .old and put default files in their location.

Maybe rotating .ini files and then bailing out before writing new ones is not 
that good an idea.

Okay, next start:
"Looks like you have no servers in your servers.ini"
... yes, thank you, you just removed them. Okay, apparently you got yourself a 
new list, too, so I won't complain much.

Well, mlnet starts up, all seems fine ...
... except, a bunch of file names from the downloads are missing, only the 
hashes show. [btw, mldonkeywatch nitpick, don't know if the problem lies with 
mldw or with mlnet itself: changing to another filename from the dropdown list 
reverts to the old filename on the next refresh]
Now, hashes got replaced by filenames as files were found.
However, "pause" settings were gone also.

Now mlnet has been running for 10h16mins.
mow      23887  9.5 65.9 87124 84016 ?       S    13:25  58:58 mlnet -daemon

That's quite a lot of memory.

Now, let's see what we have here:
# ldd /usr/local/bin/mlnet
    libz.so.1 => /usr/lib/libz.so.1 (0x4001d000)
    libm.so.6 => /lib/libm.so.6 (0x4002c000)
    libdl.so.2 => /lib/libdl.so.2 (0x4004e000)
    libc.so.6 => /lib/libc.so.6 (0x40052000)
    /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
those libs pointing to:
-rwxr-xr-x   1 root     root        55772 Mar 11  2002 /usr/lib/libz.so.1.1.4*
-rwxr-xr-x   1 root     root       561250 May 26  2001 /lib/libm-2.2.3.so*
-rwxr-xr-x   1 root     root        72701 May 26  2001 /lib/libdl-2.2.3.so*
-rwxr-xr-x   1 root     root      4787276 Jan 11  2002 /lib/libc-2.2.3.so*
-rwxr-xr-x   1 root     root       432647 May 26  2001 /lib/ld-2.2.3.so*

Compiler:
# gcc --version
2.95.3

Machine:
Linux good 2.2.26ext3 #4 SMP Thu Feb 25 05:30:18 CET 2004 i686 unknown

configure invocation:
  $ ./configure
(auto-gets and builds ocaml-3.07pl2)
... followed by make depend and make, as told.

Now, it would be *really* interesting whether a mldonkey-2-5-12-static built in 
the same way as the 2.5-4-static leaks or not. Could the one who produced those 
binaries please do so again, that I can test whether it's mldonkey or just my 
environment that acts up?

BTW: This is a machine with two DSL lines. Default route is on the PPPOE 
interface, thus changing IP every 24 hours.
The old 2.5-4 first ran on the PPPOE line, then kept getting more and more 
(Overnet?) connects over the fixed IP line, leading to, at first, IP-flapping 
in the mldonkeywatch Settings:Core window, then after a few weeks it stood 
solid on the fixed IP. A problem was, that some servers then complained "ERROR: 
Your userhash doesn't match the login one", so it seems mldonkey sent something 
based on the wrong IP. Unfortunately, I can't check that with 2-5-12 because it 
simply doesn't run long enough.

Hope that made sense.
MOW
(mlnet has now 100840K. Shutting down in a few.)

Follow-up Comments
------------------


-------------------------------------------------------
Date: Fri 07/23/04 at 00:23         By: spiralvoice
You are not the first one mentioning problems on a SMP system. What is the 
situation now?

-------------------------------------------------------
Date: Sat 05/08/04 at 20:35         By: m-o-w
Okay, following up with the information I gathered since.

Memory use does not continuously increase, but also goes back down, sometimes a 
lot, e.g. when mldonkeywatch is closed or a big file is finished. This might 
mean there is not really a memory leak, just mlnet has become a lot more bloaty 
somewhere since 2.5-4 (see also bugs #8187). With 2.5-16l I have even seen 
memory usage going down from about 80MB back to 20.

Your mldonkey-2.5-12b.static.i386-Linux.rar exhibited exactly the same 
behavior, aside from the multithreading which I disabled in my compile. [1]

With the 2.5-16 cores, I noticed that the behavior chanced so that mlnet didn't 
lockup after a while but still responded, albeit slowly due to swapping.
However, this brought a new problem: As mlnet continues to work, it also 
continues to grow.
After a while, this leads to the machine freezing. I even replaced the power 
supply before I concluded it's mlnet's fault (and the kernel's, due to 
inability to handle such a growing user program properly).

I then set ulimits:
ulimit -d 100000 -l 100000 -m 102400 -v 110000
(this is a bit redundant, but you never know ...)
This caused mlnet to at least shut down not too ungracefully, leaving the linux 
box alive. However, yesterday I noticed it triggered another problem, see bugs 
#8580 - and mlnet still runs just for 1-2 days before it gets so big that it 
runs out of memory, which apparently is a condition certain programmers thought 
of as something that can never happen.

I'll now go and download 2.5-16m.
Is it already safe to try your 2.5-21b?

[1] weird stuff with threads: mlnet starts with 3 threads, then after a while a 
fourth appears. Is this normal? I have an SMP machine.

-------------------------------------------------------
Date: Tue 05/04/04 at 19:18         By: spiralvoice
Please check with current CVS versions if the memory leak is still there. 
Regarding the saving of ini files with low free HDD space there is a new option:
2004/03/22: Fabrice
  - Safer options-saving at exit: close all the sockets to prevent "not
     enough file descriptors" error and remove a file called
    "config_files_space.tmp" to free 'config_files_security_space' megabytes
    created at startup.

-------------------------------------------------------
Date: Tue 03/16/04 at 00:29         By: spiralvoice
You can get static binaries here: http://www.8ung.at/spiralvoice/




CC List
-------

CC Address                          | Comment
------------------------------------+-----------------------------
t8m                                 | 









For detailed info, follow this link:
<http://savannah.nongnu.org/bugs/?func=detailitem&item_id=7967>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.nongnu.org/