Re: [Axiom-mail] Server goes OOM when

[Geeth Munasinghe]

Hi All

Mistakenly email was sent without correct subject

Subject should be "Axis2 server goes OOM when ServiceTCCL parameter is used"

Thanks

Geeth

On Wed, Sep 17, 2014 at 1:07 PM, Geeth Munasinghe <address@hidden> wrote:

Hi all,

We are using axis2-1.6.1 version, it uses axiom 1.2.11 version, We recently encountered a OOM problem with two services which uses ServiceTCCL parameter. We are using this parameter because both those services use spring and hibernate with them. Scenario is one service is calling the other service. So one service acts as the client.

We analyzed the heap dumps and found out that OOM issue was caused by org.apache.axiom.om.util.StAXUtils class. There are few weakhashmaps used in that class. Two of them are

1. inputFactoryPerCLMap
2. outputFactoryPerCLMap

And those two weak hashmaps has other weak hashmaps inside of them. Those inner hashmaps cause the issue.

We found out that when we use ServiceTCCL parameter  in the service.xml, in AbstractMessageReceiver.java  [1] of axis2 (line number 152 - 170) creates a new class loader object of the MultiParentClassLoader [2] for every request. So in StAXUtils class [3] in axiom, methods [4] and [5] uses inner  weak hashmaps of inputFactoryPerCLMap,  outputFactoryPerCLMap and fill them with the class loader as the key and XMLInputFactory / XMLOutputFactory as the value. 

Because every request gets a new class loader object of the MultiParentClassLoader, their hash values are different. So they are keep getting filled into those inner weak hash maps. Because the same key (classloder instance) gets inserted into both weak hashmaps, garbage collector does not remove them. So server goes OOM, When we analyze the heap dumps we found out that java.util.WeakHashMap fills over 80% of the memory when it goes OOM.

I have made fix in StAXUtils in axiom as follows.

Instead map.get(cl) I change it to map.get(cl.getClass().getName())

Instead map.put(cl, factory) I changed it to map.put(cl.getClass().getName(), factory)

I have attached the fix (svn diff) here with email. I am not sure I have done the correct fix for the issue. But I found that it solves my problem. Can some one please verify weather I have done the correct fix.

Please consider that upgrading to new axis2 is not a solution for us at the moment.

[1] http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modules/kernel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java

[2] org.apache.axis2.classloader.MultiParentClassLoader

[3] org.apache.axiom.om.util.StAXUtils

[4] getXMLInputFactory_perClassLoader(StAXParserConfiguration configuration)

[5] getXMLOutputFactory_perClassLoader(StAXWriterConfiguration configuration)

Thanks in advance.

Geeth