| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| JDBC adapter memory blown |
« View previous topic :: View next topic » |
| Author |
Message
|
| chrisc |
 Posted: Sun Jul 23, 2006 9:57 pm Post subject: JDBC adapter memory blown |
 |
|
Voyager
Joined: 19 Mar 2006
Posts: 77
|
Hi there,
We've got a problem with our JDBC adapter at the moment that we're struggling to come to grips with. Our WBIMB message flows are delivering XML messages to the adapter for loading into the database, and each object is approximately 4MB in size.
However, we keep getting JVM OutOfMemoryExceptions getting thrown by the adapter, and it seems that it's choking on the concurrent processing of these messages. 
We have increased our JVM max heap size to 512MB, and reduced our jms.NumConcurrentRequests to 3, but it is still dying. It's hard to believe that 3 x 4MB messages is capable of exceeding 512MB of heap space, but that certainly seems to be the case.
One of the other guys at our work as said anecdotally that they can't seem to run messages more than 20MB in size through the adapter, and yet we can't even reach that.
Does anybody have any ideas as to why this is happening, whether this is a known issue with the JDBC adapter and how we can get around it? It seems a pity to have to cripple the adapter by reducing the threads to 1 or 2, just because a few of our messages are larger in size...
Any help or ideas would be greatly appreciated.
Chris |
|
| Back to top |
|
 |
| chrisc |
| Posted: Sun Jul 23, 2006 10:49 pm Post subject: |
|
|
Voyager
Joined: 19 Mar 2006
Posts: 77
|
Just to add to my message above, we are using AIX machines, running version 2.6.0 of the adapter framework. There is ample free memory on the machine, even with all the other processes running (i.e. about 2GB of free memory!)
We just ran a Java trace and found that it starts off with 28MB in memory, and is failing on a memory allocation request for about 520MB (in one hit), which is why it's exceeding the 512MB limit.
I know we could just bump up the heap space, but 512MB for a couple of 4MB messages seems like a pretty inefficient use of resources!!! :S |
|
| Back to top |
|
|
| djeripo |
| Posted: Tue Jul 25, 2006 11:31 am Post subject: |
|
|
Master
Joined: 25 Jan 2004
Posts: 225
|
There are couple of things which you can try,
First run topas to see the memory usage on the box.If you dont have acces to topas(not even a pseudo access) try downloading NMON from IBM site and run it.You should see something like this.
Physical PagingSpace pages/sec In Out FileSystemCache │
│% Used 99.9% 79.0% to Paging Space 0.9 0.0 Process & 20.2% │
│% Free 0.1% 21.0% to File System 0.0 1.4 System 79.7% │
│MB Used 4093.5MB 6473.5MB Page Scans 0.0 │
│MB Free 2.4MB 1718.5MB Page Cycles 0.0 Free 0.1% │
│Total(MB) 4095.9MB 8192.0MB Page Steals 0.0 ------ │
│ Page Faults 561.6 Total 100.0% │
By that you can findout whats the physical memory available on the box and the amount utilized.And also see whether its paging out or not.
The reason why I am telling all of the above things is that,Even if you try to increase the -xMx value to 1024,Its not going to be helpful since the box might not have enough memory available and can throw "Panic OutOfMemory".
If you have enough memory available on the box,increase the -xMx parameter in the adapterEnv.sh to 1024 and give it a try.
If there is no sufficient memory available on the box,Try tweaking the MAXDATA of LDR_CNTRL.
By default it would be set to,
echo $LDR_CNTRL=MAXDATA
MAXDATA=0x30000000=MAXDATA
Increase that to,
export LDR_CNTRL=MAXDATA=0x80000000.
Let me know how it goes. |
|
| Back to top |
|
|
| chrisc |
| Posted: Wed Jul 26, 2006 4:33 pm Post subject: |
|
|
Voyager
Joined: 19 Mar 2006
Posts: 77
|
Actually I have just been informed we are running on HPUX, not AIX. I'm not sure if the LDR_CNTRL tweak still applies?
We did some profiling and there is definitely enough physical memory on the machine (the machine has something like 6GB, and there was more than 2GB free).
We have increased the max memory to 1024 and we can now process all the messages without a problem, with 10 threads! I'm wondering if there is perhaps a bare minimum that the adapter requires, and it is a fairly large chunk.
Also, does the trace level have an effect on the memory consumption of the adapter?
Cheers,
Chris
[/i] |
|
| Back to top |
|
|
| djeripo |
| Posted: Fri Jul 28, 2006 6:14 am Post subject: |
|
|
Master
Joined: 25 Jan 2004
Posts: 225
|
Glad to hear that you have overcome that problem.TraceLevel doesnt make any impact on the memory rather it has impact on the performance(Throughput).So,On a production system Tracelevel should always be '0'.
If you think that you have increased the heap size to a fairly large extent,Enable verbose:gc on your adapter.After collecting verbose:gc log perform a thorough analysis on it which would tell you exactly how much max and min heap you would require.While collecting verbose gc logs make sure that,You run maximum load.
You can get a free tool called GCCollector from IBM site.It gives you a graphical analysis of your memory usage once you provide verbose gc log as input. |
|
| Back to top |
|
|
| chrisc |
| Posted: Sun Jul 30, 2006 3:00 pm Post subject: |
|
|
Voyager
Joined: 19 Mar 2006
Posts: 77
|
Thanks for the tip - that GCCollector looks very interesting. I'll have to check it out and see what it's trying to do behind the scenes.
Incidentally, some of the testing guys found just how BIG the impact is of not setting TraceLevel to 0... They accidentally left it on 5 after doing some diagnosis work, and then did some stress and volume testing with it still on 5. They wasted about a day wondering why everything that was previously taking 15 minutes to complete blew out to over an hour!  |
|
| Back to top |
|
|
| djeripo |
| Posted: Tue Aug 01, 2006 8:20 pm Post subject: |
|
|
Master
Joined: 25 Jan 2004
Posts: 225
|
Changing the tracelevel from '0' to '5' has degraded the performance by 45 mts (60-15)? Surprising !
How large are your BO's ? Are you using any WrapperBO's? |
|
| Back to top |
|
|
| chrisc |
| Posted: Tue Aug 01, 2006 8:58 pm Post subject: |
|
|
Voyager
Joined: 19 Mar 2006
Posts: 77
|
Yes, in fact it was probably more like 1hr 15 mins instead of 15 mins, so about 5 times longer. 
Our business objects are nearly all composite (i.e. header with multiple details) and in some cases even more complex than this (header, details, sub-details). I think the ones that really killed it with the trace level on 5 were the ones with a single header and thousands of detail lines, because every individual line required a database insert and all the associated logging stuff.
I guess that is one of the downsides of a generic adapter that aims to fit many purposes. We could write a much more efficient adapter for our particular case, but it has business implications in that such an adapter would not be IBM-supported. Risk vs reward and all that, I guess. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
