| Author |
Message
|
| mqwbiwf |
 Posted: Thu Nov 13, 2008 7:16 am Post subject: Failed to allocate memory abend error |
 |
|
Centurion
Joined: 21 Jul 2006
Posts: 126
|
Hi,
We have a message flow that seems to be consuming a lot of memory and eventually unable to process the message that we are trying to process. The msg size itself is just about 10MB.
We are running MQ 6.0 FP 4, Message Broker 6.0 FP 7 on AIX 5.3 RL 8.
Below are some of the variables that we tried to increase, but still nothing seems to be working our way.
LDR_CNTRL is set to use 7 segments (tried increasing, Broker wasn't even strating at that point)
MQSI_THREAD_STACK_SIZE is increased to use 8MB(tried using 24MB too, didn't work).
I am sure message broker would handle much larger data quite easily, but for some reason in our case it isn't. And this is raising serious red flags all across. I would greatly appreciate any expert advice to address our issue.
Thanks. |
|
| Back to top |
|
 |
| m4c0 |
| Posted: Thu Nov 13, 2008 7:20 am Post subject: |
|
|
Novice
Joined: 07 Nov 2008
Posts: 17
|
| Which node is consuming lots of memory? Compute, Mapping, Java Compute? |
|
| Back to top |
|
|
| mqwbiwf |
| Posted: Thu Nov 13, 2008 7:50 am Post subject: |
|
|
Centurion
Joined: 21 Jul 2006
Posts: 126
|
| We have about 13 Compute nodes, a few output nodes, one http request node, a couple of filter nodes and 2 small sub flows. |
|
| Back to top |
|
|
| Tibor |
| Posted: Thu Nov 13, 2008 8:00 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| Can you post the beginning of the stack trace from the abend file? |
|
| Back to top |
|
|
| mqwbiwf |
| Posted: Thu Nov 13, 2008 8:12 am Post subject: |
|
|
Centurion
Joined: 21 Jul 2006
Posts: 126
|
abend record for pid 753704 tid 6426 time Thu Nov 13 08:57:25 2008
File: /build/S600_P/src/CommonServices/Unix/ImbAbend.cpp
Line: 1958
Function: ImbAbend::newHandler
---- Inserts ----
Failed to allocate memory
-----------------
----------------------------- Stack dump for current thread ( 6426)
(malloc allowed)
(0xd1a539d8+0x00000204) dumpCurrentStacksToFile__8ImbAbendFPCvi [/broker_install_path/lib/libCommonServices.a(libCommonService
s.a.so)]
(0xd1a515e4+0x00001198) dumpToFile__8ImbAbendFPcPCciPCcRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertRCQ2_8Im
bAbend6InsertRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertPCv [/broker_install_path/lib/libCommonServices.a(libCommonServices.a.s
o)]
(0xd19bcfc0+0x00000180) logSystemErrorInternal__8ImbAbendFPCciPCcRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6Inse
rtRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertPCvb [/broker_install_path/lib/libCommonServices.a(libCommonS
ervices.a.so)]
(0xd199c314+0x00000110) logSystemErrorAndTerminateProcessInternal__8ImbAbendFPCciPCcRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6Insert
RCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertRCQ2_8ImbAbend6InsertPCvb [/broker_install_path/lib/libCommonSe
rvices.a(libCommonServices.a.so)]
(0xd1b23948+0x0000006c) newHandler__8ImbAbendFv [/broker_install_path/lib/libCommonServices.a(libCommonServices.a.so)]
(0xd03167a0+0x00000118) __nw__FUl [/usr/lib/libC.a(ansicore_32.o)]
(0xd7336edc+0x000002e4) __ct__Q2_19ImbXmlMemoryManager10MemoryPoolFUlT1 [/broker_install_path/lib/libbipprefixmgr.a(libbippref
ixmgr.a.so)]
(0xd73365e8+0x00000100) __ct__19ImbXmlMemoryManagerFUl [/broker_install_path/lib/libbipprefixmgr.a(libbipprefixmgr.a.so)]
(0xd757cfb4+0x000001c4) __ct__12XmlImbParserFP15ImbMessageGroupRC10ImbWstringibUl [/broker_install_path/lil/GenXmlParser2.lil]
(0xd757c388+0x000002ec) createParser__19XmlImbParserFactoryFRC10ImbWstringP15ImbMessageGroup [/broker_install_path/lil/GenXmlP
arser2.lil]
(0xd200749c+0x00000720) createParserInner__15ImbMessageGroupFRC10ImbWstring [/broker_install_path/lib/libMessageServices.a(lib
MessageServices.a.so)]
(0xd2006ce0+0x00000210) createParser__15ImbMessageGroupFRC10ImbWstringb [/broker_install_path/lib/libMessageServices.a(libMess
ageServices.a.so)]
(0xd203602c+0x00000058) createLastChild__16ImbSyntaxElementFRC10ImbWstring [/broker_install_path/lib/libMessageServices.a(libM
essageServices.a.so)]
(0xd69ddd68+0x000000b8) createParserLastChild__Q2_10ImbMessage11WriteCursorFRC10ImbWstring [/broker_install_path/lib/libImbRdl
.a(libImbRdl.a.so)]
(0xd69dd0d8+0x0000041c) createAsLastChild__14SqlPathElementCFR14SqlWriteCursorQ2_21ImbMessageDefinitions11CompareModeiRC10ImbW
stringRC10ImbWstringPC27SqlDefaultsForMsgNodeCreate [/broker_install_path/lib/libImbRdl.a(libImbRdl.a.so)]
@ |
|
| Back to top |
|
|
| Tibor |
| Posted: Thu Nov 13, 2008 8:30 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| What about the user limits of your broker user? Could you send an output of ulimit -a? |
|
| Back to top |
|
|
| mqwbiwf |
| Posted: Thu Nov 13, 2008 8:57 am Post subject: |
|
|
Centurion
Joined: 21 Jul 2006
Posts: 126
|
$ ulimit -a
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 2000 |
|
| Back to top |
|
|
| mqwbiwf |
| Posted: Thu Nov 13, 2008 10:22 am Post subject: |
|
|
Centurion
Joined: 21 Jul 2006
Posts: 126
|
We increased the stack_hard limit in the '/etc/security/limits' file to unlimited and tried. Still the same result.
Any advice from WMB experts? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Nov 13, 2008 10:50 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| What does User Trace show that the flow is doing? Where does it show that the flow is spending all of it's time? |
|
| Back to top |
|
|
| mqwbiwf |
| Posted: Thu Nov 13, 2008 11:34 am Post subject: |
|
|
Centurion
Joined: 21 Jul 2006
Posts: 126
|
I think it's taking a lot of time at the last compute node where we have PROPGATE stmt.
The flow basically produces about 2000+ messages from one single message. |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Fri Dec 05, 2008 1:47 am Post subject: Similar errors on Solaris |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
This may not help on AIX but check it.
On Solaris ulimit stacksize unlimited results in 2Gb of memory being reserved for the stack, which violently reduces the amount of process space available for normal processing ( shot in the paddock! ).
Took us along time to elicit this information from Sun/IBM.
Perhaps the same or similar applies in AIX. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Dec 05, 2008 3:18 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
|
| Back to top |
|
|
| giorginus80 |
| Posted: Tue May 05, 2009 6:44 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
|
| Back to top |
|
|
| chanduy9 |
| Posted: Mon Dec 20, 2010 9:21 pm Post subject: How did you sloved the problem?? |
|
|
Disciple
Joined: 28 Nov 2001
Posts: 177
Location: USA
|
Hi,
We are encountering similar probleme with 100MB files inserting in to DB, can you share how did you solved the problem . We are getting same error...
Thanks,
Chandra.
_________________
Chandra,
IBM WebSphere MQ Certified. |
|
| Back to top |
|
|
| bloomy |
| Posted: Wed Dec 22, 2010 9:02 am Post subject: |
|
|
Acolyte
Joined: 11 Feb 2009
Posts: 61
Location: Bloomington, IL USA
|
we have had similar problems with AIX and WMB6. We had a PMR with IBM support and worked for a few months to figure out and convince IBM that there was actually a memory leak happening in some nodes.
They adviced us to set an env variable:
MQSI_FREE_MASTER_PARSERS=1.
Please use this only if adviced by IBM as there are some problems encountered using this setting and MBFE and file nodes also, IBM provided us with some fixes for those issues.
So please open a PMR and find out from IBM to see if this is valid in your case. |
|
| Back to top |
|
|
|
|
