| Author |
Message
|
| PeterPotkay |
 Posted: Tue May 13, 2008 3:52 pm Post subject: MQ 6.0.2.1 and Windows and Memory - XY129005 |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
I sent this out last fall but never got a response. Its reared its head again. Has anyone seen this and determined what the root cause is? We do have a PMR open but as they say, no joy yet.
An amqrmppa process goes bad. It start writing to an FDC over and over.
The QM and system MQ error logs get the below error over and over. I can
get out of this by killing the PID that corresponds to the amqrmppa
process. Obviously those channels using that amqrmppa process get whacked,
but all others and the QM remain fine. The channels that got whacked
reconnect via the other existing amqrmppa processes, or the listener
spawns a new one if needed. I see this only on the Windows 2000 servers.
Maybe once every couple weeks, whereas before the MQ 6.0 upgrade it would
happen once a year if that.
Task Manager says there is plenty of memory available while this is
happening.
Sometimes killing the PID once is enough and its fine until the next scheduled reboot which I assume cleans up memory. But other times it only buys us a few hours.
10/19/2007 06:55:57 - Process(5104.28402) User(mqadmin_ad1)
Program(amqrmppa.exe)
AMQ6090: MQM could not display the text for error 268460419.
----- amqxfdcp.c : 810
--------------------------------------------------------
10/19/2007 06:55:57 - Process(5104.28402) User(mqadmin_ad1)
Program(amqrmppa.exe)
AMQ6090: MQM could not display the text for error 536895769.
----- amqxfdcp.c : 766
--------------------------------------------------------
10/19/2007 06:55:57 - Process(5104.28402) User(mqadmin_ad1)
Program(amqrmppa.exe)
AMQ6090: MQM could not display the text for error 268460419.
----- amqxfdcp.c : 810
--------------------------------------------------------
10/19/2007 06:55:57 - Process(5104.28403) User(mqadmin_ad1)
Program(amqrmppa.exe)
AMQ6090: MQM could not display the text for error 536895769.
----- amqxfdcp.c : 766
--------------------------------------------------------
10/19/2007 06:55:57 - Process(5104.28403) User(mqadmin_ad1)
Program(amqrmppa.exe)
AMQ6090: MQM could not display the text for error 268460419.
----- amqxfdcp.c : 810
--------------------------------------------------------
10/19/2007 06:55:57 - Process(5104.28403) User(mqadmin_ad1)
Program(amqrmppa.exe)
AMQ6090: MQM could not display the text for error 536895769.
----- amqxfdcp.c : 766
--------------------------------------------------------
10/19/2007 06:55:57 - Process(5104.28403) User(mqadmin_ad1)
Program(amqrmppa.exe)
AMQ6090: MQM could not display the text for error 268460419.
And the FDC gets this over and over, getting HUGE until we kill the PID:
+-----------------------------------------------------------------------------+
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Fri October 19 06:49:54 Eastern Daylight Time 2007
|
| Host Name :- MyServernName(Windows 2000, Build 2195: Service
Pack 4)|
| PIDS :- 5724H7200 |
| LVLS :- 6.0.2.1 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- XY129005 |
| Application Name :- MQM |
| Component :- xstConnectExtent |
| SCCS Info :- lib/cs/pc/winnt/amqxsten.c, 1.45.1.5 |
| Line Number :- 1013 |
| Build Date :- Mar 26 2007 |
| CMVC level :- p600-201-070323 |
| Build Type :- IKAP - (Production) |
| UserID :- mqadmin_ad1 |
| Process Name :- E:\Programs\MQSeries\bin\amqrmppa.exe |
| Addressing mode :- 32-bit |
| Process :- 00005104 |
| Thread :- 00028192 |
| ConnId(4) App :- 0 |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6090: MQM could not display the text for error
|
| 536895769. |
| FDCSequenceNumber :- 34102 |
| Arith1 :- 536895769 20006119 |
| Arith2 :- 8 8 |
| Comment1 :- Failed to attach shared memory map:
Handle(0002BE28) |
| [rc=536895 |
| Comment2 :- Not enough storage is available to process this |
| command. |
| |
+-----------------------------------------------------------------------------+
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Tue May 13, 2008 5:47 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Does task manager or process explorer show any unusual or consistent pattern of any type of memory related resource when the amqrmppa.exe goes bad? Do you have a large number of channels start and stopping (over time) thru this process?
_________________
Glenn |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Tue May 13, 2008 8:06 pm Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
I guess, you will get rid of these errors once you upgrade to
6.0.2.3,
its pretty robust and stable, as what i feel
Regards
Gayathri
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Wed May 14, 2008 9:11 am Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
| I ran into the same problem (same version and platform) about a month ago. We encounter this problem about 3-4 times a year. When I sent the FDC report to IBM, they said it was a memory issue. even though we have a lot of RAM (and, yes, our monitoring software did not detect a memory issue). The only other twist is that we run IIS and we suspect it's related to this component, because if we restart it, the MQ problem goes away as well. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed May 14, 2008 11:22 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Gaya3 wrote: |
I guess, you will get rid of these errors once you upgrade to
6.0.2.3,
its pretty robust and stable, as what i feel
Regards
Gayathri |
Why upgrade? What is the issue in 6.0.2.1 that is specificaly fixed by 6.0.2.3? "Just upgrade" is a cop out unless the version you are on is VERY old or you can point to a specific APAR that is resolved in the new version that was buggy in the old. Otherwise you waste time and effort with the upgrade, which is not easy in a real environment with real customer with real change control when the time could be better spent trying to find root cause.
How often will you get lucky just upgrading and fixing a bug that was not known and somehow was magically fixed with a newer version?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| ThePostman |
| Posted: Tue Feb 17, 2009 7:51 am Post subject: |
|
|
Newbie
Joined: 21 Jan 2008
Posts: 6
|
Hi!
Did you find a solution to your issue?
We got exactly the same problem, but in our case it is at level of the MQCONN (see stack trace). Like the application is running for years without issue, we are not able to explain what the root cause is. The only change that has been done on the machine was changing his place into the blade enclosure (due or a defect hardware).
MQM Function Stack
ziiMQCONN
ziiConnectToAgent
zcpAttachPipe
xcsConnectSharedMemSet
xstAttachSharedMemSet
xstConnectExtent
xcsFFST
J-F |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Feb 17, 2009 11:21 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
My particular problem was caused by a memory leak in a very old version of a security exit. Every MQConn cauesed another 1K in memory to be allocated to the amqrmppa process till eventually.... 
I'd bet this isn't the only way to cause these symptoms, but that's what my root cause was.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
