| Author |
Message
|
| rahuldhanpal |
 Posted: Fri Aug 20, 2010 6:48 am Post subject: FFST messages |
 |
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
hi experts,
i see the following messages in the message broker log for the CMGR qmgr continuously
Aug 20 10:46:38 MTDAPPLX01 MQSeries: FFST (26,2,10,536895768) failed: /var/mqm/errors/AMQ15438.0.FDC (errno=0)
Aug 20 10:46:38 MTDAPPLX01 MQSeries: FFST record created in /var/mqm/errors/AMQ15438.0.FDC
Aug 20 10:47:38 MTDAPPLX01 MQSeries: FFST (24,129,11,545284631) failed: /var/mqm/errors/AMQ15438.0.FDC (errno=0)
Aug 20 10:47:38 MTDAPPLX01 MQSeries: FFST record created in /var/mqm/errors/AMQ15438.0.FDC
Aug 20 10:47:38 MTDAPPLX01 MQSeries: FFST (26,2,10,536895768) failed: /var/mqm/errors/AMQ15438.0.FDC (errno=0)
Aug 20 10:47:38 MTDAPPLX01 MQSeries: FFST record created in /var/mqm/errors/AMQ15438.0.FDC
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Wednesday August 18 12:55:56 EDT 2010 |
| Host Name :- MTDAPPLX01.OWS.TEST.LAB (Linux 2.6.18-164.2.1.el5) |
| PIDS :- 5724H7210 |
| LVLS :- 6.0.2.9 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| Probe Id :- XY129011 |
| Application Name :- MQM |
| Component :- xstConnectExtent |
| SCCS Info :- lib/cs/unix/amqxstex.c, 1.83.1.12 |
| Line Number :- 1933 |
| Build Date :- Mar 16 2010 |
| CMVC level :- p600-209-100316 |
| Build Type :- IKAP - (Production) |
| UserID :- 00000550 (wbrkadm) |
| Program Name :- kmqlocal |
| Addressing mode :- 32-bit |
| Process :- 15438 |
| Thread-Process :- 15438 |
| Thread :- 3 |
| ThreadingModel :- PosixThreads |
| QueueManager :- CMGRDEV_QM |
| ConnId(4) App :- 0 |
| Last HQC :- 1.0.0-31552 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- xecX_E_RESOURCE_ERROR |
| Minor Errorcode :- OK |
| Probe Type :- INCORROUT |
| Probe Severity :- 2 |
| Probe Description :- AMQ6125: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| Arith1 :- 2195465 218009 |
| Arith2 :- 12 c |
| Comment1 :- Address space cannot accommodate this request: |
| shmat(ShmId 0x00218009) [rc=-1 errno=12] Cannot allocate memory |
| Comment2 :- Cannot allocate memory |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
MQCONN
can anybody share some pointers on this issue...
thanks
RD |
|
| Back to top |
|
 |
| Gaya3 |
| Posted: Fri Aug 20, 2010 6:54 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 20, 2010 6:55 am Post subject: Re: FFST messages |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Aside from the obvious comment about these files being produced when there's an internal error within the software, and the many discussions on details diagnosis based on their information, I think this:
| rahuldhanpal wrote: |
| Comment1 :- Address space cannot accommodate this request: |
| shmat(ShmId 0x00218009) [rc=-1 errno=12] Cannot allocate memory |
| Comment2 :- Cannot allocate memory |
|
may be at the heart of the issue.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Fri Aug 20, 2010 7:14 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
thanks for the answers guys..
but everything was working fine till yesterday ..i am not sure why this address allocation problem is coming up ..we are running RHEL 64-bit OS and MQ is 64-bit and java is 64-bit ..
i am looking into the url provided by gaya3...
thanks
RD |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 20, 2010 7:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rahuldhanpal wrote: |
| but everything was working fine till yesterday |
So what did the sys admins change on the box yesterday?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjava |
| Posted: Fri Aug 20, 2010 7:53 am Post subject: |
|
|
Voyager
Joined: 25 May 2009
Posts: 80
Location: New Jersey
|
What OS, what version of MQ and MB?
Did u do the IBM recommened Kernel configuration prior to installing?
Thanks |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Fri Aug 20, 2010 8:06 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
hi guys,
seems like i found something .. we have Omegamon agent monitoring the qmgrs...some how something got wierd and maybe the agent is not able to report the data of qmgr as it was not finding its related temporary queues on the qmgr for collecting the information. i assume that might be one reason it was creating FDC and updating it every minute for allocating/creating the resource on the qmgr..
when you see my first post it says the program name: kmqlocal (which is the agent process which monitors the qmgr and collects info)
one thing i am trying to understand is everything was running good until this FDC was generated first at AUG 18th ..and surely this has to do something with the 32-bit java allocations. i will have to troubleshoot and identify what has changed at that time and see if this is the potential cause of this problem..
i will still analyse the problem and update the thread..
And currently i dont see the FDC file getting updated every minute..
thanks
RD |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 20, 2010 8:19 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
one of the mentioned resolutions in that technote was "restart the program periodically".
When was the last time you restarted kqlocal before the first instance of the error? if the answer is "i don't know" or "never", then.... maybe you should institute a normal procedure to restart things every couple of months or so. |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Fri Aug 20, 2010 9:46 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
we restart the kqlocal process reguarly but there is no certain timely procedure to do this.. we do it when some system upgrades are scheduled or for any critical system restarts... but from now on i will make sure we restart the required processes in periodical instances...
thanks for all your vaulable comments...i will update the thread if i find any additional information.. |
|
| Back to top |
|
|
|
|
