| Author |
Message
|
| vanushreevyas |
 Posted: Wed Sep 12, 2012 2:09 am Post subject: MQ v7.0.1.8 Linux x86_64 - FDC generated - Probe ID ZS402020 |
 |
|
Novice
Joined: 28 Nov 2011
Posts: 20
|
Hi,
We have MQ v7.0.1.8 installed on a Linux x86_64 - RHCS clustered environment. There are 8 queue managers in the cluster.
FDCs are being randomly generated for any of the 8 queue managers as below with Probe ID ZS402020.
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| LVLS :- 7.0.1.8 |
| Product Long Name :- WebSphere MQ for Linux (x86 platform) |
| Vendor :- IBM |
| Probe Id :- ZS402020 |
| Application Name :- MQM |
| Component :- zstAsyncConsume |
| SCCS Info :- lib/zst/amqzstd0.c, 1.199.1.9 |
| Line Number :- 1406 |
| Build Date :- Feb 24 2012 |
| CMVC level :- p701-108-120224 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 1000 (mqm) |
| Real UserID :- 1000 (mqm) |
| Program Name :- amqrmppa |
| Addressing mode :- 32-bit |
| Process :- 9268 |
| Process(Thread) :- 9564 |
| Thread :- 34 |
| ThreadingModel :- PosixThreads |
| QueueManager :- QMGR |
| UserApp :- FALSE |
| ConnId(1) IPCC :- 1301 |
| Last HQC :- 1.0.0-2456000 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- krcE_UNEXPECTED_ERROR |
| Minor Errorcode :- OK |
| Probe Type :- INCORROUT |
| Probe Severity :- 2 |
| Probe Description :- AMQ6125: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
zstAsyncConsumeThread
zstAsyncConsume
xcsFFST
MQM Trace History
.
.
.
--} ziiSendReceiveAgent rc=lpiRC_WAITCANCELLED_ASYNCCONSUME
.
.
.
} zstMQGETM rc=lpiRC_WAITCANCELLED_ASYNCCONSUME
.
.
.
--} ziiSendReceiveAgent rc=MQRC_SYNCPOINT_NOT_AVAILABLE
.
.
.
} zstMQGETM rc=MQRC_SYNCPOINT_NOT_AVAILABLE
---------------------------------------------------
Also these FDCs are being generated randomly and not when a queues is GET_INHIBITED.
We have both ETC and normal client connected to the server and FDCs are being generated for ETC clients as well.
Any help or pointers would be appreciated.
Regards,
Vanu |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Sep 12, 2012 5:36 am Post subject: Re: MQ v7.0.1.8 Linux x86_64 - FDC generated - Probe ID ZS40 |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vanushreevyas wrote: |
| We have both ETC and normal client connected to the server and FDCs are being generated for ETC clients as well. |
Given the first hit for that probe id is a fix for XA handling the failure of ETC clients is hardly surprising.
What do you use as an XA resource controller?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Andyh |
| Posted: Wed Sep 12, 2012 1:31 pm Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 237
|
This looks like a possible match for an MQ V7 problem that IBM recently came across during internal testing. A fix has been developed and is scheduled to be included in future fix packs.
I'd suggest you contact MQ support. If you reference defect 14051 it may help expedite resolution. |
|
| Back to top |
|
|
| vanushreevyas |
| Posted: Thu Sep 13, 2012 4:23 am Post subject: |
|
|
Novice
Joined: 28 Nov 2011
Posts: 20
|
Hi Andyh,
Thanks for the pointer.Could you please provide me some links for this as am unable to locate any.
Thanks and Regards,
Vanu |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 13, 2012 5:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vanushreevyas wrote: |
| Could you please provide me some links for this as am unable to locate any. |
You couldn't locate http://www-947.ibm.com/support/entry/portal/?? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| vanushreevyas |
| Posted: Wed Sep 19, 2012 12:56 am Post subject: |
|
|
Novice
Joined: 28 Nov 2011
Posts: 20
|
Hi Vitor,
What i meant was I couldnt find any reference to related defect 14051  .
Thanks and Regards,
Vanu |
|
| Back to top |
|
|
| vanushreevyas |
| Posted: Wed Sep 19, 2012 2:15 am Post subject: |
|
|
Novice
Joined: 28 Nov 2011
Posts: 20
|
Hi Vitor,
Apologies am not aware of XA feature much.
We have not configured any XA resource manager on queue managers if that is what you are looking for.
If anything is configured or can be configured on application side then am not aware as we do not have much visibility to app servers.
However if it can also be configured on app servers, then can ask them relevant questions.
Also we have noted that the amqrrmfa process on the servers starts consuming high CPU utilisation. This increases to the extent that we have to go for a queue manager bounce almost everyday.
Is there any limit to clusters that a queue manager can be part of ? The 8 queue managers here are part of clusters ranging from 230 to 410 clusters. Not sure if that kind of load is causing the repos process problems.
We have raised a PMR with IBM but have not yet got any fruitful response as yet.
Thanks and Regards,
Vanu |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 4:06 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
| vanushreevyas wrote: |
Is there any limit to clusters that a queue manager can be part of ? The 8 queue managers here are part of clusters ranging from 230 to 410 clusters. Not sure if that kind of load is causing the repos process problems.
|
I'm confused...
When you say that there are 8 qmgrs that are members of 230 to 410 clusters, do you mean 230-410 cluster queue instances across 8 qmgrs?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| vanushreevyas |
| Posted: Wed Sep 19, 2012 4:57 am Post subject: |
|
|
Novice
Joined: 28 Nov 2011
Posts: 20
|
| Each of these 8 queue managers is part of different and multiple MQ clusters. 1 QM is part of 230 MQ clusters in total - either as FR or PR. 1 other QM is part of 410 MQ clusters - either as FR or PR. Rest of the queue managers lie somewhere in between. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Sep 19, 2012 4:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vanushreevyas wrote: |
| What i meant was I couldnt find any reference to related defect 14051 |
You won't - that's an internal IBM defect number.
What @Andyh meant was that you should raise a PMR which references that defect number in the text of the PMR. If there is a fix (as he indicates) then support may be able to provide it to you as an APAR prior to it's inclusion in a future fix pack, or provide you other remediation strategies.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Sep 19, 2012 5:01 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| vanushreevyas wrote: |
| Each of these 8 queue managers is part of different and multiple MQ clusters. 1 QM is part of 230 MQ clusters in total - either as FR or PR. 1 other QM is part of 410 MQ clusters - either as FR or PR. Rest of the queue managers lie somewhere in between. |
I'm wondering if you're working at a site where they held a recent Olympics/Paralympics, a site that thought (at the time, and quite possibly still do) that they could isolate different applications by cluster?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Sep 19, 2012 5:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vanushreevyas wrote: |
| Is there any limit to clusters that a queue manager can be part of ? The 8 queue managers here are part of clusters ranging from 230 to 410 clusters. Not sure if that kind of load is causing the repos process problems. |
Well you've going to get a lot of load trying to sync all those clusters.
As I said in another thread (!)
- why so many clusters?
- why so many clusters for so few queue managers
- what requirement does this meet?
- what idiot thought this was a good idea?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 5:10 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
| vanushreevyas wrote: |
Is there any limit to clusters that a queue manager can be part of ? |
Hmmmm. Is a POS (point-of-sale) application, where each cluster is a store? Just guessing.
I'd suspect that there is an architectural limit - probably 32k or something like that.
But there would also be a practical limit. Each cluster imposes cluster administrative message flows between and among the FRs and PRs; so I'd expect to see elevated processor and channel activity with 400+ clusters.
I'm not curious enough to model this configuration, though.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| vanushreevyas |
| Posted: Wed Sep 19, 2012 5:16 am Post subject: |
|
|
Novice
Joined: 28 Nov 2011
Posts: 20
|
Hi Vitor,
To answer your queries :
1) why so many clusters? - we have more than 800+ MQ servers on our estate with one or multiple queue managers on them. The queue managers on this server have MQ clusters with quite a few of the other MQ queue managers on rest of the estate. Hence so many clusters.
2) why so many clusters? - Cause of the number of queue managers on the estate. and number of MQ interfaces accross entire MQ estate.
3) what requirement does this meet? - We had more than 1000 MQ servers with each application having their own server and queue manager. We have tried to reduce the number of MQ servers by sharing 1 server for a few applications. This reduces the need for applications to have MQ server installed as they can use MQ client.
4) what idiot thought this was a good idea? - am myself looking for that idiot 
Thanks and Regards,
Vanu |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Sep 19, 2012 5:22 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| vanushreevyas wrote: |
| 2) why so many clusters? - Cause of the number of queue managers on the estate. and number of MQ interfaces accross entire MQ estate. |
The number of queue managers should not be directly related to the number of clusters - unless they're pub-sub clusters which imposes its own limit, i.e. 100 per cluster (if memory serves).
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
|
|
