| Author |
Message
|
| velocity |
 Posted: Wed Sep 29, 2010 7:40 am Post subject: Performance testing issue on z/OS |
 |
|
Centurion
Joined: 30 Nov 2007
Posts: 126
|
Hello,
We have two mainframe QMs communicating with request reply architecture. A request is initiated from QM A and hits QM B and is picked up by an IMS application and sends back a reply to QM A. Most of the replies come in time (sub seconds) but some are taking upto 2 minutes. Is there anyway we can point out where the issue is. I dont think this is an MQ issue but you never know. Both the QMs have V 7011 installed on them.
Please let me know your thoughts.
Thanks |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Wed Sep 29, 2010 9:06 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Your sysprog can do some basic p-d in real-time with Omegamon or some other performance tools. If this doesn't offer a clue, your sysprogs can turn on SMF and RMF data collection for MQ and IMS, and if necessary VTAM, to try to isolate the underlying problem.
The first place to start with any sudden behavior change is: What has changed? A new IMS or MQ app? Or a modified IMS or MQ app?
Moved to Mainframe and CICS forum.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| velocity |
| Posted: Wed Sep 29, 2010 9:19 am Post subject: |
|
|
Centurion
Joined: 30 Nov 2007
Posts: 126
|
| Thanks for the response. I will have IMS team setup traces also. This started happening in the performance testing environment from day 1. The only thing that has changed is the number of transactions since this is s stress test. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 29, 2010 9:36 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Sub-second to 2 minutes is alarming.
Is the test environment (LPAR) configured the same as PROD?
Is WLM alive and well? Is it giving the test environment sufficient resources?
What else is different?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| velocity |
| Posted: Wed Sep 29, 2010 9:41 am Post subject: |
|
|
Centurion
Joined: 30 Nov 2007
Posts: 126
|
| Quote: |
| Is the test environment (LPAR) configured the same as PROD? |
The only difference is QM A and QM B are in different clusters. So I had implemented a gateway QM for both applications to communicate.
| Quote: |
| Is WLM alive and well? Is it giving the test environment sufficient resources? |
What type of resources? Can you please explain.
Nothing else is different. |
|
| Back to top |
|
|
| velocity |
| Posted: Wed Sep 29, 2010 9:48 am Post subject: |
|
|
Centurion
Joined: 30 Nov 2007
Posts: 126
|
| One correction! I have setup overlapping clusters not a gateway QM. Sorry for the confusion. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 29, 2010 9:52 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Please refer my questions to your sysprog, who will understand LPAR, WLM and resource issues.
Basically, WLM allocates cpu time, i/o device time, real (central) storage, - all of the resources that help or hinder performance.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu Sep 30, 2010 3:52 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
If you can identify the request and reply messages by content, and if these messages are persistent, then you can find them in the log using CSQ1LOGP. You then can check when the request was put to QMA,. when it was transmitted by the channel, when it arrived in QMB, when it was get by IMS and the same for the way back. It is not very difficult if you know a bit about the logs.....
maybe this will not show where the time was wasted, but it may show that it was not wasted when the messages was in MQ transfer .... (e.g. if you find the reply was put 2 minutes after the request was got on QMB).
_________________
Regards, Butcher |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Sep 30, 2010 5:01 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| I have seen this where the app was incorrectly coded using syncpointed puts which were not committed immediately. Test each half in isolation using a message tool like RFHUTILC to see where the delay is coming from. |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu Sep 30, 2010 6:06 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
maybe not that easy if it only occurs occasionally .....
_________________
Regards, Butcher |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 30, 2010 6:24 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| The only thing that has changed is the number of transactions since this is s stress test. |
Are you using benchmark test data? Can you repeat the exact same test? If you run the same benchmark test data, is the dreadfully slow transaction-time result consistent? If so, it's most likely data-related - like a deadly embrace.
Anything interesting in the syslog from the IMS lock manager?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Sep 30, 2010 6:31 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| bruce2359 wrote: |
| Quote: |
| The only thing that has changed is the number of transactions since this is s stress test. |
Are you using benchmark test data? Can you repeat the exact same test? If you run the same benchmark test data, is the dreadfully slow transaction-time result consistent? If so, it's most likely data-related - like a deadly embrace.
Anything interesting in the syslog from the IMS lock manager? |
Also please check the qmgr's error logs and verify that the cluster channels don't have out of sequence numbers on it...(I've seen weird stuff happen that way).
You may also want to check the repository and the qmgr's QMID to make sure you have no cluster problem.
Check that none of the receiving queues/xmitqs is at or near the max msgs mark and finally check that none of the page sets is at a high water mark.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 30, 2010 6:48 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
...and display channel status on all the involved qmgrs to see if any of the channels are in RETRY in the 2-minute gap. Incorrectly set retry intervals might cause a channel-in-error to wait too long between short and long retries.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
