| Author |
Message
|
| rangyaonmq |
 Posted: Mon Apr 06, 2009 2:08 am Post subject: MQ performance issue |
 |
|
Newbie
Joined: 06 Apr 2009
Posts: 6
|
We are using MQSeries on Mainframes.
MQ Series version is 5.3.
Here at the start of the business hour, we receive thousands of messages in a minute or less. To process the messages we have a CICS transaction which picks up the messages from one queue. From this queue it gets a order number and it looks for the data of this order in another program.
At the pick period we are facing performance problems, the messages are being processed very very slow, generally it takes arround 2-3 minutes to process but in pick period it takes 3-5 hours.
See below the process input output count has gone up.
TASK TASK S TERM PROGRAM RESOURCE RESOURCE USER STORAGE ---TOTALS---
# ID T ID NAME TYPE NAME ID USED CPU || I/O
12254 ORL2 S CSQCTRUE MQSeries TASKSWCH CICSP08 186720 6.134 20154.
Please suggest to improve the performance. |
|
| Back to top |
|
 |
| zpat |
| Posted: Mon Apr 06, 2009 2:24 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Are you using Trigger Every? If so then don't or at least TCLASS the CICS transaction so that a controlled number of them execute at the same time.
Determine the optimum number of CICS transactions to run at the same time by doing some performance testing with different TCLASS values.
Depending what is causing your delays, you might even want to use more than one CICS region.
Aside from not swamping the CICS region with trigger every, detailed mainframe application performance tuning is usually performed by specialists.
You may have these people on site or could request help from IBM or other vendors. The mainframe is the ultimate in a controllable, tunable and high-performance environment, so it will be possible to solve this. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 06, 2009 3:13 am Post subject: Re: MQ performance issue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rangyaonmq wrote: |
| At the pick period we are facing performance problems, the messages are being processed very very slow, generally it takes arround 2-3 minutes to process but in pick period it takes 3-5 hours. |
How have you determined the problem is within WMQ? Rather than in this other program where it gets the order data? Or possibly in a database being used somewhere in this design? A simple thing like an application not using a database index but doing a tablescan can have a drastic effect on response times.
I should point out that in 99% of cases I've seen where this happens, the application in question was supposed to be using the index, had been coded to use the index and an unfortunate sequence of events was preventing it.
As zpat correctly says, the mainframe has a wealth of tools to aid your tracing of the bottleneck, and many possible solutions.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Mon Apr 06, 2009 6:10 am Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
It sound as if you're using a single task to process the inbound messages. If my assumption is correct, then this identifies at least one bottleneck. Some M/F shops address this type of problem by forgoing triggering, and develop home-grown code which act as a service-oriented layer to poll the queue(s) - anyone remember Dennis Miller's SLI concept?. When a user-specified thresholds are reached, e.g. the number of messages exceed n, then additional tasks are started to service the queue.
Other areas to look at within CICS are DBMS performance, single-thread/serialization, and so forth. Outside of CICS, there are the usual suspects to examine, e.g. paging, I/O contention, etc. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Apr 06, 2009 6:23 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Yes, if using the IBM trigger monitor, you may want to use trigger(every) - as long as you also TCLASS the transactions to control the parallelism.
Otherwise you could re-write the transaction to use MQGET WAIT in a loop and keep it running all the time, then start manually more than one of these.
Your first post talks about the input count going up - which made me assume that the queue was being opened more than once and therefore the problem was too many, rather than too few, transactions running.
What sort of triggering is used? How many CICS transactions are started? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Apr 06, 2009 6:42 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'd suspect some CICS tuning is in order. Get your CICS sysprog to look at cics stats.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rangyaonmq |
| Posted: Mon Apr 06, 2009 11:50 pm Post subject: |
|
|
Newbie
Joined: 06 Apr 2009
Posts: 6
|
Thanks for the responses and valued comments..
Here the trigger Type is FIRST.
Only one transaction is started. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 06, 2009 11:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rangyaonmq wrote: |
| Only one transaction is started. |
Then it's a simple question of finding out where the time is being taken in that application, and why so much more is taken at peak times.
Straightforward tuning. No problem. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Apr 07, 2009 1:26 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Check that the transaction keeps the input queue open and loops around for multiple messages (until RC 2033 queue empty) before terminating.
What is the program logic inside this transaction?
Or try running several at once (trigger every and tclass limit of 10 ). |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Apr 08, 2009 4:47 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
rangyaonmq.
What have you discovered?
Moderator(s): please move this to the Mainframe and CICS topic.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
