| Author |
Message
|
| jsp_ejb |
 Posted: Thu Oct 29, 2009 2:56 pm Post subject: One transaction slows down the other |
 |
|
Novice
Joined: 27 Jun 2004
Posts: 21
Location: Chicago
|
Our environment has MQ V6 with CICS 3.2 on Mainframe zos 1.7.
We have two different CICS transactions A and B calling the same program C. Program C uses MQ to put a message as request and wait for its response matching by CorrelationID. If Transation A runs along, it runs at expected pace. But, if B runs at the same time, A slows down dramatically and B runs OK.
Transaction A is long running (single instance). It reads from another Queue (queue is different from program C queues) and processes about 500 messages each run.
Transaction B is t-classed with 4 instances. It only process one message at a time.(Difference messages from Transaction A).
There were plenty of system resources when both of them start. If program C is not called by Transaction A (yes, we can disable it on the fly), everything is OK.
What are possible causes of this kind of interference?
Thanks for sharing your advice. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Oct 29, 2009 3:25 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
What are the cics programs doing besides get/put to queues? Is there a back-end database involved in any of the programs?
Are the programs accessing other mvs resources (sequential files, pds's)?
Is the cics region at or near maxtask or amaxtask?
Anything of note in the syslog for the cics in question?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jsp_ejb |
| Posted: Thu Oct 29, 2009 6:46 pm Post subject: |
|
|
Novice
Joined: 27 Jun 2004
Posts: 21
Location: Chicago
|
Yes, indeed, there are DB2 access involved. Program C does not do any database calls, but, both A and B perform DB2 SQLs. There is no other mvs resources (sequential files, pds's) involved.
Even with both transactions running, cics region is not even close to any maxtask or amaxtask setting. And there is nothing in the cics syslog in question either. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 29, 2009 8:23 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Just for my peace of mind could you please confirm the following:
- Each transaction instance (B or A) creates its own connection handle to the qmgr and does not share it with any other instance
- Each transaction operates in a UOW and uses syncpoint
- Transaction B (fast running) does not access the same rows in the DB as transaction A
- The DB used (DB2?) uses record locking level and not page locking level
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 29, 2009 9:04 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
MQCONN calls are ignored for cics. Connections are performed by the cics-mq adapter; and the adapter flows the mq calls from cics to the qmgr.
How many CKTI's are executing in the cics region?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jsp_ejb |
| Posted: Fri Oct 30, 2009 7:11 am Post subject: |
|
|
Novice
Joined: 27 Jun 2004
Posts: 21
Location: Chicago
|
Thanks for all of your considerations.
Transaction A and B do not share any handle to any queues. A and B run their own UOW. None of A and B is triggered by MQ message. A wakes up in a fixed time interval and loop for 500 times. Each time it gets a message with syncpoint. B is called by other systems.
Program C does both put-message and get message to two different queues, both with NO syncpoint.
B and A might access the same row in DB2, but, we monitored DB and no issue has been spotted. Reports claim there are long waiting time in MQGET within transaction A.
Only one CKTI is running in the region. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Oct 30, 2009 7:16 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Reports claim there are long waiting time in MQGET within transaction A. |
I'm not clear on what this means.
Are you saying that the app does a get with wait, with waitinterval set to a long period of time? Like 5 minutes? or 5 hours?
Anything in the cics log regarding ckti or the offending applications?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Oct 30, 2009 8:10 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
One CKTI can manage a maximum of 8 concurrent flows. Are you running other cics-mq transactions - other than the ones in this post? How many concurrent cicq-mq transactions are you running?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jsp_ejb |
| Posted: Fri Oct 30, 2009 8:49 am Post subject: |
|
|
Novice
Joined: 27 Jun 2004
Posts: 21
Location: Chicago
|
We do have some other CICS-MQ transactions. But, none of A and B is triggered by message in a queue. Does CKTI still matter?
Program C does MQGet with wait time of 30 second. But, this is not counted in the report. The IBM APA report indicates longer waiting time for MQGET within transaction A.
A couple of observations
1. The queue that tran A gets from has trigger on without any trigger data, init queue or process defined.
2. CHIDISPS 5 and CHIADAPS 8.
If it is the channel issue, why would it not slow down transaction B? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Oct 30, 2009 9:02 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| We have two different CICS transactions A and B calling the same program C. Program C uses MQ to put a message as request and wait for its response matching by CorrelationID |
.
| Quote: |
| 1. The queue that tran A gets from has trigger on without any trigger data, init queue or process defined. |
Pardon my confusion...
Program C waits for the response from what program? If the queue that program C puts a msg to is not triggered, then the consuming application is not starting when a message arrives. Is the consuming app CICS-based? Or is it an MVS app?
As to the channel question... does all this activity take place on the same qmgr and same cics? Is all of this in the same LPAR? Does the request message and/or reply message flow across a network - to an AIX or Windows qmgr, for example?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jsp_ejb |
| Posted: Fri Oct 30, 2009 10:09 am Post subject: |
|
|
Novice
Joined: 27 Jun 2004
Posts: 21
Location: Chicago
|
The consuming app is j2ee based and hosted in AIX. The queue that Program C puts a message into is a remote queue.
All activities take place on the same qmgr and same cics in z. And AIX has its own qmgr. So, obviously, request/reply messages flow through mq channels across a network (LAN).
If the channel is the problem, why would it only impact transaction A? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Oct 30, 2009 10:20 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
The consuming app is j2ee based and hosted in AIX. The queue that Program C puts a message into is a remote queue.
All activities take place on the same qmgr and same cics in z. And AIX has its own qmgr. So, obviously, request/reply messages flow through mq channels across a network (LAN). |
These two statements are contradictory. This application (request through reply) spans two qmgrs across a network.
When next you post this, please explain from beginning to end how the entire transaction gets processed. Like this:
1. cics prog C opens a remote queue def, then
2. puts a message to an xmit queue, and waits for a reply
3. the xmit queue is triggered on first;
4. the message flows to an AIX qmgr across point-to-point channels (sender-receiver)
5. the destination queue on AIX is triggered (or not)
6. the consuming application does database work (or not)
7. whatever...
| Quote: |
| If the channel is the problem, why would it only impact transaction A? |
There are two channels - one for the request message to flow on, the other for the reply to flow on. AIX and z/ are vastly different in horsepower and other resources.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jsp_ejb |
| Posted: Fri Oct 30, 2009 12:18 pm Post subject: |
|
|
Novice
Joined: 27 Jun 2004
Posts: 21
Location: Chicago
|
What I said was that, in Z, we only use one qmgr and one CICS in these transactions. And there is one qmgr in AIX also.
Transaction A:
Loop 500 times, each loop --
0. Get a Message from q0 with syncpoint, call program C
1. cics prog C opens a remote queue def if it is not already opened, then
2. puts a message to an xmit queue, and waits for a reply (30 sec)
3. the xmit queue is triggered on first;
4. the message flows to an AIX qmgr across point-to-point channels (sender-receiver)
5. the destination queue on AIX is triggered by Websphere Message Driven Bean.
6. the consuming application does buessine rules
7. the consuming application put the reply on a remote queue
8. response message flows to z
9. Program C gets the reply message
10.and returns to Transaction A
11. Transaction A does other DB2 calls, commit.
Transaction B: get called by an external system through cics transaction gateway (IBM CICS CTG) and perform same steps as transaction A 1-9. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Oct 30, 2009 1:32 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
2. puts a message to an xmit queue, and waits for a reply (30 sec)
...
7. the consuming application put the reply on a remote queue |
The replying app doesn't send the reply msg to a reply-to-queue specified in the MQMD by the requesting app?
At the time of the put (2), is the channel in RUNNING state on both ends?
Is the AIX box under stress (not enough resources)?
What version/release of mq on z?
What version/release of mq on AIX?
Can you can turn on monitoring at the channel ends? (perhaps to eliminate the channels as the source of your odd behavior)
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jsp_ejb |
| Posted: Fri Oct 30, 2009 3:39 pm Post subject: |
|
|
Novice
Joined: 27 Jun 2004
Posts: 21
Location: Chicago
|
We did not code to use reply-to-q. Channels are running on both ends when the slowdown happens. AIX has plenty of resources all the time. We have trace within the j2ee app. It has no issue at any time.
The version is V6 on both z and aix systems. |
|
| Back to top |
|
|
|
|
