| Author |
Message
|
| askthiru |
 Posted: Tue Mar 11, 2008 2:47 pm Post subject: MQ series issue with CICS |
 |
|
Newbie
Joined: 11 Mar 2008
Posts: 7
|
We have a strange problem happening in CICS.
We have two different applications in one system. A MQ message gets written to two different queues by an external system. The two queues are defined in the same queue manager. The two queues kick-off two separate transactions (programs) in the CICS region. The problem is happening first time in the morning when CICS is recycled.
We have displays throughout the program and what's happening is that Program A is reading the queue message belonging to Program B. Program B is trying to read its queue and gives a 2033 because there is no more message in the queue. Both the programs get initiated at the same time and they establish the connection at the same time but for some reason, when reading the queue, Program A is reading the other queue. We have checked the queue names and it is properly hard-coded. The confusing part is that after a few tries (usually 2 or 3), this whole interface works fine i.e., the program reads the proper queue and does what it needs to do. First time thru, we get this problem. The only thing I could see is that the external system that sends these messages use the same message id in both the messages. Eventhough our Program does not use the message-id to read the queue, is it possible that the program gets confused when there are two queues in which there is a message with the same message id. The other option we are planning is to induce a delay in between writing the messages to the two queues.
Any suggestions or past experiences like this? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Tue Mar 11, 2008 3:09 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| A MQ message gets written to two different queues by an external system. |
| Quote: |
| Program A is reading the queue message belonging to Program B |
Is it the same exact identical message that gets written (MQPUT) to two different queues? If they are identical, how does one of them belong to Program A. What creates this affinity?
I'm guessing that you mean two different messages are MQPUT to two different queues...
As a problem determination step, I'd suggest browsing the messages in the two queues BEFORE you start CICS to validate that the A message is really in queue A; and that the B message is in queue B.
| Quote: |
| Both the programs get initiated at the same time and they establish the connection at the same time |
The CICS adapter does the MQCONNect to the queue manager, not the application program.
You have my curiousity going...
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| askthiru |
| Posted: Tue Mar 11, 2008 3:22 pm Post subject: |
|
|
Newbie
Joined: 11 Mar 2008
Posts: 7
|
Bruce,
The messages in the two queues are different. Thats how we know that Program A is reading the other queue. The external system does two separate MQPUTs to the two queues but the MQPUT occurs one after the other immediately (no complexing processing inbetween the two MQPUTs).
When Program A reads program B's queue, we do a display of the working storage and it clearly shows program B's queue record. When I go in the MQ panel and read Program A's original queue, the message is still there and has the proper message. i dont understand what makes Program A go after the other queue.. Puzzling factor is that, why this works fine after a couple of tries... |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Mar 11, 2008 3:31 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| i dont understand what makes Program A go after the other queue |
You've looked at the source code for both programs? The programmer fills in the MQOD (object descriptor) with the name of the queue to MQOPEN.
For program A, what is this queue name?
For program B, what is this queue name?
Display the object definitions for both these queues. Cut and paste into a reply post.
Are these programs C or COBOL or what?
Cut and paste the source code that creates the MQOD into a reply post.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Tue Mar 11, 2008 11:17 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
for me it is very suspicious that this only occurs on cics start.
At startup cics connects to mqseries, the cics trigger monitor opens the initiation queue, and - if queue A and queue B has a message each - both queues will create a trigger at almost the same time starting the proper transactions.
And now program A reads queue of Program B? Sounds like a programming issue (some kind of storage overlay / mix? Bad compile options?
Best way for triggering is to get the queue name out of the MQTM block you get passed by the CICS trigger monitor, and not to use something hardcoded (see application programmers reference)
_________________
Regards, Butcher |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Wed Mar 12, 2008 9:15 am Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
| Sounds like a coding issue, e.g. quasi-reentrancy not being observed. |
|
| Back to top |
|
|
| askthiru |
| Posted: Wed Mar 12, 2008 6:30 pm Post subject: |
|
|
Newbie
Joined: 11 Mar 2008
Posts: 7
|
Thanks for the replies..
We included a delay (a do-while loop for about 10000) times in the external system inbetween the two MQPUTs and that worked... I know its ugly but it worked for now.. |
|
| Back to top |
|
|
| askthiru |
| Posted: Wed Mar 12, 2008 6:36 pm Post subject: |
|
|
Newbie
Joined: 11 Mar 2008
Posts: 7
|
| bruce2359 wrote: |
| Quote: |
| i dont understand what makes Program A go after the other queue |
You've looked at the source code for both programs? The programmer fills in the MQOD (object descriptor) with the name of the queue to MQOPEN.
For program A, what is this queue name?
For program B, what is this queue name?
Display the object definitions for both these queues. Cut and paste into a reply post.
Are these programs C or COBOL or what?
Cut and paste the source code that creates the MQOD into a reply post. |
The two programs are clones of each other except for the queue names and their own processing.. The MQ call logic are done in a common program that both these programs link to. The MQOD-OBJECT-NAME is being moved from a work record which is being populated by these two programs via hard-coding... We made sure that all these are right.. Also the fact that this works after a few tries negates our theory that queue definitions are wrong...
What we did find based on our display statements was that both the programs tried to do the MQCONN at the same time and both attempted the read message queue at the same time.. This is clearly visible from the display statements in the common module making the MQ calls. When the message is read and when we display the message, we see that Program A got the message from the queue meant for the other program..
As rtsujimoto mentioned, it looks to me like a quasi-reentrancy problem or a initialization problem in CICS first time through.. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 12, 2008 8:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
It may just be a concurrency problem and sharing memory area where you should really initialize your own.
So if the 2 programs share memory and do not initialize their own, if the dispatcher program sets QA in the shared memory and an other instance sets QB in the shared memory the programs might just pick up the latest value QB...
You need to make sure that Q information gets moved from shared memory to program memory as fast as possible and only used from program memory. It should really never be shared across programs (A & B) and is really to be only shared between the trigger monitor and its invoked program.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Mar 13, 2008 6:43 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| The MQOD-OBJECT-NAME is being moved from a work record which is being populated by these two programs via hard-coding... |
Work record? If both programs start, do they both see the same work record? Or does an instance of a work record have affinity to one program?
| Quote: |
| The MQ call logic are done in a common program that both these programs link to. |
Is the called program linked to with EXEC CICS LINK? Or is it actually linkedited into the calling programs?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| askthiru |
| Posted: Thu Mar 13, 2008 3:16 pm Post subject: |
|
|
Newbie
Joined: 11 Mar 2008
Posts: 7
|
| Ok.. The thing here is that we are converting our IDMS/ADSO environment into DB2/CICS... In the IDMS/ADSO environment this works fine. Its the DB2/CICS version that doesnt work first time thru.. Also, the code is not written in pure Db2/CICS... The IDMS/ADSO statements were converted by a third party company into CICS statements.. So technically there are no direct EXEC CICS LINK statements in the program eventhough it is probably there in the black-box that converts the code into CICS... |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Mar 13, 2008 5:57 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
(I'm smacking my forehead with the palm of my hand) Why didn't I ask you if this has ever run successfully before! Oy.
| Quote: |
| The IDMS/ADSO statements were converted by a third party company into CICS statements.. So technically there are no direct EXEC CICS LINK statements in the program eventhough it is probably there in the black-box that converts the code into CICS... |
You say a 3rd party converted IDMS to CICS statements. Then there are CICS statements, yes? Can you look at these statements? Do you have access to source code? Does this vendor guarantee their work? Do they have any technical support for this product?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| askthiru |
| Posted: Sat Mar 15, 2008 6:13 am Post subject: |
|
|
Newbie
Joined: 11 Mar 2008
Posts: 7
|
| bruce2359 wrote: |
(I'm smacking my forehead with the palm of my hand) Why didn't I ask you if this has ever run successfully before! Oy.
| Quote: |
| The IDMS/ADSO statements were converted by a third party company into CICS statements.. So technically there are no direct EXEC CICS LINK statements in the program eventhough it is probably there in the black-box that converts the code into CICS... |
You say a 3rd party converted IDMS to CICS statements. Then there are CICS statements, yes? Can you look at these statements? Do you have access to source code? Does this vendor guarantee their work? Do they have any technical support for this product? |
Bruce,
Ofcourse the third party vendor is assisting us in finding the problem.. We have access to the converted code, but in the converted code they make calls to their 'I/O modules' which we do not have access to.. For example, for a simple Link statement, they have various move statements followed by a call to their i/o module..Its not the most efficient way to convery but the decision to do this was not made by us.
Our biggest question is whether this is a mq problem or a CICS problem or their conversion problem.. If its a problem with the source code or queue definitions , this should have never worked, but since the whole interface works fine after 2-3 tries, we are puzzled.. In the idms/adso world this doesnt happen but idms definitions for MQ are different because there is an aquasoft product sitting between idms and MQ which is not needed in CICS... |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Mar 15, 2008 9:54 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Our biggest question is whether this is a mq problem or a CICS problem or their conversion problem.. |
I am sympathetic with your plight. Other than an odd behavior, at this point, there is little/no actual evidence to point to a specific problem or problems.
Some givens: MQ works, MQ object definitions don't usually cause the symptoms you are experiencing, CICS works, there are lots of applications in existence that successfully interface between MQ and CICS.
Add me to the list of those with CICS and MQ experience who suspect that the vendor-supplied code is misbehaving. I'd actually place a money bet on this.
Please update this post when you learn more.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| askthiru |
| Posted: Sun Mar 16, 2008 7:55 am Post subject: |
|
|
Newbie
Joined: 11 Mar 2008
Posts: 7
|
| Thanks bruce.. and thanks everyone with your suggestions and tips.. I will keep the thread updated if we learn more on this issue.. |
|
| Back to top |
|
|
|
|
