| Author |
Message
|
| HenriqueS |
 Posted: Wed Feb 13, 2008 7:22 am Post subject: Urgent - uncommited messages under CICS |
 |
|
Master
Joined: 22 Sep 2006
Posts: 235
|
Our production system is not delivering the messages to our application.
There is a trigger that goest into a local queue to read every new incoming message. A new CICS transaction is forked to GET it, but it stalls, MQ says that the message is uncommited and our app does not receive it.
After, some sort of timeout occurs. The message disappears from the queue (at least is not browsable and the the queue depth lowers!) and is not delivered to the application!
Other new CICS transactions are forked to receive the later messages but the same thing happens.
What can I do? 
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Feb 13, 2008 7:23 am Post subject: Re: Urgent - uncommited messages under CICS |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| HenriqueS wrote: |
| What can I do? |
Work out why the sending application isn't commiting the message?
If I had to guess, the sending application is trying to do the put as part of an XA transaction & something's preventing that from completing (database deadlock perhaps). The transaction timeout expires, the MQ message is rolled back as intended and your CICS transaction trying to do the get is left high & dry.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Wed Feb 13, 2008 9:11 am Post subject: Re: Urgent - uncommited messages under CICS |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
Sending app? The sending app is in a remote server. I did not know that messages flowing from a remote server must be commited there...really?
| Vitor wrote: |
| HenriqueS wrote: |
| What can I do? |
Work out why the sending application isn't commiting the message?
If I had to guess, the sending application is trying to do the put as part of an XA transaction & something's preventing that from completing (database deadlock perhaps). The transaction timeout expires, the MQ message is rolled back as intended and your CICS transaction trying to do the get is left high & dry. |
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Feb 13, 2008 10:05 am Post subject: Re: Urgent - uncommited messages under CICS |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
| HenriqueS wrote: |
...MQ says that the message is uncommited and our app does not receive it.
|
I don't understand that. What do you mean MQ "says" its uncommited?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Wed Feb 13, 2008 12:35 pm Post subject: Re: Urgent - uncommited messages under CICS |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
It is not showing now, but the "Uncommitted" column kept showing "Y". Watching the MQ-CICS interface transaction (CKCQ) it seemed that the consuming transactions were spending too much time on the GET operation.
Worst, CICS was giving up the transaction without rolling back the GET. Messages were going to nowhere and disappearing from the queue they were on...
No meaninful messages on the *MSTR sysouts were given nor in the CICS sysouts.
We stopped the trigger, waited for half hour and reestarted, and it went ok.
We suspected that our consuming app was entering in some loop or condition it coldn´t handle, but then we get to this:
a) With the timeout, why the GET operation was not rolled back?
b) This app runs continously for the last 5 years and never such problem happened.
c) All message parsing functions in the app return status code into a sysout. This sysout was not even touched.
d) The same application (under the same CICS region) is used to receive data from several other partners (but they do leave their messages in other queues), the issue only happened with one partner, while others, with with 10 times more message delivery throughput, did not show anything wrong.
| Code: |
LIST LOCAL QUEUES - STATUS - BCMP ROW 1 OF 2
TYPE ACTION CODES, THEN PRESS ENTER. PRESS F11 TO DISPLAY ALL OPEN QUEUES.
1=DISPLAY APPLICATIONS 3=ALTER 4=MANAGE
QUEUE NAME DISPOSITION DEPTH
LAST PUT TIME LAST GET TIME TIME ON QUEUE MAX AGE
OUTPUT USE INPUT USE UNCOMMITTED MSGS
<> QL.REQ.78* PRIVATE BCMP
QL.REQ.78626983.00038166.01 QMGR BCMP 0
2008-02-13 17.32.05 2008-02-13 17.32.05 646630 6281012
******** End of list ********
|
| PeterPotkay wrote: |
| HenriqueS wrote: |
...MQ says that the message is uncommited and our app does not receive it.
|
I don't understand that. What do you mean MQ "says" its uncommited? |
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
| MucheIsMyHero |
| Posted: Wed Feb 13, 2008 2:58 pm Post subject: |
|
|
Novice
Joined: 29 Jun 2005
Posts: 15
|
There's nothing inherently wrong with the screen showing uncommitted messages while processing is going on. When you say CICS was giving up the transaction, do you mean that the transaction would abend or just end appropriately with the data not being processed? Did you run it through CEDX? Did the message have an expiry value? What's the poison message logic look like on the getter, if any? Does your putter have any logging that indicates successful completion or an issue?
Don't want to get you chasing your tail, these were just the questions that immediately popped into my head. MQ subsystem's not the one losing messages here, I'd bet. |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Thu Feb 14, 2008 9:25 am Post subject: |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
Trying to answer:
1) The transaction just ended but produced no results on the target database. No error on the database was found. No error in the getter/parser was found. And the message disappeared from the origin queue.
2) No we did not run through CEDX, I check about this CICS administrative transaction too see what can it help us. I´ll talk to the dinos here to know more about.
3) We stopped the queue trigger and browsed some messages waiting to be processed - the expiry value was '-1'.
4) 'poison message logic'? I did not get it...but in a few steps:
1) Get message
2) Check header
3) Check time (operational hours) constraint
4) Decipher payload
5) Write clear message to DB
AFAIK, all those steps are logged into a SYSOUT, but in our specific case, we found nothing written there.
5) The putter runs in a parter, 1000 miles from us. For them, the messages were being sent with no issues.
| MucheIsMyHero wrote: |
There's nothing inherently wrong with the screen showing uncommitted messages while processing is going on. When you say CICS was giving up the transaction, do you mean that the transaction would abend or just end appropriately with the data not being processed? Did you run it through CEDX? Did the message have an expiry value? What's the poison message logic look like on the getter, if any? Does your putter have any logging that indicates successful completion or an issue?
Don't want to get you chasing your tail, these were just the questions that immediately popped into my head. MQ subsystem's not the one losing messages here, I'd bet. |
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Thu Feb 14, 2008 10:10 am Post subject: |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
This may help? It means like the message is not yet sucessfully PUTTed into my local queue?
This was the queue status when the problems started...
11:53:48.28 CHEIMA 00000290 MQ DISPLAY QSTATUS(QL.REQ.78626983.00038166.01)
11:53:48.32 STC18602 00000090 CSQM293I MQ CSQMDRTC 1 QSTATUS FOUND MATCHING REQUEST CRITERIA
11:53:48.33 STC18602 00000090 CSQM201I MQ CSQMDRTC DISPLAY QSTATUS DETAILS 327
327 00000090 QSTATUS(QL.REQ.78626983.00038166.01)
327 00000090 TYPE(QUEUE)
327 00000090 OPPROCS(1)
327 00000090 IPPROCS(0)
327 00000090 CURDEPTH(2)
327 00000090 UNCOM(YES)
327 00000090 MONQ(OFF)
327 00000090 QTIME( , )
327 00000090 MSGAGE( )
327 00000090 LPUTDATE( )
327 00000090 LPUTTIME( )
327 00000090 LGETDATE( )
327 00000090 LGETTIME( )
327 00000090 QSGDISP(QMGR)
327 00000090 END QSTATUS DETAILS
11:53:48.33 STC18602 00000090 CSQ9022I MQ CSQMDRTC ' DISPLAY QSTATUS' NORMAL COMPLETION
So, probably no COAs were even returned to them?
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Thu Feb 14, 2008 11:02 am Post subject: |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
To end the story:
1) BROWSE operations are DIRTY READ. The message arrived but were not 'fully written' (commited) in the local queue, whilst it could be browsed. (assumption again)
2) The task timed-out. (fact)
3) The partner has no COAs to show. (fact)
*Still the last question: why the trigger was raised ?
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 14, 2008 12:12 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| HenriqueS wrote: |
| *Still the last question: why the trigger was raised ? |
From the APG:
| Quote: |
For non-shared local queues, the queue manager counts both committed and uncommitted messages when it assesses whether the conditions for a trigger event exist. Consequently an application might be started when there are no messages for it to retrieve because the messages on the queue have not been committed. In this situation, consider using the wait option with a suitable WaitInterval, so that the application waits for its messages to arrive.
|
I've added bold to the relevant section. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Thu Feb 14, 2008 12:30 pm Post subject: |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
Thanks, we´ve got an unique situation here, because the wait interval seemed not enough.
And about the browse? Are dirty reads (browse) possible or we just got confused?
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Feb 14, 2008 12:34 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
The QM may create a trigger message for non committed messages, but that trigger message itself will also be uncommitted until the application commits or rolls back the application message that caused the trigger event to be true. So no triggering will happen until then, even though the trigger message may have been created.
I have never heard of a dirty read. If you can browse the message you can destructivly get it.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Thu Feb 14, 2008 1:00 pm Post subject: |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
What you said goes against what the APG states IMHO: "Consequently an application might be started when there are no messages for it to retrieve because the messages on the queue have not been committed".
If an application was started it is logical to assume that a trigger message was commited, or else it would not start.
But, whatever, the issue is over. Next time this happens we will disable the trigger for a few minutes. IF the partner claims that we´ve lost messages, we will ask him the respective COAs, they won´t have them, and we will be free from hassle.
| PeterPotkay wrote: |
The QM may create a trigger message for non committed messages, but that trigger message itself will also be uncommitted until the application commits or rolls back the application message that caused the trigger event to be true. So no triggering will happen until then, even though the trigger message may have been created.
I have never heard of a dirty read. If you can browse the message you can destructivly get it. |
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 15, 2008 12:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| HenriqueS wrote: |
| And about the browse? Are dirty reads (browse) possible or we just got confused? |
You got confused.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| HenriqueS |
| Posted: Fri Feb 15, 2008 9:23 am Post subject: |
|
|
Master
Joined: 22 Sep 2006
Posts: 235
|
Update: we advised our partners to make their unit of work smaller. They were sending batches of 200 messages before a commit on their side.
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator |
|
| Back to top |
|
|
|
|
