ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Mainframe, CICS, TXSeries » Urgent - uncommited messages under CICS

Post new topic  Reply to topic Goto page 1, 2  Next
 Urgent - uncommited messages under CICS « View previous topic :: View next topic » 
Author Message
HenriqueS
PostPosted: Wed Feb 13, 2008 7:22 am    Post subject: Urgent - uncommited messages under CICS Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

Our production system is not delivering the messages to our application.

There is a trigger that goest into a local queue to read every new incoming message. A new CICS transaction is forked to GET it, but it stalls, MQ says that the message is uncommited and our app does not receive it.

After, some sort of timeout occurs. The message disappears from the queue (at least is not browsable and the the queue depth lowers!) and is not delivered to the application!

Other new CICS transactions are forked to receive the later messages but the same thing happens.

What can I do?
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Feb 13, 2008 7:23 am    Post subject: Re: Urgent - uncommited messages under CICS Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

HenriqueS wrote:
What can I do?


Work out why the sending application isn't commiting the message?

If I had to guess, the sending application is trying to do the put as part of an XA transaction & something's preventing that from completing (database deadlock perhaps). The transaction timeout expires, the MQ message is rolled back as intended and your CICS transaction trying to do the get is left high & dry.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Wed Feb 13, 2008 9:11 am    Post subject: Re: Urgent - uncommited messages under CICS Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

Sending app? The sending app is in a remote server. I did not know that messages flowing from a remote server must be commited there...really?

Vitor wrote:
HenriqueS wrote:
What can I do?


Work out why the sending application isn't commiting the message?

If I had to guess, the sending application is trying to do the put as part of an XA transaction & something's preventing that from completing (database deadlock perhaps). The transaction timeout expires, the MQ message is rolled back as intended and your CICS transaction trying to do the get is left high & dry.

_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Feb 13, 2008 10:05 am    Post subject: Re: Urgent - uncommited messages under CICS Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

HenriqueS wrote:
...MQ says that the message is uncommited and our app does not receive it.

I don't understand that. What do you mean MQ "says" its uncommited?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Wed Feb 13, 2008 12:35 pm    Post subject: Re: Urgent - uncommited messages under CICS Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

It is not showing now, but the "Uncommitted" column kept showing "Y". Watching the MQ-CICS interface transaction (CKCQ) it seemed that the consuming transactions were spending too much time on the GET operation.

Worst, CICS was giving up the transaction without rolling back the GET. Messages were going to nowhere and disappearing from the queue they were on...

No meaninful messages on the *MSTR sysouts were given nor in the CICS sysouts.

We stopped the trigger, waited for half hour and reestarted, and it went ok.

We suspected that our consuming app was entering in some loop or condition it coldn´t handle, but then we get to this:
a) With the timeout, why the GET operation was not rolled back?
b) This app runs continously for the last 5 years and never such problem happened.
c) All message parsing functions in the app return status code into a sysout. This sysout was not even touched.
d) The same application (under the same CICS region) is used to receive data from several other partners (but they do leave their messages in other queues), the issue only happened with one partner, while others, with with 10 times more message delivery throughput, did not show anything wrong.

Code:

                      LIST LOCAL QUEUES - STATUS - BCMP              ROW 1 OF 2
                                                                               
 TYPE ACTION CODES, THEN PRESS ENTER.  PRESS F11 TO DISPLAY ALL OPEN QUEUES.   
  1=DISPLAY APPLICATIONS   3=ALTER   4=MANAGE                                 
                                                                               
     QUEUE NAME                                       DISPOSITION   DEPTH     
       LAST PUT TIME        LAST GET TIME         TIME ON QUEUE       MAX AGE 
         OUTPUT USE  INPUT USE   UNCOMMITTED MSGS                             
 <>  QL.REQ.78*                                       PRIVATE BCMP             
     QL.REQ.78626983.00038166.01                      QMGR    BCMP  0         
       2008-02-13 17.32.05  2008-02-13 17.32.05   646630    6281012           
                      ******** End of list ********                           
         





PeterPotkay wrote:
HenriqueS wrote:
...MQ says that the message is uncommited and our app does not receive it.

I don't understand that. What do you mean MQ "says" its uncommited?

_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
MucheIsMyHero
PostPosted: Wed Feb 13, 2008 2:58 pm    Post subject: Reply with quote

Novice

Joined: 29 Jun 2005
Posts: 14

There's nothing inherently wrong with the screen showing uncommitted messages while processing is going on. When you say CICS was giving up the transaction, do you mean that the transaction would abend or just end appropriately with the data not being processed? Did you run it through CEDX? Did the message have an expiry value? What's the poison message logic look like on the getter, if any? Does your putter have any logging that indicates successful completion or an issue?

Don't want to get you chasing your tail, these were just the questions that immediately popped into my head. MQ subsystem's not the one losing messages here, I'd bet.
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Thu Feb 14, 2008 9:25 am    Post subject: Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

Trying to answer:

1) The transaction just ended but produced no results on the target database. No error on the database was found. No error in the getter/parser was found. And the message disappeared from the origin queue.

2) No we did not run through CEDX, I check about this CICS administrative transaction too see what can it help us. I´ll talk to the dinos here to know more about.

3) We stopped the queue trigger and browsed some messages waiting to be processed - the expiry value was '-1'.

4) 'poison message logic'? I did not get it...but in a few steps:
1) Get message
2) Check header
3) Check time (operational hours) constraint
4) Decipher payload
5) Write clear message to DB
AFAIK, all those steps are logged into a SYSOUT, but in our specific case, we found nothing written there.

5) The putter runs in a parter, 1000 miles from us. For them, the messages were being sent with no issues.



MucheIsMyHero wrote:
There's nothing inherently wrong with the screen showing uncommitted messages while processing is going on. When you say CICS was giving up the transaction, do you mean that the transaction would abend or just end appropriately with the data not being processed? Did you run it through CEDX? Did the message have an expiry value? What's the poison message logic look like on the getter, if any? Does your putter have any logging that indicates successful completion or an issue?

Don't want to get you chasing your tail, these were just the questions that immediately popped into my head. MQ subsystem's not the one losing messages here, I'd bet.

_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Thu Feb 14, 2008 10:10 am    Post subject: Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

This may help? It means like the message is not yet sucessfully PUTTed into my local queue?

This was the queue status when the problems started...

11:53:48.28 CHEIMA 00000290 MQ DISPLAY QSTATUS(QL.REQ.78626983.00038166.01)
11:53:48.32 STC18602 00000090 CSQM293I MQ CSQMDRTC 1 QSTATUS FOUND MATCHING REQUEST CRITERIA
11:53:48.33 STC18602 00000090 CSQM201I MQ CSQMDRTC DISPLAY QSTATUS DETAILS 327
327 00000090 QSTATUS(QL.REQ.78626983.00038166.01)
327 00000090 TYPE(QUEUE)
327 00000090 OPPROCS(1)
327 00000090 IPPROCS(0)
327 00000090 CURDEPTH(2)
327 00000090 UNCOM(YES)
327 00000090 MONQ(OFF)
327 00000090 QTIME( , )
327 00000090 MSGAGE( )
327 00000090 LPUTDATE( )
327 00000090 LPUTTIME( )
327 00000090 LGETDATE( )
327 00000090 LGETTIME( )
327 00000090 QSGDISP(QMGR)
327 00000090 END QSTATUS DETAILS
11:53:48.33 STC18602 00000090 CSQ9022I MQ CSQMDRTC ' DISPLAY QSTATUS' NORMAL COMPLETION


So, probably no COAs were even returned to them?
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Thu Feb 14, 2008 11:02 am    Post subject: Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

To end the story:
1) BROWSE operations are DIRTY READ. The message arrived but were not 'fully written' (commited) in the local queue, whilst it could be browsed. (assumption again)

2) The task timed-out. (fact)

3) The partner has no COAs to show. (fact)

*Still the last question: why the trigger was raised ?
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Feb 14, 2008 12:12 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

HenriqueS wrote:
*Still the last question: why the trigger was raised ?


From the APG:

Quote:
For non-shared local queues, the queue manager counts both committed and uncommitted messages when it assesses whether the conditions for a trigger event exist. Consequently an application might be started when there are no messages for it to retrieve because the messages on the queue have not been committed. In this situation, consider using the wait option with a suitable WaitInterval, so that the application waits for its messages to arrive.


I've added bold to the relevant section.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Thu Feb 14, 2008 12:30 pm    Post subject: Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

Thanks, we´ve got an unique situation here, because the wait interval seemed not enough.

And about the browse? Are dirty reads (browse) possible or we just got confused?
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Feb 14, 2008 12:34 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

The QM may create a trigger message for non committed messages, but that trigger message itself will also be uncommitted until the application commits or rolls back the application message that caused the trigger event to be true. So no triggering will happen until then, even though the trigger message may have been created.



I have never heard of a dirty read. If you can browse the message you can destructivly get it.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Thu Feb 14, 2008 1:00 pm    Post subject: Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

What you said goes against what the APG states IMHO: "Consequently an application might be started when there are no messages for it to retrieve because the messages on the queue have not been committed".

If an application was started it is logical to assume that a trigger message was commited, or else it would not start.

But, whatever, the issue is over. Next time this happens we will disable the trigger for a few minutes. IF the partner claims that we´ve lost messages, we will ask him the respective COAs, they won´t have them, and we will be free from hassle.

PeterPotkay wrote:
The QM may create a trigger message for non committed messages, but that trigger message itself will also be uncommitted until the application commits or rolls back the application message that caused the trigger event to be true. So no triggering will happen until then, even though the trigger message may have been created.



I have never heard of a dirty read. If you can browse the message you can destructivly get it.

_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Feb 15, 2008 12:48 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

HenriqueS wrote:
And about the browse? Are dirty reads (browse) possible or we just got confused?


You got confused.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
HenriqueS
PostPosted: Fri Feb 15, 2008 9:23 am    Post subject: Reply with quote

Master

Joined: 22 Sep 2006
Posts: 235

Update: we advised our partners to make their unit of work smaller. They were sending batches of 200 messages before a commit on their side.
_________________
HenriqueS
Certified Websphere MQ 6.0 System Administrator
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » Mainframe, CICS, TXSeries » Urgent - uncommited messages under CICS
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.