ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Challenge Forum » Challenge Question - 12 / 2008

This forum is locked: you cannot post, reply to, or edit topics.  This topic is locked: you cannot edit posts or make replies. Goto page 1, 2  Next
 Challenge Question - 12 / 2008 « View previous topic :: View next topic » 
Author Message
Challenger
PostPosted: Mon Dec 01, 2008 1:20 am    Post subject: Challenge Question - 12 / 2008 Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Well, here comes your newest Challenger person

December 2008 Challenge:

A client application in a windows environment is configured to be triggered from a queue under a Solaris queue manager

The trigger monitor is the client version, running as a Windows service

The queue has the following attributes set:

TRIGGER
TRIGDPTH(1)
TRIGTYPE(FIRST)

The queue manager's trigger interval attribute has a very low value 10000 , compared to the default 999999999.

During testing the application team complains that multiple instances of the application are getting spawned which causes the windows server to crash. They request the MQ Admin to check if the trigger configuration is done correctly. The MQ Admin rechecks the configuration, confirms the queue is set to TRIGGER(FIRST) and advises the application team that the best practice for the triggered application is to read all the messages off the queue and exit only when it gets a 2033. The application team replies that they are indeed doing it so.

The Apps team and the MQ admin folks fight on blaming each other back and forth , finally the issue is reported resolved.

Tell me what may have caused the issue, which side do you think won out the argument, and exactly how was it eventually resolved ?

All the best!!
Back to top
View user's profile Send private message
AkankshA
PostPosted: Tue Dec 02, 2008 3:02 am    Post subject: Reply with quote

Grand Master

Joined: 12 Jan 2006
Posts: 1494
Location: Singapore

My guess,

TrigInt is set to 10 secs & Trig type is first.

Under normal conditions a trigger message would be generated whenever the queue depth reaches from 0 to 1.

Now if the application could not get started in 10 seconds(quite probable as network delay since windows to solaris) that is, the OpenInputCount for application local queue attribute is still zero, another trigger message would get generated to start application. and continue till one instance opens the queue for processesing.

Hence multiple instances of application would be started.

Now Each instance will try and read the messages from the queue. If, the application is opening the queue for exclusive input (which it should) - then only one instance will get messages

Every other instance will then throw errors, presumable. This will cause the trigger monitor to dump the initiation message on the DLQ.

As all the messages are processed, application instance will get 2033 and close the connection.

Onus lies on both application and MQ admin here...
_________________
Cheers
Back to top
View user's profile Send private message Visit poster's website
fjb_saper
PostPosted: Tue Dec 02, 2008 3:55 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Is there a message affinity question embedded there?
You did not tell why the apps folk were so upset having multiple processes spawned. What was the max number of processes ever spawned?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
PeterPotkay
PostPosted: Tue Dec 02, 2008 5:41 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

To add on to what AkankshA already said:
http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/topic/com.ibm.mq.csqzal.doc/fg13910_.htm

Quote:
If you set TriggerInterval to a low value, and there is no application serving the application queue, trigger type FIRST might behave like trigger type EVERY. This depends on the rate that messages are being put onto the application queue, which in turn might depend on other system activity. This is because, if the trigger interval is very small, another trigger message is generated each time that a message is put onto the application queue, even though the trigger type is FIRST, not EVERY. (Trigger type FIRST with a trigger interval of zero is equivalent to trigger type EVERY.)


It sounds like this application takes longer than 10,000ms to open the queue after being triggered. Maybe it takes that long to initialize itself (connect to DBs, etc).

Solution is to up the Trigger Interval or if that can't be done (its a QM level attribute shared by all apps), change the app to MQOPEN the queue ASAP after being triggered, if multiple instances are indeed a problem rather than a curiosity.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Challenger
PostPosted: Tue Dec 02, 2008 4:21 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Ok. At this moment I don't want to comment on what Akanksha and PeterPotkay has replied.Lets see if some one can look at this problem from a different angle.

To answer fjb_saper, the windows server in which the application runs is quite heavily loaded and the business is quite happy with having a single instance of receiver application processing the messages sequentially off the queue. The spawning of multiple processes is impacting other critical applications which run in the same physical windows server.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Dec 02, 2008 9:19 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Allow more time for the trigint...

Have the application open the queue in exclusive mode. You may run a DLQ handler to delete rc 265, could not start app.

You may also attempt to register the app at start up with some kind of registry. This registry keeps track of how many instances of the app are running and allows up to a predetermined max. When the application starts it tries to register. If it is past the max instances the registry returns some kind of information and the instance of the app shuts down. The registry returns to the app the ordinal and the max allowed. If the ordinal is greater than the max allowed the app signals to the registry that it is shutting down and does so.

This allows for instant modification if the registry has a method for reloading its .ini file. You can adjust the amount of resources dedicated to processing the queue according to the overall workload. At regular intervalls the instances check with the registry and shut down if the boxes workload has put their ordinal above the max allowed. If the instance's ordinal is below the max allowed it may query the registry about the total # of instances actually running. If the total # running is below the max it spawns off a new instance and initializes it (it needs to run in a different thread) to increase load and parallel processing.

If the instance has no more work it signals the registry that it is shutting down and does so.

I know this sounds quite complicated but allows for dynamic load management on the triggered side... and of course the assumption here is that there is no message affinity.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Challenger
PostPosted: Sun Dec 07, 2008 1:47 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

OK.We got a couple of solutions to the problem. But all of them point to the fact that there was a delay of 10 seconds before the triggered application opened the queue for input.

The triggered application intitializes db resources before it opens the queue. This is a resource heavy process and this is a reason why triggering of multiple application instances is not preferred. However, it was observed that this intitialization occurs within a few seconds (~less than 2 seconds). Also during the blame game period, MQAdmin changed the trigger interval parameter to the default value 999999999. Still the strange behaviour was observed.

What could have created such a bizarre situation?
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Sun Dec 07, 2008 5:39 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Challenger wrote:
However, it was observed that this intitialization occurs within a few seconds (~less than 2 seconds). Also during the blame game period, MQAdmin changed the trigger interval parameter to the default value 999999999. Still the strange behaviour was observed.

What could have created such a bizarre situation?

The app was successfully getting triggered, opening the queue, and then closing the queue (with more than 0 messages on it), but not ending. As soon as the app closed the non empty queue, another triggered instance was started, and if that one did the same thing but didn't end, a 3rd instance was triggered, etc, etc.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Challenger
PostPosted: Sun Dec 07, 2008 7:31 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Ok. Now we got a different view of the problem. Peter has suggested a possible scenario which could result in that bizarre situation. But wait, isn't the TRIGGINT meant to address this problem. Or is it?
Back to top
View user's profile Send private message
xhaxk
PostPosted: Sun Dec 07, 2008 10:45 pm    Post subject: Reply with quote

Apprentice

Joined: 30 Oct 2008
Posts: 31

No. TRIGINT only applies to msgs being put to the queue; a trigger msg generated when a non-empty queue is closed does not consider TRIGINT.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Mon Dec 08, 2008 6:52 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

xhaxk wrote:
No. TRIGINT only applies to msgs being put to the queue;

except on z/OS, where you do not need another message to be put. On z/OS Trigger Inteval passing by without another message being put will be enough to generate a new trigger message.

This doesn't apply to this Challenge which is pure Windows, but thought I'd mention it.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
xhaxk
PostPosted: Mon Dec 08, 2008 1:45 pm    Post subject: Reply with quote

Apprentice

Joined: 30 Oct 2008
Posts: 31

Quote:
except on z/OS, where you do not need another message to be put.


I did not know that. Do you have a reference for this, or a quote from a manual? I could not find it in the trigger conditions in the APG.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Mon Dec 08, 2008 2:01 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Its not in the manuals, and should be.

See this link. You can apply a patch to make it work on z/OS like it used to. In my opinion they should change distributed to match. If I set Trigger Interval to x, regardless of platform, I think it would be more intuitive if the new trigger message came out without waiting for a new message to be put. I like the way z/OS does it now.

http://www-01.ibm.com/support/docview.wss?rs=171&context=SSFKSJ&context=SSEP7X&dc=DB560&dc=DB520&q1=trigger+interval&uid=swg21225765&loc=en_US&cs=utf-8&lang=en




.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Dec 12, 2008 8:52 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Quote:
In my opinion they should change distributed to match.

The why of this is the abundant horsepower in the mainframe to take care of missing triggers, discarding expired messages, writing log buffers to logs (see below), as background tasks, with little/no impact to SLAs.

Although not part of this challenge, the same should be said about _SYNCPOINT vs. _NO_SYNCPOINT.

It would be a delight if WMQ behaved consistently across all platforms. Of course, this would cut into some of my consulting...
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Challenger
PostPosted: Thu Dec 18, 2008 3:09 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Many thanks to all for particpating in this edition of the challenge. I would like to announce Peter Potkay as the winner.

PeterPotkay wrote:
The app was successfully getting triggered, opening the queue, and then closing the queue (with more than 0 messages on it), but not ending. As soon as the app closed the non empty queue, another triggered instance was started, and if that one did the same thing but didn't end, a 3rd instance was triggered, etc, etc.


The real reason for this behaviour was the way in which the application has coded the loop.

Code:
boolean qEmpty = false;

while ( not(qEmpty))
{
MQConnect(QMGR);
MQOPEN(Q);
MQREAD(Q);
MQCLOSE(Q);
MQDISC(Q);

Exception : if mqrc = 2033
             qEmpty = true;
            else
              Exception Handler code
              break;

}


A trigger message was getting generated each time the application closed the queue within the for loop.

A special mention to xhaxk for clariying that TRIGINT is not applicable for such instances and for fjb_saper for presenting a new architecture to do dynamic load management across triggered applications.

-Challenger
Back to top
View user's profile Send private message
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.  This topic is locked: you cannot edit posts or make replies. Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » Challenge Forum » Challenge Question - 12 / 2008
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.