| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| message occasionally, randomly & inexplicably not picked |
« View previous topic :: View next topic » |
| Author |
Message
|
| mikiu |
 Posted: Mon Dec 03, 2012 12:23 pm Post subject: message occasionally, randomly & inexplicably not picked |
 |
|
Acolyte
Joined: 23 Jul 2002
Posts: 61
Location: toronto
|
Hello wise and gentle MQ folks ... a low volume, high visibility business system experiences random situations where messages are not processed in time: a CICS COBOL program puts a non-persistent message with an expiry time of 90 secs. on a remote queue to an AIX queue manager where the remote queue is also defined remote and it goes to an external (another compnay) AIX queue manager where the queue is also defined remote pointing to a z/OS queue manager where the queue is (finally) local and triggers (every) a CICS transaction. Our internal CICS COBOL program, you guessed it: issues immediately after the MQPut an MQGet with correl Id and a wait of 90 secs.
Normally (99.5%or so) the process averages 4.5 secs and never gets a "2033" on the MQGet WAIT CORRELID ... but we experience random (but daily) occurrences where the processing is in the 40 - 60 sec range. We put sniffers on wires and ran IP Traces for the port MQ uses on the AIX box where the queue manager resides and isolated a bunch of cases (and it indeed comes in bunches) where MQMD_Expiry in the outgoing message header was already depleted when the messages left our environment. I suspect it spent quality time waiting siting on the XMIT queue defined on the remote queue definiton. There are no network outages, I wrote a heartbeat program that uses remote queues for loop-back over the same channels, and that would report any network interruptions. So my question is: does the external partner not pick it up in time (and what could the possible causes be) or can it be that something on our side that keeps it?
Would appreciate your thoughts.
best regards,
MikiU |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Mon Dec 03, 2012 1:50 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Does it make sense to make important messages expire - for whatever reason they rest in any queue along their travels?
If they expire in the xmitq, make sure the xmitq is triggered to start the channel. Make sure batchsize and batchinterval are set low, so msgs don't wait for other messages to arrive before a batch is sent.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Dec 03, 2012 4:47 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
You say you have evidence that there is a significant loss of Expiry time in the message before it ever leaves your environment. I say you focus on that. The external partner can't pick it up until you deliver it.
Bruce's advice is good. Additionally, are there messages in the MQ error logs showing potential issues with the channels in your environment?
Are there potential lots of very large (100MB) messages being sent over these same channels, causing your timely messages to sit and wait their turn in the XMITQ while the channel and network deals with the jumbo messages?
Are these SNDR/RCVR channel pairs?
If the messages are getting there quickly, but the replies are still coming back late:
Trigger Every - unless their apps are written to read the queue until no more messages, you are almost guaranteed to sooner or later to have a rolling back log of app messages on the app queue, where message #23523 is sitting an waiting in the queue, until the queue triggers (with only ONE trigger message) when message #23524 finally arrives.
Maybe their channels are not triggered properly, or have a giant Batch Interval.
How do you know its not their app or network that is slow? Or maybe they put the reply but don't commit that MQPUT immediately sometimes.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Dec 03, 2012 6:18 pm Post subject: Re: message occasionally, randomly & inexplicably not pi |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| mikiu wrote: |
| Hello wise and gentle MQ folks ... a low volume, high visibility business system experiences random situations where messages are not processed in time: a CICS COBOL program puts a non-persistent message with an expiry time of 90 secs. |
High visibility means the messages are important, and should not be lost?
1) Use persistent messages
2) Use unlimited expiry
There are many steps involved in processing the request and reply messages. The culprit is probably one of these. Exactly how much Expiry time is depleted on the outgoing messages by the time it is tranported over the channel?
_________________
Glenn |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Tue Dec 04, 2012 8:49 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
| On the AIX systems, ensure that ULIMIT requirements are met for both mqm and root. I've seen issues where the tuning got lost at a version upgrade of AIX. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Dec 04, 2012 8:51 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Are the channels disconnecting upon inactivity? The first message after that might take longer to arrive as the channels have to re-connect. However your heartbeat messages would persumably keep the channels active. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
