Author |
Message
|
tekt9 |
Posted: Tue Sep 25, 2012 6:15 am Post subject: Transmission queues = FIFO queues |
|
|
Novice
Joined: 13 Aug 2012 Posts: 15
|
Hi…
I had an issue recently which made me wonder how the transmission queue works.
We had the following scenario:
Computer “A” has a remote queue “RemQ A” which sends data to a local queue “LocQ B” on Computer “B”.
At a given point “LocQ B” ran full and we reached the max queue depth of this queue. This caused the messages from “RemQ A” to get stuck in the transmission queue on Computer “A” and filled it up.
There is however other services (message flows) which are dependent of the transmission queue to send data to Computer “B”. Due to the transmission queue being full these messages were not put on the queue. As the solution functions as backend for a high-availability service, it is a must, that the lines are open.
The best solution I could come up with is creating different transmission queues so we can avoid the single point of failure. Do you guys have other ideas of how to solve the issue?
/regards Tharani |
|
Back to top |
|
 |
Vitor |
Posted: Tue Sep 25, 2012 6:26 am Post subject: Re: Transmission queues = FIFO queues |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
tekt9 wrote: |
As the solution functions as backend for a high-availability service, it is a must, that the lines are open. |
Then where's your monitoring? When the local queue started to fill up on B your support people should have been paged and the queue manually unloaded / the max depth increased / the sending application stopped / etc.
When the transmission queue got above 20% full not only should the support people have been paged but the building fire alarm should have gone off! A non-empty transmission queue is always a sign something bad is happening and prompt action is needed, especially in a high availability service.
tekt9 wrote: |
Do you guys have other ideas of how to solve the issue?
|
Better monitoring, and properly sized queue depths. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
mqjeff |
Posted: Tue Sep 25, 2012 1:21 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
You're talking about what's known as 'quality of service'. This tends to be implemented using MQ clusters, with different clusters for each QoS.
But!
First, you should ensure that you have proper management of the quality of service you have right now - which, again, requires monitoring of your queue manager and proper response to monitoring alerts...
Second, you can do it without using MQ clusters, you just have to have a reasonable understanding of how MQ name resolution works, and more importantly have a good solid control over how applications specify the names of MQ objectst that they access - i.e. you need to make sure you can change the names of those objects without requiring a change to the application itself. |
|
Back to top |
|
 |
zpat |
Posted: Wed Sep 26, 2012 1:23 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Make sure the XMIT queue has enough depth. But if it's not being emptied by the channel you have some sort of problem in your infrastructure.
Unless you have massive volumes, or a slow network, a single channel is unlikely to be a bottleneck.
The Message flow which encounters the queue full error (RC 2053) will decide how to handle it. I am not sure exactly what WMB does in this situation - I imagine that it either sends it to the DLQ or rolls back the transaction. |
|
Back to top |
|
 |
tekt9 |
Posted: Wed Sep 26, 2012 4:14 am Post subject: Thanks for the input |
|
|
Novice
Joined: 13 Aug 2012 Posts: 15
|
I've talked with the people whe are responsible for the Computer "B". The solution we've come up with, is to use automated recovery, instead of creating different transmission queues. The messages are not of high priority, and we've made the queue throw a full event, which clears some of the messages. This solution is somewhat more elegant as we won't juggle with several transmission queues. |
|
Back to top |
|
 |
zpat |
Posted: Wed Sep 26, 2012 4:32 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
That sounds to me like dealing with the symptom of the problem and not the cause.
Why would your transmission queue fill up in the first place?
MQ is good at buffering message volumes, but only if you allocate enough space on the queue for expected peaks.
Throwing messages away seems like a bad design. If messages are not wanted after a certain elapsed time, use a MQMD expiry value.
You can use message priority of course as a standard MQ feature.
You should normally use NPMSPEED(FAST) on the sender channel.
If you want to detect channel problems - use channel events rather than waiting for the xmit queue to become (partially or totally) full.
Or use any of the free or paid-for monitoring tools. Detecting channel problems is a requirement that dates back (and has been solved) since the neolithic era. |
|
Back to top |
|
 |
PeterPotkay |
Posted: Wed Sep 26, 2012 4:59 pm Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
You should have a dead letter queue in use on the queue manager on computer b.
You should tune the message retry interval and the message retry count on the RCVR channel on Queue Manager B so that the messages for LocQ B are put to the DLQ as quickly or as slowly as you want when LocQ B fills up. Set the count to zero and the messages will immediately go to the DLQ allowing other messages to other queues to flow.
Automatically deleting messages when the queue is full is ridiculous.
Exactly what is the Max Q depth and Max Message size of this queue? _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
Vitor |
Posted: Wed Sep 26, 2012 5:36 pm Post subject: Re: Thanks for the input |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
tekt9 wrote: |
This solution is somewhat more elegant as we won't juggle with several transmission queues. |
That's as elegant as fixing a stuck door by throwing a brick through your window!
Seriously, if the messages are so "not high priority" that they can be deleted at a whim then why send them? How do you decide how many to remove? The only reason you can delete them in such an arbitrary fashion is because they're no longer needed due to the passage of time, in which case they should be set to expire.
There's also no need to "juggle" several transmission queues. I and others have pointed out how to use a single transmission queue to achieve what you need.
I also ask this questions: what are you going to do 6 months from now, when someone starts sending messages with a value that can't just be flushed when you need some space? _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
tekt9 |
Posted: Wed Sep 26, 2012 10:37 pm Post subject: large amount of data |
|
|
Novice
Joined: 13 Aug 2012 Posts: 15
|
I was discurraged to buffer messages on the queues unless they need to be processed, as the MQ did some internal indexing which may influence performance if I had 50000 messages in a queue. An IBM consultant told me this, so I just figured it was a bad idea in general to have large amount of messages in queues. |
|
Back to top |
|
 |
Vitor |
Posted: Thu Sep 27, 2012 4:33 am Post subject: Re: large amount of data |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
tekt9 wrote: |
I was discurraged to buffer messages on the queues unless they need to be processed, as the MQ did some internal indexing which may influence performance if I had 50000 messages in a queue. |
It does, it's not a good idea to rountinely store large numbers messages for extended periods but this is an exceptional failure situation and how is just trashing some of the messages a better idea???
If you're that gung ho to get them off the queue unload them to the file system and reload them when you're ready to process.
tekt9 wrote: |
An IBM consultant told me this |
Who? What documentation did he offer to back this?
tekt9 wrote: |
, so I just figured it was a bad idea in general to have large amount of messages in queues. |
So you never stopped, while figuring, to wonder why if this is such a bad thing than IBM has MAXDEPTH allow such a large possible value? You think that generally this sort of situation is going to happen to you?
I still can't believe your business users are happy to just watch their messages being trashed because the technology is full. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
zpat |
Posted: Thu Sep 27, 2012 5:11 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
One of the main reasons for using MQ is to provide queueing of messages...
Indexing on queues only applies on z/OS. You don't have to index queues if you don't want to.
For a XMIT queue, there is no point in doing so.
Why use an "assured" delivery protocol and then "bin" the messages? - it would be funny if it wasn't so sad...
You have to allocate enough disk space to the queue manager of course, but having a queue depth of (for example) 200,000 (or more) is not really a problem if the circumstances justify it. |
|
Back to top |
|
 |
Vitor |
Posted: Thu Sep 27, 2012 5:30 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
zpat wrote: |
One of the main reasons for using MQ is to provide queueing of messages... |
Quite.
zpat wrote: |
Indexing on queues only applies on z/OS. You don't have to index queues if you don't want to. |
z/OS allows for explict indexes on the msg & correl ids. Distributed queues are still indexed internally for the queue manager's own purposes, but I can't see any scenario where this trivial indexing would have any measurable impact and certainly not on a queue of only 50,000 messages unless the hardware has to be wound with a key every few hours!
I also have a suspicion that this "IBM consultant" is in fact "a consultant who claimed expertise with IBM software".
zpat wrote: |
For a XMIT queue, there is no point in doing so. |
And as I indicated a few posts back, if there are enough messages on an XMITQ for this to have an impact, you're in very serious trouble.
zpat wrote: |
Why use an "assured" delivery protocol and then "bin" the messages? - it would be funny if it wasn't so sad...  |
Well the delivery is still assured; there's just no assurance your message will still be there when you want to read it......  _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
exerk |
Posted: Thu Sep 27, 2012 6:03 am Post subject: |
|
|
 Jedi Council
Joined: 02 Nov 2006 Posts: 6339
|
Vitor wrote: |
I also have a suspicion that this "IBM consultant" is in fact "a consultant who claimed expertise with IBM software". |
Or managed to swing a contract with them...  _________________ It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
Back to top |
|
 |
zpat |
Posted: Thu Sep 27, 2012 6:24 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
IBM Global Services hire people off the street like any other company - working for IBM doesn't make them experts automatically. |
|
Back to top |
|
 |
Vitor |
Posted: Thu Sep 27, 2012 6:55 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
zpat wrote: |
IBM Global Services hire people off the street like any other company - working for IBM doesn't make them experts automatically. |
Hey, they used to give me contracts when they were short handed. Once they even made me wear the suit. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
|