ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportTransmission queues = FIFO queues

Post new topicReply to topic Goto page 1, 2  Next
Transmission queues = FIFO queues View previous topic :: View next topic
Author Message
tekt9
PostPosted: Tue Sep 25, 2012 6:15 am Post subject: Transmission queues = FIFO queues Reply with quote

Novice

Joined: 13 Aug 2012
Posts: 15

Hi…
I had an issue recently which made me wonder how the transmission queue works.

We had the following scenario:
Computer “A” has a remote queue “RemQ A” which sends data to a local queue “LocQ B” on Computer “B”.

At a given point “LocQ B” ran full and we reached the max queue depth of this queue. This caused the messages from “RemQ A” to get stuck in the transmission queue on Computer “A” and filled it up.

There is however other services (message flows) which are dependent of the transmission queue to send data to Computer “B”. Due to the transmission queue being full these messages were not put on the queue. As the solution functions as backend for a high-availability service, it is a must, that the lines are open.

The best solution I could come up with is creating different transmission queues so we can avoid the single point of failure. Do you guys have other ideas of how to solve the issue?

/regards Tharani
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Sep 25, 2012 6:26 am Post subject: Re: Transmission queues = FIFO queues Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

tekt9 wrote:
As the solution functions as backend for a high-availability service, it is a must, that the lines are open.


Then where's your monitoring? When the local queue started to fill up on B your support people should have been paged and the queue manually unloaded / the max depth increased / the sending application stopped / etc.

When the transmission queue got above 20% full not only should the support people have been paged but the building fire alarm should have gone off! A non-empty transmission queue is always a sign something bad is happening and prompt action is needed, especially in a high availability service.

tekt9 wrote:
Do you guys have other ideas of how to solve the issue?


Better monitoring, and properly sized queue depths.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Sep 25, 2012 1:21 pm Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

You're talking about what's known as 'quality of service'. This tends to be implemented using MQ clusters, with different clusters for each QoS.

But!

First, you should ensure that you have proper management of the quality of service you have right now - which, again, requires monitoring of your queue manager and proper response to monitoring alerts...

Second, you can do it without using MQ clusters, you just have to have a reasonable understanding of how MQ name resolution works, and more importantly have a good solid control over how applications specify the names of MQ objectst that they access - i.e. you need to make sure you can change the names of those objects without requiring a change to the application itself.
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Sep 26, 2012 1:23 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

Make sure the XMIT queue has enough depth. But if it's not being emptied by the channel you have some sort of problem in your infrastructure.

Unless you have massive volumes, or a slow network, a single channel is unlikely to be a bottleneck.

The Message flow which encounters the queue full error (RC 2053) will decide how to handle it. I am not sure exactly what WMB does in this situation - I imagine that it either sends it to the DLQ or rolls back the transaction.
Back to top
View user's profile Send private message
tekt9
PostPosted: Wed Sep 26, 2012 4:14 am Post subject: Thanks for the input Reply with quote

Novice

Joined: 13 Aug 2012
Posts: 15

I've talked with the people whe are responsible for the Computer "B". The solution we've come up with, is to use automated recovery, instead of creating different transmission queues. The messages are not of high priority, and we've made the queue throw a full event, which clears some of the messages. This solution is somewhat more elegant as we won't juggle with several transmission queues.
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Sep 26, 2012 4:32 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

That sounds to me like dealing with the symptom of the problem and not the cause.

Why would your transmission queue fill up in the first place?

MQ is good at buffering message volumes, but only if you allocate enough space on the queue for expected peaks.

Throwing messages away seems like a bad design. If messages are not wanted after a certain elapsed time, use a MQMD expiry value.

You can use message priority of course as a standard MQ feature.

You should normally use NPMSPEED(FAST) on the sender channel.

If you want to detect channel problems - use channel events rather than waiting for the xmit queue to become (partially or totally) full.

Or use any of the free or paid-for monitoring tools. Detecting channel problems is a requirement that dates back (and has been solved) since the neolithic era.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Sep 26, 2012 4:59 pm Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

You should have a dead letter queue in use on the queue manager on computer b.
You should tune the message retry interval and the message retry count on the RCVR channel on Queue Manager B so that the messages for LocQ B are put to the DLQ as quickly or as slowly as you want when LocQ B fills up. Set the count to zero and the messages will immediately go to the DLQ allowing other messages to other queues to flow.


Automatically deleting messages when the queue is full is ridiculous.

Exactly what is the Max Q depth and Max Message size of this queue?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Sep 26, 2012 5:36 pm Post subject: Re: Thanks for the input Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

tekt9 wrote:
This solution is somewhat more elegant as we won't juggle with several transmission queues.


That's as elegant as fixing a stuck door by throwing a brick through your window!

Seriously, if the messages are so "not high priority" that they can be deleted at a whim then why send them? How do you decide how many to remove? The only reason you can delete them in such an arbitrary fashion is because they're no longer needed due to the passage of time, in which case they should be set to expire.

There's also no need to "juggle" several transmission queues. I and others have pointed out how to use a single transmission queue to achieve what you need.

I also ask this questions: what are you going to do 6 months from now, when someone starts sending messages with a value that can't just be flushed when you need some space?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
tekt9
PostPosted: Wed Sep 26, 2012 10:37 pm Post subject: large amount of data Reply with quote

Novice

Joined: 13 Aug 2012
Posts: 15

I was discurraged to buffer messages on the queues unless they need to be processed, as the MQ did some internal indexing which may influence performance if I had 50000 messages in a queue. An IBM consultant told me this, so I just figured it was a bad idea in general to have large amount of messages in queues.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 27, 2012 4:33 am Post subject: Re: large amount of data Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

tekt9 wrote:
I was discurraged to buffer messages on the queues unless they need to be processed, as the MQ did some internal indexing which may influence performance if I had 50000 messages in a queue.


It does, it's not a good idea to rountinely store large numbers messages for extended periods but this is an exceptional failure situation and how is just trashing some of the messages a better idea???

If you're that gung ho to get them off the queue unload them to the file system and reload them when you're ready to process.

tekt9 wrote:
An IBM consultant told me this


Who? What documentation did he offer to back this?

tekt9 wrote:
, so I just figured it was a bad idea in general to have large amount of messages in queues.


So you never stopped, while figuring, to wonder why if this is such a bad thing than IBM has MAXDEPTH allow such a large possible value? You think that generally this sort of situation is going to happen to you?

I still can't believe your business users are happy to just watch their messages being trashed because the technology is full.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Sep 27, 2012 5:11 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

One of the main reasons for using MQ is to provide queueing of messages...

Indexing on queues only applies on z/OS. You don't have to index queues if you don't want to.

For a XMIT queue, there is no point in doing so.

Why use an "assured" delivery protocol and then "bin" the messages? - it would be funny if it wasn't so sad...

You have to allocate enough disk space to the queue manager of course, but having a queue depth of (for example) 200,000 (or more) is not really a problem if the circumstances justify it.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 27, 2012 5:30 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

zpat wrote:
One of the main reasons for using MQ is to provide queueing of messages...


Quite.

zpat wrote:
Indexing on queues only applies on z/OS. You don't have to index queues if you don't want to.


z/OS allows for explict indexes on the msg & correl ids. Distributed queues are still indexed internally for the queue manager's own purposes, but I can't see any scenario where this trivial indexing would have any measurable impact and certainly not on a queue of only 50,000 messages unless the hardware has to be wound with a key every few hours!

I also have a suspicion that this "IBM consultant" is in fact "a consultant who claimed expertise with IBM software".

zpat wrote:
For a XMIT queue, there is no point in doing so.


And as I indicated a few posts back, if there are enough messages on an XMITQ for this to have an impact, you're in very serious trouble.

zpat wrote:
Why use an "assured" delivery protocol and then "bin" the messages? - it would be funny if it wasn't so sad...


Well the delivery is still assured; there's just no assurance your message will still be there when you want to read it......
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
exerk
PostPosted: Thu Sep 27, 2012 6:03 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

Vitor wrote:
I also have a suspicion that this "IBM consultant" is in fact "a consultant who claimed expertise with IBM software".

Or managed to swing a contract with them...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Sep 27, 2012 6:24 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

IBM Global Services hire people off the street like any other company - working for IBM doesn't make them experts automatically.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 27, 2012 6:55 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

zpat wrote:
IBM Global Services hire people off the street like any other company - working for IBM doesn't make them experts automatically.


Hey, they used to give me contracts when they were short handed. Once they even made me wear the suit.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexGeneral IBM MQ SupportTransmission queues = FIFO queues
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.