ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » Sender Channel Overload

Post new topic  Reply to topic
 Sender Channel Overload « View previous topic :: View next topic » 
Author Message
insertcoin
PostPosted: Thu Sep 06, 2007 1:05 am    Post subject: Sender Channel Overload Reply with quote

Acolyte

Joined: 07 Jun 2007
Posts: 61
Location: who knows!?

Hi Guys

My sender channel hands a large quantity of messages per day, and sometimes the messages are put on the queue quicker than they get sent.
This causes the transmission queue to fill up. At one point the queue will get too full and the sender channel will fail with this error:

Cannot send message through channel 'QM_SEF_P01.PGEN1'.

The channel has closed because the remote queue manager cannot receive a message.

Contact the systems administrator who should examine the error files of the remote queue manager, to determine why the message cannot be received, and then restart the channel.

AND

Channel 'QM_SEF_P01.PGEN1' has ended because the remote queue manager did not accept the last batch of messages.

The error log for the channel at the remote site will contain an explanation of the failure. Contact the remote Systems Administrator to resolve the problem.


Here are the settings I have set up for a sender channel:

Channel Name:......................X.TO.Y
Type:....................................Sender
Transmission Protocol:............TCP/IP

Maximum Msg Length:............4194304
HeartBeat Interval:................300
Sequence Number Wrap:........999999999
Non-persistant Msg Speed:......FAST
Batch Size:............................50
Batch Interval:.......................0
Disconnect Interval.................6000
Data Conversion:...................No
Put Authority:........................NULL
Batch HeartBeat Interval:.......0

Short Retry Count:.................10
Short Retry Interval:..............60
Long Retry Count:..................999999999
Long Retry Interval:...............1200


I dont know why this fails like this, and this is our production queue with important info. The only way I get the queue back up is to STOP RESET and RESOLVE with COMMIT until the queue gets empty enough to resume normal sending.

I know this is on the other side, the remote queue side, and the company on the other side knows as much as I do, maybe less...
Please could you advise at to what steps I could take to prevent this from happening again, and perhaps suggestions I can give to the MQ Admin on the other side so we can get over the PROD Channel Failing!

Thanks so much!
Justin
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 06, 2007 1:11 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

1) Increase the max depth of your transmission queue (or the space available to it if it's actually full instead of logically full)

2) Get the MQ admin the other side to increase the max depth and / or space available to the receiving queues

3) Get the developers the other side to develop (or redevelop) their application so it processes messages faster. Or spin up more instances of it when the system's under load. There are a number of ways they could detect this & responde automatically.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
insertcoin
PostPosted: Thu Sep 06, 2007 1:14 am    Post subject: Reply with quote

Acolyte

Joined: 07 Jun 2007
Posts: 61
Location: who knows!?

Quote:
Or spin up more instances of it when the system's under load. There are a number of ways they could detect this & responde automatically.


Could you advise me on how to do this?

Ill speak to the other Side and get them to increase the max depth!

Thanks Vitor!

editted:

Some more Info about the transmission Q:


Maximum Queue Depth:............100000
Maximum MessgaeLength:.........4194304
Shareability:.............................Shareable
Default Input Open Option:.........Shared
Message Delivery Sequence:.......Priority
Retention Interval:.....................99999999
Definition Types(greyed):...........Predefined
Distribution Lists:.......................Enabled

Maximum Depth Event: .............Enabled
High Depth Event:......................Disabled
High Depth Limit:.......................80
Low Depth Event:.......................Disabled
Low Depth Limit:........................20
Service Interval Event:...............None
Service Interval:........................999999999


Is there anything else that could affect the sender channel going down?

Thanks


Last edited by insertcoin on Thu Sep 06, 2007 1:22 am; edited 1 time in total
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 06, 2007 1:22 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

insertcoin wrote:
Could you advise me on how to do this?


1) Buy monitoring software

2) Use Depth triggering

3) Long running task with PCF commands (like 1 but cheaper and simpler)

4) Any combination of the above

IMHO your best short term fix would be to increase storage your side. It's under your control and it's wise to have adequate storage in a prod environment to deal with outages like network failures. It sounds from your description that this is happening frequently, implying your depth is too small for the load you're taking. As a rule of thumb, the xmitq should be able to hold 75% of the largest possible number of messages that could be sent in the longest possible outage allowed by the SLA.

This formula obviously yield a wide variety of thumb sizes.

I'm sure others will have differing views & other monitoring suggestions, all of which may be equally valid.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
insertcoin
PostPosted: Thu Sep 06, 2007 1:25 am    Post subject: Reply with quote

Acolyte

Joined: 07 Jun 2007
Posts: 61
Location: who knows!?

What a legend! Thanks Vitor!

Monitoring Software Suggestions:??
Money is no problem
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 06, 2007 1:26 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

insertcoin wrote:
Is there anything else that could affect the sender channel going down?


Whatever it is, it's in the log of the receiver MCA. Look in that (or get the other side to).

This is another reason for increasing sender side xmitq depth. It covers you against any failure their side.

What's your message throughput? I notice xmitq maxdepth is 100000, how long does it take to hit that? It might give a clue to what's going on their side (though the most likely explaination for that message remains receiver queue full IMHO)
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Sep 06, 2007 3:11 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

insertcoin wrote:
What a legend! Thanks Vitor!

Monitoring Software Suggestions:??
Money is no problem

Is there anything else that could affect the sender channel going down?

Have a look at the monitoring forum. You'll need to write up your requirements, shop around and get the best fit/value.

Anything else that could affect the sender channel going down...
How about making sure the receiving side has a DLQ defined for the qmgr?
This does not mean checking the existence of the SYSTEM.DEAD.LETTER.QUEUE but checking the properties of the qmgr:

dis qmgr deadq

Enjoy
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Thu Sep 06, 2007 3:29 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 06, 2007 3:32 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

fjb_saper wrote:
How about making sure the receiving side has a DLQ defined for the qmgr?


Ooo... good point!

But would you be able to restart the channel in that circumstance? Would the RESOLVE with commit clear out the maladdressed message?

Never a pair of test queue managers around when you need them...
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 06, 2007 4:19 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Vitor wrote:
But would you be able to restart the channel in that circumstance? Would the RESOLVE with commit clear out the maladdressed message?


The answer would seem to be yes provided you reset the sequence numbers as well. The maladdressed message gets lost in the void between queue managers.....
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Sep 06, 2007 4:24 am    Post subject: Re: Sender Channel Overload Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7722

insertcoin wrote:
The only way I get the queue back up is to STOP RESET and RESOLVE with COMMIT until the queue gets empty enough to resume normal sending.

Don't do this unless your log specifically tells you you have sequence # errors or the channel is in doubt. The reason you think it worked is because by the time you went thru the motions the other side's queue drained a bit and when you manually restarted the channel it was able to move some more messages.

The other side should have a DLQ. If they don't your channel hard stops when the destination q fills. If they do have a DLQ your channel will stop when the destination q AND the DLQ fills up.

Your sender channel is not failing because your XMITQ filled up, it stops because the other side had some problem, which we all are assuming is queues filling up. Until you verify this by seeing what their logs say be aware it might be something else causing the problem.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 06, 2007 4:31 am    Post subject: Re: Sender Channel Overload Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

PeterPotkay wrote:
Until you verify this by seeing what their logs say be aware it might be something else causing the problem.


A very good point and ties back to my earlier post, which remains valid despite what my little experiement has shown.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General Discussion » Sender Channel Overload
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.