| Author |
Message
|
| insertcoin |
 Posted: Thu Sep 06, 2007 1:05 am Post subject: Sender Channel Overload |
 |
|
Acolyte
Joined: 07 Jun 2007
Posts: 61
Location: who knows!?
|
Hi Guys
My sender channel hands a large quantity of messages per day, and sometimes the messages are put on the queue quicker than they get sent.
This causes the transmission queue to fill up. At one point the queue will get too full and the sender channel will fail with this error:
Cannot send message through channel 'QM_SEF_P01.PGEN1'.
The channel has closed because the remote queue manager cannot receive a message.
Contact the systems administrator who should examine the error files of the remote queue manager, to determine why the message cannot be received, and then restart the channel.
AND
Channel 'QM_SEF_P01.PGEN1' has ended because the remote queue manager did not accept the last batch of messages.
The error log for the channel at the remote site will contain an explanation of the failure. Contact the remote Systems Administrator to resolve the problem.
Here are the settings I have set up for a sender channel:
Channel Name:......................X.TO.Y
Type:....................................Sender
Transmission Protocol:............TCP/IP
Maximum Msg Length:............4194304
HeartBeat Interval:................300
Sequence Number Wrap:........999999999
Non-persistant Msg Speed:......FAST
Batch Size:............................50
Batch Interval:.......................0
Disconnect Interval.................6000
Data Conversion:...................No
Put Authority:........................NULL
Batch HeartBeat Interval:.......0
Short Retry Count:.................10
Short Retry Interval:..............60
Long Retry Count:..................999999999
Long Retry Interval:...............1200
I dont know why this fails like this, and this is our production queue with important info. The only way I get the queue back up is to STOP RESET and RESOLVE with COMMIT until the queue gets empty enough to resume normal sending.
I know this is on the other side, the remote queue side, and the company on the other side knows as much as I do, maybe less...
Please could you advise at to what steps I could take to prevent this from happening again, and perhaps suggestions I can give to the MQ Admin on the other side so we can get over the PROD Channel Failing!
Thanks so much!
Justin |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Sep 06, 2007 1:11 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
1) Increase the max depth of your transmission queue (or the space available to it if it's actually full instead of logically full)
2) Get the MQ admin the other side to increase the max depth and / or space available to the receiving queues
3) Get the developers the other side to develop (or redevelop) their application so it processes messages faster. Or spin up more instances of it when the system's under load. There are a number of ways they could detect this & responde automatically.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| insertcoin |
| Posted: Thu Sep 06, 2007 1:14 am Post subject: |
|
|
Acolyte
Joined: 07 Jun 2007
Posts: 61
Location: who knows!?
|
| Quote: |
| Or spin up more instances of it when the system's under load. There are a number of ways they could detect this & responde automatically. |
Could you advise me on how to do this?
Ill speak to the other Side and get them to increase the max depth!
Thanks Vitor!
editted:
Some more Info about the transmission Q:
Maximum Queue Depth:............100000
Maximum MessgaeLength:.........4194304
Shareability:.............................Shareable
Default Input Open Option:.........Shared
Message Delivery Sequence:.......Priority
Retention Interval:.....................99999999
Definition Types(greyed):...........Predefined
Distribution Lists:.......................Enabled
Maximum Depth Event: .............Enabled
High Depth Event:......................Disabled
High Depth Limit:.......................80
Low Depth Event:.......................Disabled
Low Depth Limit:........................20
Service Interval Event:...............None
Service Interval:........................999999999
Is there anything else that could affect the sender channel going down?
Thanks
Last edited by insertcoin on Thu Sep 06, 2007 1:22 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 06, 2007 1:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| insertcoin wrote: |
| Could you advise me on how to do this? |
1) Buy monitoring software
2) Use Depth triggering
3) Long running task with PCF commands (like 1 but cheaper and simpler)
4) Any combination of the above
IMHO your best short term fix would be to increase storage your side. It's under your control and it's wise to have adequate storage in a prod environment to deal with outages like network failures. It sounds from your description that this is happening frequently, implying your depth is too small for the load you're taking. As a rule of thumb, the xmitq should be able to hold 75% of the largest possible number of messages that could be sent in the longest possible outage allowed by the SLA.
This formula obviously yield a wide variety of thumb sizes.
I'm sure others will have differing views & other monitoring suggestions, all of which may be equally valid.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| insertcoin |
| Posted: Thu Sep 06, 2007 1:25 am Post subject: |
|
|
Acolyte
Joined: 07 Jun 2007
Posts: 61
Location: who knows!?
|
What a legend! Thanks Vitor!
Monitoring Software Suggestions:??
Money is no problem |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 06, 2007 1:26 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| insertcoin wrote: |
| Is there anything else that could affect the sender channel going down? |
Whatever it is, it's in the log of the receiver MCA. Look in that (or get the other side to).
This is another reason for increasing sender side xmitq depth. It covers you against any failure their side.
What's your message throughput? I notice xmitq maxdepth is 100000, how long does it take to hit that? It might give a clue to what's going on their side (though the most likely explaination for that message remains receiver queue full IMHO)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Sep 06, 2007 3:11 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| insertcoin wrote: |
What a legend! Thanks Vitor!
Monitoring Software Suggestions:??
Money is no problem
Is there anything else that could affect the sender channel going down? |
Have a look at the monitoring forum. You'll need to write up your requirements, shop around and get the best fit/value.
Anything else that could affect the sender channel going down...
How about making sure the receiving side has a DLQ defined for the qmgr?
This does not mean checking the existence of the SYSTEM.DEAD.LETTER.QUEUE but checking the properties of the qmgr:
dis qmgr deadq
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 06, 2007 3:29 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 06, 2007 3:32 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
How about making sure the receiving side has a DLQ defined for the qmgr?
|
Ooo... good point!
But would you be able to restart the channel in that circumstance? Would the RESOLVE with commit clear out the maladdressed message?
Never a pair of test queue managers around when you need them... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 06, 2007 4:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Vitor wrote: |
But would you be able to restart the channel in that circumstance? Would the RESOLVE with commit clear out the maladdressed message?
|
The answer would seem to be yes provided you reset the sequence numbers as well. The maladdressed message gets lost in the void between queue managers.....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Sep 06, 2007 4:24 am Post subject: Re: Sender Channel Overload |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| insertcoin wrote: |
The only way I get the queue back up is to STOP RESET and RESOLVE with COMMIT until the queue gets empty enough to resume normal sending.
|
Don't do this unless your log specifically tells you you have sequence # errors or the channel is in doubt. The reason you think it worked is because by the time you went thru the motions the other side's queue drained a bit and when you manually restarted the channel it was able to move some more messages.
The other side should have a DLQ. If they don't your channel hard stops when the destination q fills. If they do have a DLQ your channel will stop when the destination q AND the DLQ fills up.
Your sender channel is not failing because your XMITQ filled up, it stops because the other side had some problem, which we all are assuming is queues filling up. Until you verify this by seeing what their logs say be aware it might be something else causing the problem.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 06, 2007 4:31 am Post subject: Re: Sender Channel Overload |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| Until you verify this by seeing what their logs say be aware it might be something else causing the problem. |
A very good point and ties back to my earlier post, which remains valid despite what my little experiement has shown.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
