ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » Cluster Sender SUBSTATE(SEND)

Post new topic  Reply to topic
 Cluster Sender SUBSTATE(SEND) « View previous topic :: View next topic » 
Author Message
vsathyan
PostPosted: Tue Aug 11, 2015 2:15 am    Post subject: Cluster Sender SUBSTATE(SEND) Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

We are facing a strange issue in production. The cluster sender is stuck at SUBSTATE(SEND) for one cluster sender on a queue manager. Other cluster sender channels in the queue manager are at SUBSTATE(MQGET).

On the receiving side, we see the channel SUBSTATE(RECEIVE) and the status is RUNNING.

The issue we are facing is messages are not flowing from the cluster transmission queue at the sender to the queues in the receiving queue manager.

Below are the output of commands
1. Sending side cluster sender details

DIS CHS(TO.QMGR)
6 : DIS CHS(TO.QMGR)
AMQ8417: Display Channel Status details.
CHANNEL(TO.QMGR) CHLTYPE(CLUSSDR)
CONNAME(1.2.3.4(23556)) CURRENT
RQMNAME(QMGR) STATUS(RUNNING)
SUBSTATE(SEND)
XMITQ(SYSTEM.CLUSTER.TRANSMIT.TO.QMGR)

2. Receiver side cluster receiver details

DIS CHS(TO.QMGR) WHERE(RQMNAME EQ QMGR2)
7 : DIS CHS(TO.QMGR) WHERE(RQMNAME EQ QMGR2)
AMQ8417: Display Channel Status details.
CHANNEL(TO.QMGR) CHLTYPE(CLUSRCVR)
CONNAME(10.178.52.235) CURRENT
RQMNAME(QMGR2) STATUS(RUNNING)
SUBSTATE(RECEIVE)

-------------------------------------------------------

Messages not flowing from cluster xmitq SYSTEM.CLUSTER.TRANSMIT.TO.QMGR in QMGR2 to QMGR.

Any suggestions, advise? Thanks in advance.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004


Last edited by vsathyan on Tue Aug 11, 2015 8:04 pm; edited 1 time in total
Back to top
View user's profile Send private message
exerk
PostPosted: Tue Aug 11, 2015 3:37 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

In regard to the channel, what's in the logs of the sending/receiving queue managers?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
vsathyan
PostPosted: Tue Aug 11, 2015 3:52 am    Post subject: Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

Nothing reported in the error logs of both queue managers...

One thing we observed - the message size is around 2.6MB and there are 2000+ messages (approximately 4.5GB).
Also, the messages have to be routed from US to East Asia. So, thinking would there be any network latency.

As a work around, we tried to push messages to the cluster transmit queue in other cluster queue manager. Same result.

We also created a new transmit queue (point to point with sender-receiver pair) and tried to move some messages to this new xmitq. Same result. In this case, the point to point sender channel also reached SUBSTATE(SEND).

We reduced the channel batchsz to 1 and currently, it is processing a bit faster..

We are continuing to monitor...

Any advise, please reply.
Thank you.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004
Back to top
View user's profile Send private message
exerk
PostPosted: Tue Aug 11, 2015 4:41 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

vsathyan wrote:
...the message size is around 2.6MB and there are 2000+ messages (approximately 4.5GB)...

For large messages it's always a good idea to have a low batch number - less to roll back.

Get your network admins involved, maybe they can see where the bottle-neck is if it's network related.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Aug 11, 2015 5:21 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

You did not specify the version of your qmgrs. Nor did you specify if any compression was being done on the channel.

For messages needing to go from one continent to the other, any time spent is a compression routine is time well spent. It will reduce the size of the message on the wire and minimize band width usage....

Keep the channel batch size at 1. Make sure the receiving channel has COMPMSG(ANY,NONE) set and set the appropriate order of compression algorithms on the sender.

And please be at least at level 7.0 if not above...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
vsathyan
PostPosted: Tue Aug 11, 2015 6:07 am    Post subject: Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

hi fjb_saper,
My bad, that I missed to update the details here.

The queue managers are running on version 7.5.0.5 on Enterprise Linux.
Currently, as mentioned in my earlier post, we have set the batchsz to 1 already on the channels.

No compression method used. Message backlog is getting reduced.

Verified the size of the queue manager transaction logs and it is greater than 4GB. Just FYI. Thank you.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Aug 11, 2015 6:45 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

You will probably get better throughput with message compression. But as stated, you need to change / review the channels settings at both ends for that.

How much improvement you may see with message compression is largely dependent on compression ratio and network quality:

The higher the compression ratio => the better the throughput
The worse the network quality, the smaller the message on the wire => the better the chances for a successful transmission.


_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
bruce2359
PostPosted: Tue Aug 11, 2015 9:24 am    Post subject: Re: Cluster Sender SUBSTATE(SEND) Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

vsathyan wrote:
The cluster sender is stuck at SUBSTATE(SEND) for one cluster sender on a queue manager. Other cluster sender channels in the queue manager are at SUBSTATE(MQGET).

On the receiving side, we see the channel SUBSTATE(RECEIVE) and the status is RUNNING.

These are normal channel states and substates, but at different times.

vsathyan wrote:
The issue we are facing is messages are not flowing

Are these new channels?

vsathyan wrote:
Nothing reported in the error logs of both queue managers...

Nothing at all in either qmgr error logs? Not even channel starting or ending?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
vsathyan
PostPosted: Tue Aug 11, 2015 7:45 pm    Post subject: Reply with quote

Centurion

Joined: 10 Mar 2014
Posts: 121

The issue is currently resolved.
The problem was the messages were big. The channel was auto defined cluster sender with batchsz(50). The channel was sending messages in batch of 50 (50 * 2.6MB = ~125MB) per batch.

The cluster sender was at SUBSTATE(SEND) because the channel was sending a batch of 125MB messages and would receive EOB after the batch was complete. By then the next batch would start and it would be in SUBSTATE(SEND) again.

We had a deployment last week end, and the publishing application was down for around 40 hours. Probably they are processing the backlog in huge quantity. Currently, the messages arriving on the xmitq is in KBs, and once the messages get transferred, the cluster sender channel SUBSTATE is returning to MQGET and is normal.

Thanks for all your help and advise.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Thu Aug 13, 2015 4:03 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

SEND / RECEIVE substates indicate that TCP send() / receive() were in progress for the Message Channel Agents to physically transfer the data over the network.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » Clustering » Cluster Sender SUBSTATE(SEND)
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.