ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » MQ cluster channel slow

Post new topic  Reply to topic
 MQ cluster channel slow « View previous topic :: View next topic » 
Author Message
KIT_INC
PostPosted: Wed Dec 07, 2016 2:06 pm    Post subject: MQ cluster channel slow Reply with quote

Knight

Joined: 25 Aug 2006
Posts: 589

We have MQV75 Qmgrs and MQ V8 Qmgrs in a cluster. We have a need to send 500K (average size 12k btes) messages from a 75 Qmgr to a V8 Qmgr. So these messages are sent over the auto defined cluster sender channel between these 2 Servers. Because we are migrating from SSL to TLS. So we actually have 2 channels running at the same time (one using TLS cipher and one using SSL Cipher).
How ever what we got is only about 5 msg/sec. This was working over night and there is no other traffic competing with us
We check with Network support to see if the bottleneck is the network. They show us a chart showing that MQ is only using 300Kb per sec and a backup job that was started right after us was using 600Kb per sec. Their point is since the backup job can do 600Kb, it means that there is bandwidth available, why is MQ using only using half.
We check memory and CPU, the CPU usage is under 10%
We need some idea on why is MQ is driving the network. 5 messages per sec is very low. Because of the low transmit rate, we have thousands of messages sitting in the SCTQ waiting to be transmitted. So it is not the program that create the message.
All the chls are using IBM defaults except maxmsglength of 100M. Any suggestion on where we should start to look ?
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Dec 07, 2016 2:10 pm    Post subject: Re: MQ cluster channel slow Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

KIT_INC wrote:
We have MQV75 Qmgrs and MQ V8 Qmgrs in a cluster. We have a need to send 500K (average size 12k btes) messages from a 75 Qmgr to a V8 Qmgr. So these messages are sent over the auto defined cluster sender channel between these 2 Servers. Because we are migrating from SSL to TLS. So we actually have 2 channels running at the same time (one using TLS cipher and one using SSL Cipher).
How ever what we got is only about 5 msg/sec. This was working over night and there is no other traffic competing with us
We check with Network support to see if the bottleneck is the network. They show us a chart showing that MQ is only using 300Kb per sec and a backup job that was started right after us was using 600Kb per sec. Their point is since the backup job can do 600Kb, it means that there is bandwidth available, why is MQ using only using half.
We check memory and CPU, the CPU usage is under 10%
We need some idea on why is MQ is driving the network. 5 messages per sec is very low. Because of the low transmit rate, we have thousands of messages sitting in the SCTQ waiting to be transmitted. So it is not the program that create the message.
All the chls are using IBM defaults except maxmsglength of 100M. Any suggestion on where we should start to look ?


You should start to hunt for full queues on the destination queue manager.
A full queue is a very bad thing in a cluster. It slows everything down to a crawl...
The traffic that's competing with you is yourself.

Assuming persistent message and a channel using the DLQ:

  • Empty Queue
    • Channel agent puts message on queue
    • done

  • Full queue
    • Channel agent tries to put message on queue
    • Channel agent waits configured time and does short retries as configured
    • Channel agent waits configured time and does long retries as configured
    • Channel agent tries to put message on DLQ reason Q_FULL
    • If still not successful channel agent stops channel


Now maybe you understand why you're at 5 msgs per sec...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
KIT_INC
PostPosted: Wed Dec 07, 2016 8:16 pm    Post subject: Reply with quote

Knight

Joined: 25 Aug 2006
Posts: 589

There is no Q full situation at the target. We never had more than 100 messages on the target Queues which is a XMITQ. BTW the target Q is a remote Q sending data to an off site location. So there is actually no processing on the receiving end.
The message path looks like

QM1 -->cluster sender --> QM2 and QM3

QM2 --> sender channel --> offsite Qmgr
QM3 --> sender channel --> offsite Qmgr

QM1, QM2, QM3 are member of the same cluster.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Dec 08, 2016 1:13 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

How can the target queue be a xmit queue? The target queue is the local queue at the target QM.

Are these messages persistent? How fast are they produced and how often are they committed?

Test with a queue loading tool like MO71, QLOAD, or MA01.

Stop the channel, load the queue up, verify the queue looks normal, then start the channel and see how fast the transfer is.

Make sure the target queue (and xmit queues) have plenty of maxdepth.

Check the QM error log at each end for anything shown.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Dec 08, 2016 4:19 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Post the complete definition of the queue that the producing app opens on the qmgr on sending side.

Post the destination qliocsl definition at the receiving end of the channel.

Post the definition of the dead-letter-queue on the destination qmgr.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Dec 08, 2016 6:46 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

KIT_INC wrote:
There is no Q full situation at the target. We never had more than 100 messages on the target Queues which is a XMITQ. BTW the target Q is a remote Q sending data to an off site location. So there is actually no processing on the receiving end.
The message path looks like

QM1 -->cluster sender --> QM2 and QM3

QM2 --> sender channel --> offsite Qmgr
QM3 --> sender channel --> offsite Qmgr

QM1, QM2, QM3 are member of the same cluster.


How can you know unless you can view the offsite Qmgr?
Is any of the XMITQ's running full on QM2 or QM3?
What is the quality of the transmition between QM2 & QM3 and the off site qmgr?
What is the max depth on the XMITQ on QM2 / QM3?
What is the max depth on the Cluster XMITQ on the sender qmgr?

BTW Full does not necessarily mean 100% full.
If the channel sends packets of say 50 messages and the queue is at max-depth - 30, then the queue is full!

The full queue needs not be related to the current push of messages.
Any message in the channel for any full queue at the destination, whether that queue is clustered or not, will slow down the channel...

For non persistent messages, if the channel sequence numbers are out of order, that also means a slow down...


_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
KIT_INC
PostPosted: Thu Dec 08, 2016 8:33 am    Post subject: Reply with quote

Knight

Joined: 25 Aug 2006
Posts: 589

QM1,QM2,QM3 are all in cluster CLUS1

QM1 is the sending QMgr. We saw thousands of messages piling up on the SCTQ, so there should be nothing to do with the sending application which MQPUT persistent messages to TARGET_Q

The XMITQ(OFFQM2) on QM2 and QM3 never had curdepth bigger than 100. So no Q full situation at the receiving end.

All Qs has max Q depth 5000 or higher

Below are MQSC output

RUNMQSC QM1

DIS QC(TARGET_Q)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QCLUSTER)
ALTDATE(2016-11-12) ALTTIME(00.30.55)
CLUSDATE(2016-11-25) CLUSTER(CLUS1)
CLUSQMGR(QM3) CLUSQT(QREMOTE)
CLUSTIME(03.47.11) CLWLPRTY(0)
CLWLRANK(0) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) QMID(QM3_2015-09-02_01.40.04)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QCLUSTER)
ALTDATE(2016-11-11) ALTTIME(23.47.20)
CLUSDATE(2016-11-25) CLUSTER(CLUS1)
CLUSQMGR(QM2) CLUSQT(QREMOTE)
CLUSTIME(03.47.11) CLWLPRTY(0)
CLWLRANK(0) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) QMID(QM2_2015-09-02_01.06.2

You can see the 2x2 autodefined cluster sender which QM1 used to send messages to QM2 and QM3. Channel ending with T is using TLS Cipher and the other one is using SSL Cipher. All 4 channels were in running status at the time.

dis clusqmgr(QM2) channel
1 : dis clusqmgr(QM2) channel
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM2) CHANNEL(TO.QM2.CLUS1)
CLUSTER(CLUS1)
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM2) CHANNEL(TO.QM2.CLUS1T)
CLUSTER(CLUS1)

:
dis clusqmgr(QM3) channel
2 : dis clusqmgr(QM3) channel
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM3) CHANNEL(TO.QM3.CLUS1T)
CLUSTER(CLUS1)
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM3) CHANNEL(TO.QM3.CLUS1)
CLUSTER(CLUS1)

QM2 and QM3 receive messages from QM1 and forward them to an off site Qmgr (OFFQM)


RUNMQSC QM2

DIS QR(TARGET_Q)
1 : DIS QR(TARGET_Q)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QREMOTE)
ALTDATE(2016-11-11) ALTTIME(23.47.20)
CLUSNL( ) CLUSTER(CLUS1)
CLWLPRTY(0) CLWLRANK(0)
CUSTOM( ) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) RQMNAME(OFFQM)
RNAME(TARGET_Q) SCOPE(QMGR)
XMITQ(OFFQM1)


RUNMQSC QM3

DIS QR(TARGET_Q)
1 : DIS QR(TARGET_Q)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QREMOTE)
ALTDATE(2016-11-12) ALTTIME(00.30.55)
CLUSNL( ) CLUSTER(CLUS1)
CLWLPRTY(0) CLWLRANK(0)
CUSTOM( ) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) RQMNAME(OFFQM)
RNAME(TARGET_Q) SCOPE(QMGR)
XMITQ(OFFQM2)
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Dec 08, 2016 11:43 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

On QM2 and QM3 run following query

Code:
dis ql(*) where ( currdepth gt 0) curdepth maxdepth


Any number close to maxdepth will need investigating...
Check the logs on QM1, QM2, QM3 for any entries about the dynamically defined cluster channel... and show what you find...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
JosephGramig
PostPosted: Fri Dec 09, 2016 6:24 am    Post subject: Reply with quote

Grand Master

Joined: 09 Feb 2006
Posts: 1230
Location: Gold Coast of Florida, USA

I had an issue like this between AIX and z/OS Qmgrs a long time ago.

We were told to check the tcp_nodelayack setting:

no -a | grep tcp_nodelayack

And if it was not set to 1 (0 being the other setting), then we needed to set it to 1.

I have been told newer versions of MQ don't need this because it is specified in the code now (but I set it any way).
Back to top
View user's profile Send private message AIM Address
PeterPotkay
PostPosted: Mon Dec 12, 2016 6:12 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

The definitions of the CLUSRCVR channels on QM2 and QM3 are relevant. Perhaps you inadvertently have a parameter that can effect throughput set wrong.

To absolve the putting app, define a test local queue on QM2 and QM3 that is clustered in the cluster that uses the TLS channels. Connect to QM1 and send a few thousand test messages. Do they move from Qm1 to QM2/QM3 as expected, or also slowly?

Or try with a plain SNDR/RCVR channel pair between QM1 and QM2 as a test. Try with and without TLS. Did you create keys with an insanely high bit count?

To absolve MQ, try sftping a large file from Server1 to Server2 or Server3. Does this non MQ transfer of data go fast, or slow?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » Clustering » MQ cluster channel slow
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.