| Author |
Message
|
| KIT_INC |
 Posted: Wed Dec 07, 2016 2:06 pm Post subject: MQ cluster channel slow |
 |
|
Knight
Joined: 25 Aug 2006
Posts: 589
|
We have MQV75 Qmgrs and MQ V8 Qmgrs in a cluster. We have a need to send 500K (average size 12k btes) messages from a 75 Qmgr to a V8 Qmgr. So these messages are sent over the auto defined cluster sender channel between these 2 Servers. Because we are migrating from SSL to TLS. So we actually have 2 channels running at the same time (one using TLS cipher and one using SSL Cipher).
How ever what we got is only about 5 msg/sec. This was working over night and there is no other traffic competing with us
We check with Network support to see if the bottleneck is the network. They show us a chart showing that MQ is only using 300Kb per sec and a backup job that was started right after us was using 600Kb per sec. Their point is since the backup job can do 600Kb, it means that there is bandwidth available, why is MQ using only using half.
We check memory and CPU, the CPU usage is under 10%
We need some idea on why is MQ is driving the network. 5 messages per sec is very low. Because of the low transmit rate, we have thousands of messages sitting in the SCTQ waiting to be transmitted. So it is not the program that create the message.
All the chls are using IBM defaults except maxmsglength of 100M. Any suggestion on where we should start to look ? |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Dec 07, 2016 2:10 pm Post subject: Re: MQ cluster channel slow |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| KIT_INC wrote: |
We have MQV75 Qmgrs and MQ V8 Qmgrs in a cluster. We have a need to send 500K (average size 12k btes) messages from a 75 Qmgr to a V8 Qmgr. So these messages are sent over the auto defined cluster sender channel between these 2 Servers. Because we are migrating from SSL to TLS. So we actually have 2 channels running at the same time (one using TLS cipher and one using SSL Cipher).
How ever what we got is only about 5 msg/sec. This was working over night and there is no other traffic competing with us
We check with Network support to see if the bottleneck is the network. They show us a chart showing that MQ is only using 300Kb per sec and a backup job that was started right after us was using 600Kb per sec. Their point is since the backup job can do 600Kb, it means that there is bandwidth available, why is MQ using only using half.
We check memory and CPU, the CPU usage is under 10%
We need some idea on why is MQ is driving the network. 5 messages per sec is very low. Because of the low transmit rate, we have thousands of messages sitting in the SCTQ waiting to be transmitted. So it is not the program that create the message.
All the chls are using IBM defaults except maxmsglength of 100M. Any suggestion on where we should start to look ? |
You should start to hunt for full queues on the destination queue manager.
A full queue is a very bad thing in a cluster. It slows everything down to a crawl...
The traffic that's competing with you is yourself.
Assuming persistent message and a channel using the DLQ:
- Empty Queue
- Channel agent puts message on queue
- done
- Full queue
- Channel agent tries to put message on queue
- Channel agent waits configured time and does short retries as configured
- Channel agent waits configured time and does long retries as configured
- Channel agent tries to put message on DLQ reason Q_FULL
- If still not successful channel agent stops channel
Now maybe you understand why you're at 5 msgs per sec... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| KIT_INC |
| Posted: Wed Dec 07, 2016 8:16 pm Post subject: |
|
|
Knight
Joined: 25 Aug 2006
Posts: 589
|
There is no Q full situation at the target. We never had more than 100 messages on the target Queues which is a XMITQ. BTW the target Q is a remote Q sending data to an off site location. So there is actually no processing on the receiving end.
The message path looks like
QM1 -->cluster sender --> QM2 and QM3
QM2 --> sender channel --> offsite Qmgr
QM3 --> sender channel --> offsite Qmgr
QM1, QM2, QM3 are member of the same cluster. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Dec 08, 2016 1:13 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
How can the target queue be a xmit queue? The target queue is the local queue at the target QM.
Are these messages persistent? How fast are they produced and how often are they committed?
Test with a queue loading tool like MO71, QLOAD, or MA01.
Stop the channel, load the queue up, verify the queue looks normal, then start the channel and see how fast the transfer is.
Make sure the target queue (and xmit queues) have plenty of maxdepth.
Check the QM error log at each end for anything shown.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Dec 08, 2016 4:19 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Post the complete definition of the queue that the producing app opens on the qmgr on sending side.
Post the destination qliocsl definition at the receiving end of the channel.
Post the definition of the dead-letter-queue on the destination qmgr.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 08, 2016 6:46 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| KIT_INC wrote: |
There is no Q full situation at the target. We never had more than 100 messages on the target Queues which is a XMITQ. BTW the target Q is a remote Q sending data to an off site location. So there is actually no processing on the receiving end.
The message path looks like
QM1 -->cluster sender --> QM2 and QM3
QM2 --> sender channel --> offsite Qmgr
QM3 --> sender channel --> offsite Qmgr
QM1, QM2, QM3 are member of the same cluster. |
How can you know unless you can view the offsite Qmgr?
Is any of the XMITQ's running full on QM2 or QM3?
What is the quality of the transmition between QM2 & QM3 and the off site qmgr?
What is the max depth on the XMITQ on QM2 / QM3?
What is the max depth on the Cluster XMITQ on the sender qmgr?
BTW Full does not necessarily mean 100% full.
If the channel sends packets of say 50 messages and the queue is at max-depth - 30, then the queue is full!
The full queue needs not be related to the current push of messages.
Any message in the channel for any full queue at the destination, whether that queue is clustered or not, will slow down the channel...
For non persistent messages, if the channel sequence numbers are out of order, that also means a slow down...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| KIT_INC |
| Posted: Thu Dec 08, 2016 8:33 am Post subject: |
|
|
Knight
Joined: 25 Aug 2006
Posts: 589
|
QM1,QM2,QM3 are all in cluster CLUS1
QM1 is the sending QMgr. We saw thousands of messages piling up on the SCTQ, so there should be nothing to do with the sending application which MQPUT persistent messages to TARGET_Q
The XMITQ(OFFQM2) on QM2 and QM3 never had curdepth bigger than 100. So no Q full situation at the receiving end.
All Qs has max Q depth 5000 or higher
Below are MQSC output
RUNMQSC QM1
DIS QC(TARGET_Q)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QCLUSTER)
ALTDATE(2016-11-12) ALTTIME(00.30.55)
CLUSDATE(2016-11-25) CLUSTER(CLUS1)
CLUSQMGR(QM3) CLUSQT(QREMOTE)
CLUSTIME(03.47.11) CLWLPRTY(0)
CLWLRANK(0) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) QMID(QM3_2015-09-02_01.40.04)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QCLUSTER)
ALTDATE(2016-11-11) ALTTIME(23.47.20)
CLUSDATE(2016-11-25) CLUSTER(CLUS1)
CLUSQMGR(QM2) CLUSQT(QREMOTE)
CLUSTIME(03.47.11) CLWLPRTY(0)
CLWLRANK(0) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) QMID(QM2_2015-09-02_01.06.2
You can see the 2x2 autodefined cluster sender which QM1 used to send messages to QM2 and QM3. Channel ending with T is using TLS Cipher and the other one is using SSL Cipher. All 4 channels were in running status at the time.
dis clusqmgr(QM2) channel
1 : dis clusqmgr(QM2) channel
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM2) CHANNEL(TO.QM2.CLUS1)
CLUSTER(CLUS1)
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM2) CHANNEL(TO.QM2.CLUS1T)
CLUSTER(CLUS1)
:
dis clusqmgr(QM3) channel
2 : dis clusqmgr(QM3) channel
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM3) CHANNEL(TO.QM3.CLUS1T)
CLUSTER(CLUS1)
AMQ8441: Display Cluster Queue Manager details.
CLUSQMGR(QM3) CHANNEL(TO.QM3.CLUS1)
CLUSTER(CLUS1)
QM2 and QM3 receive messages from QM1 and forward them to an off site Qmgr (OFFQM)
RUNMQSC QM2
DIS QR(TARGET_Q)
1 : DIS QR(TARGET_Q)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QREMOTE)
ALTDATE(2016-11-11) ALTTIME(23.47.20)
CLUSNL( ) CLUSTER(CLUS1)
CLWLPRTY(0) CLWLRANK(0)
CUSTOM( ) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) RQMNAME(OFFQM)
RNAME(TARGET_Q) SCOPE(QMGR)
XMITQ(OFFQM1)
RUNMQSC QM3
DIS QR(TARGET_Q)
1 : DIS QR(TARGET_Q)
AMQ8409: Display Queue details.
QUEUE(TARGET_Q) TYPE(QREMOTE)
ALTDATE(2016-11-12) ALTTIME(00.30.55)
CLUSNL( ) CLUSTER(CLUS1)
CLWLPRTY(0) CLWLRANK(0)
CUSTOM( ) DEFBIND(NOTFIXED)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DESCR( )
PUT(ENABLED) RQMNAME(OFFQM)
RNAME(TARGET_Q) SCOPE(QMGR)
XMITQ(OFFQM2) |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 08, 2016 11:43 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
On QM2 and QM3 run following query
| Code: |
| dis ql(*) where ( currdepth gt 0) curdepth maxdepth |
Any number close to maxdepth will need investigating...
Check the logs on QM1, QM2, QM3 for any entries about the dynamically defined cluster channel... and show what you find...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Fri Dec 09, 2016 6:24 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
I had an issue like this between AIX and z/OS Qmgrs a long time ago.
We were told to check the tcp_nodelayack setting:
no -a | grep tcp_nodelayack
And if it was not set to 1 (0 being the other setting), then we needed to set it to 1.
I have been told newer versions of MQ don't need this because it is specified in the code now (but I set it any way). |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Dec 12, 2016 6:12 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
The definitions of the CLUSRCVR channels on QM2 and QM3 are relevant. Perhaps you inadvertently have a parameter that can effect throughput set wrong.
To absolve the putting app, define a test local queue on QM2 and QM3 that is clustered in the cluster that uses the TLS channels. Connect to QM1 and send a few thousand test messages. Do they move from Qm1 to QM2/QM3 as expected, or also slowly?
Or try with a plain SNDR/RCVR channel pair between QM1 and QM2 as a test. Try with and without TLS. Did you create keys with an insanely high bit count?
To absolve MQ, try sftping a large file from Server1 to Server2 or Server3. Does this non MQ transfer of data go fast, or slow?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
