| Author |
Message
|
| sarathmattam |
 Posted: Mon Jul 15, 2013 2:47 am Post subject: Slowness in Transferring Messages |
 |
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Dear All,
I am here to get an advice on the issue and the remedy that we have taken (at least temporarily) mentioned below
Issue :
Slowness in Transferring the messages. We have 2 queue managers QM1 and QM2 on different network, geographically located at a distance of around 200 KM. We are using distributed queuing and a single channel connecting these two queue managers. An application writes messages to a remote queue, QM1.RM(on QM1) and is expected to be delivered to QM2.LQ (on QM2).The average size of the message being passed through the channel is 2 MB. For security reason ( as per the network team), we have a network switch in place which would disconnect the networks intermittently (say if its connected for 5 min, then it would be disconnected for the next 5 min). Messages are being written to the queue at a speed of 1 msg/6 sec. We have enough band width (as per network team based on their analysis) to send the messages out. But, we are experiencing a slowness in transferring the messages.
Solution Proposed :
1. Delete the Remote queue, QM1.RM. Create a local queue with the same name so that no changes are required at the application end
2. Create remote queues pointing to the same destination queue
3. Create additional channels between QM1 and QM2
4. Configure Trigger on QM1.RM to distribute messages to the remote queues. ( For this we are making use of the QLOAD utility)
Kindly let me know how practical is this approach.
Regards,
Sarath |
|
| Back to top |
|
 |
| sumit |
| Posted: Mon Jul 15, 2013 3:47 am Post subject: Re: Slowness in Transferring Messages |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| sarathmattam wrote: |
| Messages are being written to the queue at a speed of 1 msg/6 sec. |
Transfer of messages from one qmgr to another qmgr is quick. Generally, the application which puts message to a queue consumes more time and it appears that MQ is taking time to process them.
Can you find out how much time does your application take to put one complete 2 MB message in a queue and then see how much time did qmgr take to send message from QM1 to QM2?
_________________
Regards
Sumit |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Wed Jul 17, 2013 2:04 am Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Dear Sumit,
I feel the application is writing the messages with a better pace that MQ transferring it .. Because i can see a pile of messages in the transmit queue ..
Is there any other parameters to be checked ?
Regards,
Sarath |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Jul 17, 2013 3:19 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| sarathmattam wrote: |
| ...I feel the application is writing the messages with a better pace that MQ transferring it... |
Evidence is a better indicator than a 'feeling', because...
| sarathmattam wrote: |
| ...Because i can see a pile of messages in the transmit queue... |
...the pile of messages can be uncommitted.
Consider: An application puts 10,000 messages to the queue and waits an hour to commit them - because WMQ cannot transmit them until they are committed, the 'feeling' is that WMQ must be slow, and this is reinforced because the application is capable of 1000 PUTS-per-second; think on that.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 17, 2013 5:44 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You might also need to check if any queue on the destination qmgr is full.
This may dramatically slow down the channel... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 17, 2013 7:49 am Post subject: Re: Slowness in Transferring Messages |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| sarathmattam wrote: |
Solution Proposed :
1. Delete the Remote queue, QM1.RM. Create a local queue with the same name so that no changes are required at the application end
2. Create remote queues pointing to the same destination queue
3. Create additional channels between QM1 and QM2
4. Configure Trigger on QM1.RM to distribute messages to the remote queues. ( For this we are making use of the QLOAD utility)
Kindly let me know how practical is this approach.
|
Why do you think this woudl get messages any quicker to the final destination? You've added extra steps and potential failure points.
| Quote: |
Messages are being written to the queue at a speed of 1 msg/6 sec. We have enough band width (as per network team based on their analysis) to send the messages out. But, we are experiencing a slowness in transferring the messages.
|
What exactly does slowness in transferring messages mean?
Maybe you have your channel retry parameters set incorrectly and the SNDR channel takes several minutes to recover every 5 minutes when the connection is purposely severed by your network team.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Jul 17, 2013 8:01 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Disconnecting the network every 5 mins is fairly odd to say the least.
However, ignoring that, change your batch size on the channel to a smaller value. Make sure TCP keepalive is set in the qm.ini file, lower the channel heartbeat interval value, reduce the long retry interval value.
(Make one change at a time to see the effect). |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 17, 2013 11:21 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
At 5 minute intervals I think Short Retry Interval also needs to be looked at on the SNDR channel.
With frequent network disruptions the AdoptNewMCA settings on the receiving side will be important.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Sat Jul 20, 2013 12:42 am Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Hi exerk, I totally agree with you .. As we have network switch in place which disconnects the network at specified time, the BATCHSIZE factor was already taken care of. Currently it is kept at a value as low as 5. As this would make MQ to do more commits, this could be one of the factors causing slowness. But, am afraid that if i keep a high value for Batch Size and if the network disconnects just before the Batch size is reached, all the messages would be rolled back  .. |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Sat Jul 20, 2013 12:52 am Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
@ fjb_saper : The queue at destination is not full or there is not pile up of messages at the target end. As and when the messages reach the target side, its getting comsumed by the target application.
@PeterPotkay : This is rational behind that thought .More the number of channels, more the number of messages it can transfer ( provided you have enough band width) . As per the network team, only 1/5th of the available band with is being utilized. This figures prompted me to think of such an approch of increasing the number of channels.
@ Zpat : Me too agree that its odd. but thats the requirement and dnt find a chance to alter this .. Now, Batch size and short retry interval is also modified and smaller values (compared to the default values) are configured now. I will look at the other parameters you mentioned and will let u know. |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Sat Jul 20, 2013 12:56 am Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
We have tested the approach mentioned in my first post in our staging environment using the QLOAD utility. It shows very positive results. We have tested with 42 messages of 2 MB each. The new approach showed almost 45-50% improvement in the time taken. But, what I need to know from experts here is that
1. Is this a good (fairly good) method ?
2. Can any one please suggest a better method with out using an external tool like QLOAD ? |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Mon Jul 22, 2013 10:32 pm Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Hi All,
Please share your thoughts ..
Regards,
Sarath |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 22, 2013 11:25 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| sarathmattam wrote: |
Hi All,
Please share your thoughts ..
Regards,
Sarath |
My thoughts are: sort out the network disruption; as pointed out in a previous post, you are adding complexity and therefore points of failure.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Tue Jul 23, 2013 2:35 am Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Hi exerk,
Unfortunately the decision to change network switch mechanism is not in my hand. I can only suggest, but seems thats not going to work out .. So what am asking for is a possibility to achieve higher through put keeping the switch in place.
I have tried the option of Pipelining of Channels by configuring the same qm.ini file. But, that also dint give me a remarkable improvement in performance.
Regards,
Sarath |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Tue Jul 23, 2013 4:56 am Post subject: Re: Slowness in Transferring Messages |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
| sarathmattam wrote: |
.... We have enough band width .....
|
Hi
what exactly is "enough" ? If it takes you 6 seconds for a message to be transfered at an average of 2MB size, it sounds like something around 2Mbit line or a bit more. What bandwidth do you have available? And is it all available for MQ ?!? (i know networking guys can limit usage somehow.....)
i would go for batchsize 1 for large messages. i would also check channel retry parameters on the receiving end (and set no retry). make sure the receiving qmgr has a DLQ assigned.
i would also do some tests with persistent / non persistent messages to check if disk i/o could be a problem.
_________________
Regards, Butcher |
|
| Back to top |
|
|
|
|
