| Author |
Message
|
| vignesh86 |
 Posted: Thu Sep 20, 2012 4:38 am Post subject: MQ PERFORMANCE ISSUES |
 |
|
Apprentice
Joined: 23 Dec 2010
Posts: 27
|
Hi Guys,
Currently we have rolled one project in production in which we are facing performance and timeout issues.
In production, application team has set 7 secs timeout interval from the time
1. Application put the request message in the request queue
2. then message passing through integration layer queue manager and then passes to broker queue manager for transformation.
3. Request is transported to receiver application side.
4. then message again passing through integration layer queue manager and then passes to broker queue manager for transformation.
5. Response sent back to the source application.
We faced frequent and inconsistent timeout issues in production.
Our production team has analyzed and found that
channel from Integration layer queue manager to source system
and channel from integration layer queue manager to receiver system is totally taking 4 secs.
Integration layer to broker -- 0.6 secs
Load : 60 msgs per day and size is in kb
For this issue, they provided below details :
Issue Analysis:
"The load on the source and receiver side mq is very less, so the channels go into INACTIVE state after a disconnect interval of 300 sec. The problem is that for some reason the time consumed to start a channel once the message is received on the transmission queue is high. This is the cause of the delay in transmission of messages even though they are simple bounce of pop messages."
Conclusion:
The time taken with Integration layer will always be around ~1 sec if the channel is in INACTIVE state.
The time taken to initiate the channel from INACTIVE state to RUNNING state takes about ~1 sec to ~1.5 sec. The thing to note here is that the channels are SSL enabled and it takes time to do the handshake after the channel has gone into INACTIVE state.
Suggestions:
1. Increase the timeout period.
2. Try and see the performance by removing the SSL from the channels.
3. Check what’s the frequency of messages coming to ISIS and based on the same modify the disconnect interval period.
My question here is " when channels goes to inactive state and after a msg has come whether it requires 2 secs time to process the msg through a channel" |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Sep 20, 2012 4:52 am Post subject: Re: MQ PERFORMANCE ISSUES |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| vignesh86 wrote: |
Suggestions:
1. Increase the timeout period.
2. Try and see the performance by removing the SSL from the channels.
3. Check what’s the frequency of messages coming to ISIS and based on the same modify the disconnect interval period.
|
1. Do you mean channel disconnect interval? If you message arrival rate is every 400 seconds, then set disconnect interval to 500 seconds. This will keep the channel from going inactive between messages.
2. You should have tested the channel without SSL to validate that the channel was behaving well before adding SSL.
What errors did you see in the errors file related to this channel?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| vignesh86 |
| Posted: Thu Sep 20, 2012 5:37 am Post subject: |
|
|
Apprentice
Joined: 23 Dec 2010
Posts: 27
|
In Integration layer production, all channels are ssl enabled.
Ans we follow the same configuration for all the channels.
There are also other connections between ISIS to other applications which takes 0.06 secs to process the request and response.Only difference is that load will be high to such application queue managers.
Then why there is a delay between isis to these 2 mq queue managers alone.Both are using MQ 7.0 and they are newly created application queue managers and load will be very less.
in these 7 secs timeout:
there are totally 8 legs involved.
Only these 2 legs taking 4 secs for transporting.
All other 6 legs taking less than 2-4 secs including transformation in broker.
Can you please let me know whether 2 secs is the time taken for channel to be running status from inactive mode and process the message?
Any other tuning that needs to be done in mq level |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 20, 2012 6:11 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Did this new application behave similarly in TEST and QA?
Is this a new queue manager?
Is the server/lpar sufficiently provisioned (CPU, RAM)?
Are you working with the platform and o/s sysadmins to identify bottlenecks?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 20, 2012 6:20 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| vignesh86 wrote: |
in these 7 secs timeout:
there are totally 8 legs involved.
Only these 2 legs taking 4 secs for transporting.
All other 6 legs taking less than 2-4 secs including transformation in broker.
Can you please let me know whether 2 secs is the time taken for channel to be running status from inactive mode and process the message?
Any other tuning that needs to be done in mq level |
What is timing out in 7 seconds? The channels? What exactly?
1 or 2 seconds is not unreasonable for channels to come to RUNNING state. Channel problems/errors are logged in the AMQERR01.LOG. Did you look for channel errors?
How long it takes to process the message is not an MQ issue. Are databases involved in your multi-legged application?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Sep 20, 2012 7:18 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Obviously 5 mins to channel inactive is way too short for this application. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vignesh86 |
| Posted: Thu Sep 20, 2012 8:26 pm Post subject: |
|
|
Apprentice
Joined: 23 Dec 2010
Posts: 27
|
@ bruce
Yes this is a new queue manager, the server details are as below
MQ Version :7.0.1.6
OS :Linux SSDZIICBISL0001 2.6.18-164.el5 #1 x86_64 x86_64 x86_64 GNU/Linux
gsk kit : gsk7bas-7.0-4.33/gsk7bas64-7.0-4.33
The Channel disconnect interval is 600 Secs.
There is no channel errors in logs..
There is a broker database involved in this flow but the broker transformation takes only 0.6 secs.
Just for information, these online messages will flows through encrypted channels from source to destination.Whether this could make a difference? |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu Sep 20, 2012 9:22 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
please check the BATCHINT / BATCHSZ values on the channels...maybe something is not set properly here.
another option is just to remove encryption and then test again.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| vignesh86 |
| Posted: Thu Sep 20, 2012 9:26 pm Post subject: |
|
|
Apprentice
Joined: 23 Dec 2010
Posts: 27
|
@saper
Currently, the load is less.. as only 60 users are doing verification testing.
By next End of next month, there should be total of 600 users who will start verification testing per day so 5mins shud be ok rite?
@ bruce : timeout 7 secs is set in application mq properties file.
7 secs is the total time required to place the request in the remote queue of source queue manager and after flowing through all 8 legs. The response should be received in the local response queue.
Another info is that " Currently there are 60 messages are tested per day.out of that average of 57 messages are successful"If we check the 3 messages timeout , the lag is in that 2 legs.
@butcher : BATCHINT and BATCHHB is 0. |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Sep 21, 2012 12:46 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| vignesh86 wrote: |
| 5mins shud be ok rite? |
As a 'standard' I usually set channel disconnect times at 1 hour, so no, I think 5 minutes is way to low if you consider that at 5 minutes for a disconnect interval the channel could potentially be stopping and starting 10 times within an hour, which is inefficient use of resource as far as I'm concerned. What are the technical reasons for setting the interval so low?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| vignesh86 |
| Posted: Sat Sep 22, 2012 3:22 am Post subject: |
|
|
Apprentice
Joined: 23 Dec 2010
Posts: 27
|
Thanks for the suggestions.
Let me increase the DISCONNECT INTERVAL AND HEART BEAT INTERVAL value for this issue in test environment and will let you know whether this is resolving the timeout issue.
Still if you have any more suggestions for improvement.. pls post it thanks.... |
|
| Back to top |
|
|
| rgw47 |
| Posted: Tue Sep 25, 2012 6:06 am Post subject: MQ reporting |
|
|
Newbie
Joined: 25 Sep 2012
Posts: 1
|
Newbie here..We have sas..mxg..and are cutting all the smf rec's needed..hopefully..
Does anyone out here have sample/key/areas to monitor..on a daily/weekly basis..and yes we have looked at many options..$$$ or need to load down client on PC..but we are locked down so that option is out as well..HANDS TIED..so looking/hoping for sample SMF/SAS coding to to satrt with and the key areas to keep a watch on.. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Sep 25, 2012 7:26 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Leaving aside the channel starting time - are you sure the receiving application is waiting for the message arrival - with a long-running MQGET (using the MQGMO_WAIT interval feature)?
Quite often latency is caused by applications "polling" the queue at intervals (i.e. repeatedly issuing MQGETs against empty queues) because the developers have failed to understand how to program the MQI properly. |
|
| Back to top |
|
|
|
|
