| Author |
Message
|
| J.D |
 Posted: Thu Jan 06, 2011 4:26 pm Post subject: Throughput Degradation |
 |
|
Voyager
Joined: 18 Dec 2009
Posts: 92
Location: United States
|
Hi,
For a new project, my developers are testing request/response application which is already in production but in different way. The response time is taking longer when number of threads are increased on sending application.
No.of Threads Response Time (Seconds)
1 1.4
10 7.8
50 38
Sending App <--> Front End MQ <--> Back End MQ <--> Receiving App
All the 50 threads are running on single machine and connecting to MQ Server using the same SVRCONN channel and port. There are no issues with the backend application in processing these messages. Both MQ Client and Server are running on Windows 2003 with v6.0.2.x. I would like to know what has to be changed to improve the response time.
_________________
IBM WebSphere MQ & WAS Administrator |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Jan 06, 2011 4:34 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| ...but in different way. |
What different way?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jan 06, 2011 5:00 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So if you have the sending app producing 100 messages a minute, and the receiving app producing 100 messages a minute... and then you increase the sending app to produce 10,000 messages a minute...
What needs to change to make sure that you still get 10,000 messages a minute back? |
|
| Back to top |
|
|
| J.D |
| Posted: Fri Jan 07, 2011 10:25 am Post subject: |
|
|
Voyager
Joined: 18 Dec 2009
Posts: 92
Location: United States
|
Receiving app is processing 30K req/rsp messages per hour in production and test environment is also capable of handling the same number.
Mine is a retail client and we have around 150 stores across the country. In each store there are around 50-100 registers to ring the transactions. Each register has its own MQ client installed and a messaging component. Now we are doing POC for a new project where around 50 Mobile devices will be handed over to sales persons so that they can ring the transactions from any where in store which means customer need not have to come to register. All the 50 Mobile devices connect to one single machine and from there transactions come to MQ.
50 Mobile Devices <--> Sending App on single machine (Messaging component built using C++) <--> MQ Servers <--> Receiving App
There is no degradation in performance when messages are sent from single mobile device but there is huge difference when msgs sent from 50 devices at same time. I would like to know what can be done on MQ server to improve the performance.
_________________
IBM WebSphere MQ & WAS Administrator |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 07, 2011 10:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Why do you think it's an MQ issue? Maybe the work is stacked up inside the sending application and not in MQ.
What is the relationship between the number of threads in the sending application and the number of MQCONNS in the sending application?
What is the nature of the MQ connection between the sending application and the queue manager? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jan 07, 2011 10:54 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| I would like to know what can be done on MQ server to improve the performance. |
Short of guessing, where is the bottleneck on the server box?
I'm guessing that the hardware is under-provisioned. If that's the case, then two possibilities come to mind:
1) add more cpu processor power
2) add more RAM
What else does the app do besides get/put messages? Is there a DB involved? Is I/O the bottleneck?
What kind of server hardware?
Number of cpus?
How much RAM?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 07, 2011 11:18 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I consider it more likely that the sending app has a single MQCONN that it is trying to send all of the work through, than that the server hosting the qmgr is underpowered.
But that is just a guess. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jan 07, 2011 11:29 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'm flipping a coin right now.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jan 07, 2011 11:32 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| For a new project, my developers are testing request/response application which is already in production but in different way. |
Again, what different way?
Clearly you or your developers have changed something. What exactly changed? Exactly what is different?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fatherjack |
| Posted: Fri Jan 07, 2011 11:41 am Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| mqjeff wrote: |
I consider it more likely that the sending app has a single MQCONN that it is trying to send all of the work through, than that the server hosting the qmgr is underpowered.
But that is just a guess. |
But the Op says there's no performance degradation with just one mobile device attached when presumably there will only be one MQCONN. I'm making the assumption here that the OP has pumped the increased volume of messages through the one device.
| mqjeff wrote: |
Why do you think it's an MQ issue? Maybe the work is stacked up inside the sending application and not in MQ.
|
Sounds like the sending app can't handle the connection of 50 mobile devices too well.
This really shouldn't be too hard to identify where the problem is. Maybe finding a solution might be a bit trickier.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 07, 2011 12:09 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fatherjack wrote: |
| mqjeff wrote: |
I consider it more likely that the sending app has a single MQCONN that it is trying to send all of the work through, than that the server hosting the qmgr is underpowered.
But that is just a guess. |
But the Op says there's no performance degradation with just one mobile device attached when presumably there will only be one MQCONN. I'm making the assumption here that the OP has pumped the increased volume of messages through the one device. |
Right but the one device is still only going to send one transaction/request at a time and wait for the reply - which will perform very very well over a single MQCONN that is also a single-threaded request/reply socket. But the minute you have anything trying to use an MQCONN to Get-with-Wait, you can't do anything else with it until that MQGET returns. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jan 07, 2011 12:34 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| All the 50 threads are running on single machine and connecting to MQ Server using the same SVRCONN channel and port. |
Are we presuming that multi-threading is the different way the OP mentioned (but didn't explain)?
I read the OP as multi-threading is/was in production, but is now misbehaving due to the different way thing.
Increase in workload is not usually described as a new way. New device types are not usually described as a new way, either.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| J.D |
| Posted: Fri Jan 07, 2011 1:45 pm Post subject: |
|
|
Voyager
Joined: 18 Dec 2009
Posts: 92
Location: United States
|
| mqjeff wrote: |
Why do you think it's an MQ issue? Maybe the work is stacked up inside the sending application and not in MQ.
What is the relationship between the number of threads in the sending application and the number of MQCONNS in the sending application?
What is the nature of the MQ connection between the sending application and the queue manager? |
Application use MQCONNX. How to find out the number of MQCONNX's in sending application? I am suspecting that all 50 threads are using single MQCONNX.
_________________
IBM WebSphere MQ & WAS Administrator |
|
| Back to top |
|
|
| fatherjack |
| Posted: Fri Jan 07, 2011 2:05 pm Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| mqjeff wrote: |
| Right but the one device is still only going to send one transaction/request at a time and wait for the reply - which will perform very very well over a single MQCONN that is also a single-threaded request/reply socket. But the minute you have anything trying to use an MQCONN to Get-with-Wait, you can't do anything else with it until that MQGET returns. |
Right. I see where you are coming from now. Sounds like a likely candidate.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 07, 2011 2:09 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| J.D wrote: |
| mqjeff wrote: |
Why do you think it's an MQ issue? Maybe the work is stacked up inside the sending application and not in MQ.
What is the relationship between the number of threads in the sending application and the number of MQCONNS in the sending application?
What is the nature of the MQ connection between the sending application and the queue manager? |
Application use MQCONNX. How to find out the number of MQCONNX's in sending application? I am suspecting that all 50 threads are using single MQCONNX. |
Look at the SVRCONN, and see how many instances it is running.
Look at DIS CONN(*) and see how many are coming from the application. |
|
| Back to top |
|
|
|
|
