| Author |
Message
|
| vincent |
 Posted: Sun May 21, 2006 6:58 am Post subject: Increasing client process does not help |
 |
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
Background:
20000 messages are processed in 77 seconds using single client process for both put and get
The same 20000 messages are processed in 35 seconds using two client processes for both put and get
But 40000 messages are not processed in 77 seconds using two client processes for both put and get?
All the clients are using the same client channel?
So when i add more and more clients , the total time taken to process the messages are doubled based on the number of messages though i am increasing the clients. I think it should be constant, right
Llet me know if i need to do any configuration changes to make it work properly?
-Vincent |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Sun May 21, 2006 7:18 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Vincent
There is no such thing as a linear curve or constant processing time.
Too many variations come to play
- network and bandwidth
- processing time on the app side for the received message (cpu)
- memory constraints on the processing app side
- database constraints if any included in the processing time
- logspace constraints
- max uncommitted
- transaction manager constraints whether MQ or external...
- serialization constraints (not allowing multithread processing)...
- whatever else I am forgetting to mention
So if your time is not linear look first if you hit a bottleneck in any of the previous points.
However I found through tests that even in the best of conditions the time is less than linear. As there is more overhead involved in multiple clients /vs single client the equation shoud always be more like
Total time = (Total Single thread time / # of threads) + (Total Single thread time * overhead % per thread * # of threads)
And note that depending on platform the overhead% per thread might be dependent on the total number of threads declared in your process.....(complex approximation of memory / vs thread / vs cpu thresholds)
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vincent |
| Posted: Sun May 21, 2006 10:09 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
What do you think, if i want to process 1500 messages per second?
I am running amqsput and amqsget sample program with persist on it.
network and bandwidth - it is a dedicated network
processing time on the app side for the received message (cpu) - sample program(no processing)
memory constraints on the processing app side - sample program
database constraints if any included in the processing time - no involvement
logspace constraints - perfectly cleared,enough free space
max uncommitted - no uncommited msges
transaction manager constraints whether MQ or external... - no external controls
serialization constraints (not allowing multithread processing)... ??????
whatever else I am forgetting to mention
Any thoughts? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Sun May 21, 2006 10:33 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| vincent wrote: |
| network and bandwidth - it is a dedicated network |
Is it a TCP/IP network? An LU 6.2 network? How fast is it? How far apart are the two machines? Are they on the same switch? Many switches between them? A VPN link? A slower WAN link?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| vincent |
| Posted: Sun May 21, 2006 12:52 pm Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
| Both the clients and server are in the same machine(Sun Solaris 8 and MQ 5.3 csd12) and communicationg via tcp/ip. |
|
| Back to top |
|
|
| vincent |
| Posted: Mon May 22, 2006 3:16 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
If I/O is taking too much time to write , what are the MQ and OS setting to be fine tuned.
I have seen it throup top utility, cpu is dedicating 50% to I/O. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 22, 2006 5:53 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| Nigelg |
| Posted: Mon May 22, 2006 6:07 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
| Quote: |
What do you think, if i want to process 1500 messages per second?
I am running amqsput and amqsget sample program with persist on it.
|
You will not get anywhere near 1500 mps with persistent msgs.
The best way to improve performance is to MQGET/MQPUT inside syncpoint, even if there is a single MQ call in the UoW. This improves concurrency in writing to the log, since the log force (the commit) does not delay the get/put of the msg.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| vincent |
| Posted: Mon May 22, 2006 1:19 pm Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
As per the IBM document,we can get maximum of 2343. But i do not know. What type of log should be used ? I think it should be circular and other parameters should be as per ibm recommendation.
If i turn off log, we are getting max of 10000 msg/sec as per IBM documentation.
How do i increase the thread of amqrmppa. I think as per IBM, it should be 12. How doi i increase it? When i use the top command, it shows max of 5. Pls advice.
vincent. |
|
| Back to top |
|
|
| vincent |
| Posted: Tue May 23, 2006 5:23 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
| Would PUT and GET happens on Same Queue give low performance? Should we specify an replyQ to retrieve? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 23, 2006 3:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The request reply pattern was made to get the reply from a different queue/qmgr from where the provider put the message. Use it!
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vincent |
| Posted: Thu May 25, 2006 4:26 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
Saper, Do you have any sample program to use this pattern?
-vincent |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu May 25, 2006 3:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| vincent wrote: |
Saper, Do you have any sample program to use this pattern?
-vincent |
This is well documented. Think as well about temporary dynamic queues if the messages are non persistent.
Surf some you will find lots of hits....
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vincent |
| Posted: Fri May 26, 2006 7:36 am Post subject: |
|
|
Voyager
Joined: 21 Apr 2006
Posts: 81
|
Messages are persistant.
Is there any way the messages can be moved from one queue to another queue within the same queue mananger without doing any coding but by configuring either Queue or Queue Manager. The message will be sent as a datagram.
Would be very helpfull if anyone share the knowledge or docuement or link.
-Vincent |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri May 26, 2006 12:15 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
The MO71 Support Pack will do this for you. Nothing in base MQ will.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
