| Author |
Message
|
| kirani |
 Posted: Thu Feb 28, 2002 5:04 pm Post subject: |
 |
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Hi,
I was wondering if anyone would like to share their experiences with Database operations in MQSI.
I am inserting 17010 records into SQL server database (on different machine) thru a message flow. My Message flow has 1 Input, 0 Output, 1 RouteToLabel, 4 Label and 4 Database nodes. However, for each input message it will go thru only 1 Database and 1 Label node. I have observed that the DB operations are very expensive in MQSI.
On a 4 CPU Compaq Server with 2GB RAM, I am getting a performance of 15.57 records/sec (18.50 recs/sec if DB is on local machine). My input XML message size is less than 2K. I am using MQSI 2.0.2 (CSD2) on Windows 2000.
If the same operation is done in C/C++ (or any other language) program using ODBC, I am sure it will give relatively high performance. MQSI also uses ODBC for Database access, so why are these operations so expensive if done thru message flow?
Am I missing anything here?
Regards,
Kiran
|
|
| Back to top |
|
 |
| dhaksr |
| Posted: Thu Feb 28, 2002 5:20 pm Post subject: |
|
|
Apprentice
Joined: 20 Feb 2002
Posts: 37
|
To justify our figures, We were playing around with the capacity planning tool (SupportPac IP03). By selecting 1 input node, 1 output node and leaving others at default values and choosing Windows NT SCSI i could see a throughput of 3070 msg/sec. However if I add 1 Insert node ... my throughput comes down to 150 msg/sec!!! 
So even the capacity planning tool recognizes that DB calls are "costly"
Has IBM done anything about this in V2.1 or any supportpacks?
P.S:
We are in a situation that we cant avoid making DB calls.
If WMQI doesnt perform well, we may be forced to go outside WMQI domain.
Rgds
Dhaks
|
|
| Back to top |
|
|
| amigupta1978 |
| Posted: Thu Feb 28, 2002 7:18 pm Post subject: |
|
|
Centurion
Joined: 22 Jan 2002
Posts: 132
Location: India
|
Hi,
If u look at in the IP04 support pack and see the results, the results you got are not surprising if u r using the persistent messages.
One of the major speed problem in when u make ur message persistant and above them if the message are in a transaction, since both of these require a I/O operation and that becomes the bottleneck. Also if u go through that document it shows clearly the difference between Persistant and non persistant messages,
(But I think u forget to mention the compute node in ur flow since there only u can set the Label for RouteToLabel node...which too is a costly feature )
Amit
|
|
| Back to top |
|
|
| kolban |
| Posted: Thu Feb 28, 2002 8:13 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2001
Posts: 1072
Location: Fort Worth, TX, USA
|
How many execution groups are you running? Have you defined multiple instancies (threads) per execution group? 15 msgs/second equates to 66 msecs/message. If you have a large input message which contains many logical fields then this may not be an unreasonable throughput.
What is the CPU consumption of the machine? If you are only running 1 execution group then, worst case, your utilization would only be 25% (or 100% of 1 of your 4 processors). I am suspecting that you will be less than that, especially if your database is off server from the broker. Can you post back and let us know? |
|
| Back to top |
|
|
| kirani |
| Posted: Fri Mar 01, 2002 9:05 am Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Neil & Amit,
Thanks for your reply. I forgot to mention that we have 1 Compute node where I set RouterList. Yes, I am setting Transaction to 'Yes' in MQInput node.
We have 4 Excution groups defined in our broker and I have total of 2-3 flows deployed to this broker (each in different execution group). While running the message flow we saw the CPU usage and it was over 94%. We are running only one instance of the message flow. In second test since our database was on the same box as broker, and 2nd CPU's usage was over 90%.
We looked at the queue manager settings and found that they have setup it for Circular Logging. Logging parameters were set to, LogBufferSize = 17, LogFilePages - 256. Can we change these to higher values?
Can we change some database parameters to get better performance?
Regards,
Kiran
|
|
| Back to top |
|
|
| kirani |
| Posted: Fri Mar 01, 2002 9:34 am Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
And all our messages are Non-Persistant.
|
|
| Back to top |
|
|
| amigupta1978 |
| Posted: Fri Mar 01, 2002 10:58 am Post subject: |
|
|
Centurion
Joined: 22 Jan 2002
Posts: 132
Location: India
|
Hi,
If u use the Transaction mode to 'Yes' in the MQInput node then also it does all the disk I/O and for that case too this becomes the bottleneck. (so this results are quite close to message being persistant).
Can u just try by removing the Transaction mode to 'No' and just chk the results.
Actually speaking I dont know the case when a person dont want to use the message without under transaction , so this bottleneck of disk I/O will always be there.
Amit |
|
| Back to top |
|
|
| kolban |
| Posted: Fri Mar 01, 2002 11:58 am Post subject: |
|
|
Grand Master
Joined: 22 May 2001
Posts: 1072
Location: Fort Worth, TX, USA
|
| Can you export and post the message flows? They may be valuable in understanding the scenario involved? |
|
| Back to top |
|
|
| kirani |
| Posted: Fri Mar 01, 2002 2:07 pm Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Hi,
I can send you an export of the message flow. Please send me your email id thru Private Message.
Regards,
Kiran
|
|
| Back to top |
|
|
| kolban |
| Posted: Fri Mar 01, 2002 3:01 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2001
Posts: 1072
Location: Fort Worth, TX, USA
|
Had a good look through your flow. Firstly, great job ... looks like youve got the hang of this WMQI thing  Your flows look good and your use and knowledge of the capabilities of the different nodes looks spot on.
Based on the flows, I can't see anything glaringly obvious that might be resulting in the performance numbers quoted. If it were me, I would start injecting ExecuteTime nodes into the flow to determine how much elapsed time is spent through each segment. Hopefully then we can see where the hot-spots lie.
See http://www-4.ibm.com/software/ts/mqseries/txppacs/ia0a.html |
|
| Back to top |
|
|
| kirani |
| Posted: Fri Mar 01, 2002 5:20 pm Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Thanks for your time. I will try using Execute Time plug-in nodes and get the stats again.
Have a great weekend!
Regards,
Kiran
|
|
| Back to top |
|
|
| kirani |
| Posted: Mon Mar 04, 2002 3:38 pm Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Hi,
I further added Execute Time plug-in nodes in my message flow to get low level stats and here are the results,
Stat 1 (1 DB node) - 12.4 msecs.
Stat 2 (1 Route To Label and 1 DB Node) - 14.4msec
Stat 3 (1 Route To Label, 1 DB Node, 1 Filter node) - 36.384 msecs
Stat 4 (1 Route To Label, 1 DB Node, 2 Filter nodes) - 60.5 msecs
These numbers are consisten with the figures that I posted earlier.
For one message, roughly it takes around 60.5 msecs to process. In earlier test for inserting 17000 messages it took me around 18 mins.
Regards,
Kiran
|
|
| Back to top |
|
|
| kirani |
| Posted: Tue Mar 05, 2002 1:54 pm Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Hi,
It seems we have some problems with our Execution Groups. The same messaeg flow is performing much better if deployed to other execution groups. Here are the sequence of activities we performed on our server.
1. Installed MQSI V2.0.2 on W2K
2. Created Broker on W2K (default EG is created automatically)
3. Created few Execution Groups (EG1, EG2, EG3)
4. Installed CSD1
5. Installed CSD2
6. Created Execution Groups (EG4)
My message flow performs very well if deployed to default or EG4 execution group. It takes 16.14 msecs for one message to process. With this rate, it can insert 17000 records in 4.5 mins.
Since other execution groups (EG1, EG2 and EG3) were created before installing any CSD, I think they are not creaetd properly. Any idea what could be wrong with these Execution Groups? Has anyone faced this problem before? Is this a known bug?
Regards,
Kiran
|
|
| Back to top |
|
|
| kolban |
| Posted: Wed Mar 06, 2002 8:25 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2001
Posts: 1072
Location: Fort Worth, TX, USA
|
| OOhh Ooohh ... Seen something like this before. By any change do you have execution group trace enabled? Trace is configured at the execution group level and if switched on, takes an explicit act to switch it off. Before you gon in and try this, run the commands (sorry, don't have them in my "wet ware memory cache") to deterime the current state of trace. Do that before trying to change anything. Please post back what you find. |
|
| Back to top |
|
|
| kirani |
| Posted: Thu Mar 07, 2002 9:26 am Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Hi,
I looked at the Operation pannel and the trace is not enabled on any of the Execution Group or Message Flows. Sometime back we enabled User trace on EG1, EG2 and EG3. After that we disabled them and did a complete deploy of the broker couple of times. This should disable the trace. I don't have direct access to our WMQI server, so I can not run 'mqsireportrace' command directly. I will try to get hold of the administrator and see if he can run this command and see if the the trace is enabled on these execution groups. We never turned on the trace on new execution group (EG4) where I am running this message flows now.
I further added ExecuteTime Plug-in into my message flow and got the new figures. It is taking 20-30 msecs for one message to process. With this stats it should take around 425 secs to insert 17000 records into remote database. Unfortunately, the message flow processing time is growing exponentially. i.e. it is taking 900 seconds to insert 17,000 records into DB. :0 No other process was running while running this message flow. This message flow was delpoyed to "good" execution group.
I checked the CPU utilization and its is not more than 25% for this message flow. Our server is Compaq Proliant ML570 (4 Pentium III Xeon Processors @700MHz) with 2 GB RAM.
We created another message flow, which accepts MRM input message of size 2004 bytes and inserts them into remote DB to compare the stats. This new message flow has same number of Filter, DB nodes. This message flow is taking same time (20-30 msecs) for processing one message. We tried inserting a batch of 1000 records, which takes 47 secs to insert them into database. Then We tried putting 17000 messages and it takes 647 secs (which is also not consistant with low level figures). These two message flows were deployed to the same Execution Group.
Do you think input queue build up is causing this? Do you think is there any memory leak going on here? Why the processing time is growing exponentially?
Thanks in advance.
Regards,
Kiran
|
|
| Back to top |
|
|
|
|
