| Author |
Message
|
| palaslet |
 Posted: Tue Mar 15, 2011 5:48 am Post subject: Performance problem using Get |
 |
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
Hi all,
I'm currently developing an application to fetch messages from IBM MQ, do some custom validation and parsing, and then putting the result into a database.
Traffic is quite high, and it's not unusual to get 2000 messages in one go from the external system posting messages to the queue. The system needs to put these messages as fast as possible into the database or else the backlog will increase infinite.
When fetching messages from the queue I can see that the first 50 or so goes fast (15 ms pr message or less), then the system slows down to about 5s pr message (I'm sure you can see that this behavior is problematic).
It's probably worth mentioning that I don't want to have many messages in memory at any time as that's not very reliable in case of failure for any reason (This is actually a customer demand, and I can't do anything about it).
I've tried several different methods, but it all comes out the same. 50 messages super fast and then the rest takes 5 seconds per message.
EDIT: I can download 2000 messages in 14 seconds if I put all of them in memory, but if I batch them in any way to reduce the number of messages in memory) it seems breaking the continuous "get-ing" makes the server down-prioritize my connection, and after about 50 or so messages, I'm down to 5 seconds per message.
Please help!
Last edited by palaslet on Tue Mar 15, 2011 5:54 am; edited 1 time in total |
|
| Back to top |
|
 |
| Mr Butcher |
| Posted: Tue Mar 15, 2011 5:53 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
help us to help you....
what plattform
what mq level
what programming language
do you mind to show the code / loop which is resposible for the mqget
how often do you commit
what database
..... more to come
_________________
Regards, Butcher |
|
| Back to top |
|
|
| palaslet |
| Posted: Tue Mar 15, 2011 6:02 am Post subject: |
|
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
Sorry...
Platform: Windows
MQ Level: ???? What do you mean by that?
Language: C#
I've tried several different commit configurations, but it doesn't seem to have any effect.
| Code: |
List<WebSphereMQMessage> uncommittedMessages = new List<WebSphereMQMessage>();
Boolean hasMoreMessages = true;
while (hasMoreMessages)
{
using (MQQueue mqQueue = mqQMgr.AccessQueue(connectionParameters.QueueName, MQC.MQOO_INPUT_SHARED + MQC.MQOO_FAIL_IF_QUIESCING))
{
MQGetMessageOptions mqGetMsgOpts = new MQGetMessageOptions();
mqGetMsgOpts.Options |= MQC.MQGMO_NO_WAIT;
mqGetMsgOpts.Options |= MQC.MQGMO_NO_SYNCPOINT;
for (Int32 i = 0; i < 25; i++)
{
WebSphereMQMessage mqMsg = null;
try
{
mqMsg = new WebSphereMQMessage();
mqMsg.CorrelationId = MQC.MQCI_NONE;
mqMsg.MessageId = MQC.MQMI_NONE;
mqQueue.Get(mqMsg.InnerMessage, mqGetMsgOpts);
uncommittedMessages.Add(mqMsg);
}
catch (MQException mqEx)
{
if (mqEx.ReasonCode == MQC.MQRC_NO_MSG_AVAILABLE)
{
hasMoreMessages = false;
break;
}
}
}
}
if (uncommittedMessages.Count > 0)
{
// Add messages to database
foreach (WebSphereMQMessage mqMsg in uncommittedMessages)
{
if (newMessageReceivedCallback == null || newMessageReceivedCallback(mqMsg))
messages.Add(mqMsg);
}
uncommittedMessages.Clear();
mqQMgr.Commit();
}
}
|
|
|
| Back to top |
|
|
| zpat |
| Posted: Tue Mar 15, 2011 6:39 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Use MQ syncpointing to protect data from memory crashes. Limiting the number in memory is not as good. Syncpoints should not be too large, I tend to suggest no more than 200 uncomitted messages in a single unit of work, although most queue managers allow up to 10,000. |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Tue Mar 15, 2011 6:44 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
just a quick finding.
if you mqget with MQGMO_NO_SYNCPOINT, there is nothing you can commit, as you get the message outside the UOW.
mq level is the version. e.g. 6.0.1.13 or whatever. type "dspmqver" on the server, it should show you the version.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| palaslet |
| Posted: Tue Mar 15, 2011 6:46 am Post subject: |
|
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
Thanks for the suggestion.
How do I best implement syncpointing in C#?
I'm somewhat unfamiliar with the MQ inner functionality and all its possibilities. Earlier we were dealing with much less data and thus the perfomance issue never occured. Simple GET -> PARSE -> COMMIT sequence was sufficient. Lately we are designing one of the largest systems in its class, and some of the customers want to use MQ to integrate with us. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Mar 15, 2011 6:53 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Single phase commit is very easy. Simply change the MQGET to have the option MQGMO_SYNCPOINT. After a certain number of these gets (and when the data has been processed successfully) you can issue the MQCMIT call. |
|
| Back to top |
|
|
| palaslet |
| Posted: Tue Mar 15, 2011 7:03 am Post subject: |
|
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
I'm sorry, but that won't help me. I'll try to be more accurate in my problem description:
I need to:
1. fetch a message from MQ
2. Validate and transform it
3. Put it into the database (Oracle)
4. commit the message in MQ
I can batch the messages from MQ before putting them into the database to improve speed. The problem is when when the batch is full (lets say 200) and I'm sending the batch of to parsing and the program returns after parsing a few seconds later, the speed of GET is down to 5 seconds pr message. And thats with no commit and no syncpoint (I just removed them to be sure nothing was interfering with performance) |
|
| Back to top |
|
|
| palaslet |
| Posted: Tue Mar 15, 2011 7:06 am Post subject: |
|
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
Here's a better code example with some test results:
1. This code fetches 200 messages in 10 seconds.
| Code: |
using (MQQueue mqQueue = mqQMgr.AccessQueue(connectionParameters.QueueName, MQC.MQOO_INPUT_SHARED + MQC.MQOO_FAIL_IF_QUIESCING))
{
List<WebSphereMQMessage> uncommittedMessages = new List<WebSphereMQMessage>();
Boolean hasMoreMessages = true;
Console.WriteLine(DateTime.Now.ToString("hh:MM:ss.fffff") + " start fetcing");
while (hasMoreMessages)
{
MQGetMessageOptions mqGetMsgOpts = new MQGetMessageOptions();
mqGetMsgOpts.Options |= MQC.MQGMO_NO_WAIT;
mqGetMsgOpts.Options |= MQC.MQGMO_NO_SYNCPOINT;
for (Int32 i = 0; i < 25; i++)
{
WebSphereMQMessage mqMsg = null;
try
{
mqMsg = new WebSphereMQMessage();
mqMsg.CorrelationId = MQC.MQCI_NONE;
mqMsg.MessageId = MQC.MQMI_NONE;
mqQueue.Get(mqMsg.InnerMessage, mqGetMsgOpts);
uncommittedMessages.Add(mqMsg);
}
catch (MQException mqEx)
{
if (mqEx.ReasonCode == MQC.MQRC_NO_MSG_AVAILABLE)
{
hasMoreMessages = false;
break;
}
ReasonCodes rc = new ReasonCodes();
if (mqEx.ReasonCode == MQC.MQRC_FORMAT_ERROR)
{
Exception ex = new Exception(rc[mqEx.ReasonCode]);
ex.Data.Add("message", mqMsg);
throw ex;
}
throw new Exception(rc[mqEx.ReasonCode]);
}
}
//if (uncommittedMessages.Count > 0)
//{
// foreach (WebSphereMQMessage mqMsg in uncommittedMessages)
// {
// if (newMessageReceivedCallback == null || newMessageReceivedCallback(mqMsg))
// messages.Add(mqMsg);
// }
// uncommittedMessages.Clear();
//}
}
Console.WriteLine(DateTime.Now.ToString("hh:MM:ss.fffff") + " end fetcing");
}
|
This is the resulting output in console:
| Code: |
03:03:20.52606 start fetching
03:03:30.94799 end fetching
|
2 If I uncomment the part handling the messages, the performance drops to 200 messages in ca 14 minutes 40 seconds
| Code: |
03:49:23.66249 start fetching
04:04:05.72180 end fetching
|
The "newMessageReceivedCallback" method will do a 20 ms sleep to simulate validation and paring into database. It shouldn't add more than 4 seconds to the total time |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Mar 15, 2011 7:18 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
how big are those messages? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| palaslet |
| Posted: Tue Mar 15, 2011 7:29 am Post subject: |
|
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
That varies greatly as they are messages to customs authorities around the world describing the goods they want to import/export. One message can have 1 goods line, or 1000 goodslines depending of the nature of the goods.
So the messages can vary from 1KB to 100 KB or even more.
For the test described above, I've created a random message generator that puts 200 random messages of 3-4KB on the queue. |
|
| Back to top |
|
|
| palaslet |
| Posted: Tue Mar 15, 2011 7:30 am Post subject: |
|
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
| Mr Butcher wrote: |
mq level is the version. e.g. 6.0.1.13 or whatever. type "dspmqver" on the server, it should show you the version. |
Name: WebSphere MQ
Version: 6.0.0.0
CMVC level: p000-L050519
BuildType: IKAP - (Production) |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Mar 15, 2011 1:20 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| palaslet wrote: |
| Mr Butcher wrote: |
mq level is the version. e.g. 6.0.1.13 or whatever. type "dspmqver" on the server, it should show you the version. |
Name: WebSphere MQ
Version: 6.0.0.0
CMVC level: p000-L050519
BuildType: IKAP - (Production) |
Upgrade soonest. That version (vanilla 6.0) was not one of the best. You should be at 6.0.2.10 or above, or at V7.0.1.4.
It is entirely possible that you will not see the problem at V 6.0.2.10
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Mar 15, 2011 3:03 pm Post subject: Re: Performance problem using Get |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| palaslet wrote: |
| When fetching messages from the queue I can see that the first 50 or so goes fast (15 ms pr message or less), then the system slows down to about 5s pr message (I'm sure you can see that this behavior is problematic). |
It looks like you are hitting some sort of resource limitation or contention, most likely in Windows managing all the processes and workload, such as memory, CPU.
MQ does not normally behave like this - it should be fast all the time.
_________________
Glenn |
|
| Back to top |
|
|
| palaslet |
| Posted: Wed Mar 16, 2011 12:27 am Post subject: |
|
|
Newbie
Joined: 15 Mar 2011
Posts: 9
|
I totally agree. The odd thing is that I can reproduce this both locally on an inhouse MQ server, and at the customer at their MQ server.
If it's worth anything I'll try to create a standalone console program to reproduce it and get the tech team to poke a hole in the firewall so any of you can run it and play around.
Just tell me to, and I'll start the coding.
Thanks for all your help so far! |
|
| Back to top |
|
|
|
|
