| Author |
Message
|
| mattfarney |
 Posted: Thu May 12, 2011 2:55 pm Post subject: Error processing speed |
 |
|
Disciple
Joined: 17 Jan 2006
Posts: 167
Location: Ohio
|
I am using MB7 on MQ7.0.1. My flows are all data conversion, formatting, and routing. No external database, java, or anything special.
In a happy day scenario, they seem to process at normal speed. (5000 messages in ~40s)
I'm trying to test some error handling (i have an alternate route linked to the Failure node of the MQInput node). To simulate a problem, I have put(disabled) a queue in the middle of a flow.
I'm seeing throughput of a second or two per transaction or worse. It took around 10 minutes for 5000 transactions. I know there is extra processing occuring, so some slow down is expected, but the scope of this slowdown is a problem. A million errors can't take me eleven days to process. 
As best as I can tell, since I'm using a flow connection to implement my own error destination, rather than MQ's default behavior, the Backout Reque Queue and Backout Threshold on the MQInput node's queue has no impact.
I'm obviously missing something, since the performance degraded so much.
Any suggestions?
-mf |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu May 12, 2011 7:35 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Hi Matt,
Are you trying to post to a full queue somewhere? Through a cluster channel?
This might degrade significantly your performance.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri May 13, 2011 3:12 am Post subject: Re: Error processing speed |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mattfarney wrote: |
| I'm trying to test some error handling (i have an alternate route linked to the Failure node of the MQInput node). ... since I'm using a flow connection to implement my own error destination, rather than MQ's default behavior, the Backout Reque Queue and Backout Threshold on the MQInput node's queue has no impact. |
The MQInput Catch terminal will likely work better than the MQInput Failure terminal, as an alternate route.
The MQInput node routes the input message to the Failure terminal when it detects an internal error in the MQInput node itself, such as rolling a message back to the input queue a number of times, until it reaches the queue's backout threshold.
When the MQInput node rolls a message back to the input queue, that terminates the transaction, and the message flow discards any information about any exceptions that might have triggered the rollback. For the final transaction attempt, the MQInput node routes the message directly from the input queue to the Failure terminal, (bypassing the Out terminal).
The MQInput node routes the input message to the Catch terminal when handling downstream exceptions from the MQInput Out terminal. It is better suited for handling errors, as the ExceptionList contains useful information about the original cause of the error; in this case, the MQInput node doesn't need to roll the message back to the input queue at all. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri May 13, 2011 4:41 am Post subject: Re: Error processing speed |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mattfarney wrote: |
| normal speed. (5000 messages in ~40s) |
If this were my flow, I would be looking at why this flow is running so sssssllllloooowwwwwwlllllyyy. In Korean, this would be "Choon choon ha me da." I would expect 5,000 message to be processed in a second or two. Thats just me, and I don't know what kind of processing you are doing in there, so I won't make a blanket statement about performance here.
If the happy path performance is this slow, I wouldn't expect the unhappy path to be any better.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri May 13, 2011 6:37 am Post subject: Re: Error processing speed |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
| mattfarney wrote: |
| normal speed. (5000 messages in ~40s) |
If this were my flow, I would be looking at why this flow is running so sssssllllloooowwwwwwlllllyyy. In Korean, this would be "Choon choon ha me da." I would expect 5,000 message to be processed in a second or two. Thats just me, and I don't know what kind of processing you are doing in there, so I won't make a blanket statement about performance here.
If the happy path performance is this slow, I wouldn't expect the unhappy path to be any better. |
Not everyone is able to run their flows on an Power 7 595 box and all the trimmings.
Some of us have to make do with 1.5Ghx Sparc or Intel Q6600 CPU's.
Now where's that Air-raid shelter. I sense incoming SCUD's.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mattfarney |
| Posted: Fri May 13, 2011 9:57 am Post subject: |
|
|
Disciple
Joined: 17 Jan 2006
Posts: 167
Location: Ohio
|
| fjb_saper wrote: |
Are you trying to post to a full queue somewhere? Through a cluster channel?
This might degrade significantly your performance. |
I write to clustered queues in both the happy day and error handling scenarios. How much slower would the write to a non-local queue be?
-mf |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri May 13, 2011 10:50 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Local queue write speed is about 0.3 milliseconds for a 2,000 byte message. If the queue is remote (ie network latency) you could see write speed around 1.5 to 3 milliseconds. Clustered queues that are remote have similar latency.
My suggestion that your processing is slow is related to your message flow itself. On a cheap Linux box, you could see TPS rates near 10,000 for a simple flow (ie. 10,000 messages per second). On a muscle AIX Power 7, with a simple message flow, 500,000 TPS.
Of course, if you bog down your flow logic with alot of functionality, it will devastate your TPS rate.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri May 13, 2011 12:38 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| lancelotlinc wrote: |
| Local queue write speed is about 0.3 milliseconds for a 2,000 byte message. If the queue is remote (ie network latency) you could see write speed around 1.5 to 3 milliseconds. Clustered queues that are remote have similar latency. |
Writes to a remote queue are actually writes to a local transmission queue.
Writes to a remote cluster queue are actually writes to a local SYSTEM.CLUSTER.TRANSMIT.QUEUE.
Unless this flow is making an MQ Client connection to another Queue Manager?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mattfarney |
| Posted: Fri May 13, 2011 12:44 pm Post subject: Re: Error processing speed |
|
|
Disciple
Joined: 17 Jan 2006
Posts: 167
Location: Ohio
|
| rekarm01 wrote: |
| The MQInput Catch terminal will likely work better than the MQInput Failure terminal, as an alternate route. |
Thanks for the correction. The processing speed using Catch nodes instead of Failure nodes was in-line with normal processing speeds.
-mf |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 13, 2011 7:18 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| mattfarney wrote: |
| fjb_saper wrote: |
Are you trying to post to a full queue somewhere? Through a cluster channel?
This might degrade significantly your performance. |
I write to clustered queues in both the happy day and error handling scenarios. How much slower would the write to a non-local queue be?
-mf |
It's not the write to non-local queue that is slow.
Think about the cluster outside of WMB. Then write (1,000 to 2,000 msgs) to a non local queue that is full and observe how the messages flow through the SCTQ...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
