| Author |
Message
|
| WBIMaestro |
 Posted: Fri Dec 23, 2011 8:49 am Post subject: creating a test harness for message broker |
 |
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
Hi All,
I have a situation where I might have to have the same flow read from a different input queue for each different customer. need to replicate the flow in this way because the sending application cannot be updated in time to put some data in the RFH2header that would diffferentiate the customer messages (if they were to come to the same queue).
Has anyone faced a similar issue and can think of any alternatives?
assuming that repliating the flow 200 times for 200 different customer input and output queues is the only way to go, I would like to do some performance tests to see what kind of resource usage such a setup has.
If I remeber right there was some way to set the i/p and o/p quue names as properties to the message flow which would just pick up the right queue name at deploy-time. Is that correct and how is that implemented?
Is there any tool for stress testing the broker. In this case I need to continously resend a particular sequence of 5 messages repeatedly. Could probably do it from command line but just checking for alternatives.
thanks and happy holidays!! |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Dec 23, 2011 9:33 am Post subject: Re: creating a test harness for message broker |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| WBIMaestro wrote: |
| I have a situation where I might have to have the same flow read from a different input queue for each different customer. need to replicate the flow in this way because the sending application cannot be updated in time to put some data in the RFH2header that would diffferentiate the customer messages (if they were to come to the same queue). |
You do know how the flow will react to 200 MQInput nodes? Stress testing might involve just starting it.
| WBIMaestro wrote: |
| Has anyone faced a similar issue and can think of any alternatives? |
Why is it important to distinguish between input from different customers? What changes need to be made in the processing between customers? What else is distinctive in the message or the MQMD? Like the ReplyTo queue manager?
| WBIMaestro wrote: |
| assuming that repliating the flow 200 times for 200 different customer input and output queues is the only way to go, I would like to do some performance tests to see what kind of resource usage such a setup has. |
You either need 200 versions of the flow with 200 names (which is a maintenance nightmare), 1 flow with 200 input nodes or 1 flow in 200 execution groups. I don't see a good option here.
| WBIMaestro wrote: |
| If I remeber right there was some way to set the i/p and o/p quue names as properties to the message flow which would just pick up the right queue name at deploy-time. Is that correct and how is that implemented? |
Look up promoted properties.
| WBIMaestro wrote: |
| Is there any tool for stress testing the broker. In this case I need to continously resend a particular sequence of 5 messages repeatedly. Could probably do it from command line but just checking for alternatives. |
The command line is the best bet, with some script. I don't mean the amqsput sample but the support pacs
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Dec 23, 2011 10:20 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Loadrunner is a tool that would do this job very nicely. Costs $$$ though.
As Vitor says your proposed solution is frankly one that I'd not even consider.
If there truly is nothing in the data that can distinguish between customers then I'd really have to say that your design is slightly flawed.
you are not alone here. We do see quite a number of 'requirement' that have been dreamed up out of thin air with no regard (or even and knowledge or understanding) of what Broker can or more importantly can't do.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 23, 2011 10:24 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
2 queues, 1 flow with a few additional instances should do it.
You need to design in such a manner that this will work.
Worst case scenario: 201 queues (1 input, 200 replyto queues)...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 23, 2011 10:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| If there truly is nothing in the data that can distinguish between customers then I'd really have to say that your design is slightly flawed. |
The question remains if the data's identical why does the processing vary by customer? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Dec 23, 2011 11:02 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
The question remains if the data's identical why does the processing vary by customer? |
Per customer billing perhaps?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Dec 23, 2011 1:12 pm Post subject: Re: creating a test harness for message broker |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| Vitor wrote: |
| You either need 200 versions of the flow with 200 names (which is a maintenance nightmare), 1 flow with 200 input nodes or 1 flow in 200 execution groups. I don't see a good option here. |
The underlying OS typically limits the maximum number of threads per process; the exact value is platform-dependent, but typically 256 threads or less.
Each input node in each message flow consumes at least two threads, so any option that involves 200 input queues would probably have to be spread out among multiple execution groups. |
|
| Back to top |
|
|
| mgk |
| Posted: Fri Dec 23, 2011 2:41 pm Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| The underlying OS typically limits the maximum number of threads per process; the exact value is platform-dependent, but typically 256 threads or less |
Hmmm, I'm not aware of one that has limits this low these days.
| Quote: |
| Each input node in each message flow consumes at least two threads |
Not true. Each Input node starts exactly one thread (assuming additional instances are 0).
Of course, whilst this scenario with 200 Input nodes may work, as already pointed out, it is still not the best design...
Kind Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Dec 23, 2011 7:00 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
It's been a while since I researched this, and it was against an older version of the broker ... That will teach me to rely on my fuzzy memory before quoting numbers.
| mgk wrote: |
| Quote: |
| The underlying OS typically limits the maximum number of threads per process; the exact value is platform-dependent, but typically 256 threads or less |
Hmmm, I'm not aware of one that has limits this low these days. |
Ok. It's 512 kernel threads per process on my ancient 32-bit system, but the point was that there is a limit, and it may be lower than expected.
| mgk wrote: |
| Quote: |
| Each input node in each message flow consumes at least two threads |
Not true. Each Input node starts exactly one thread (assuming additional instances are 0). |
Oops. What I meant was: 1 thread per message flow + 1 thread per input node + 1 thread per additional instance, so each message flow that had an input node would consume at least 2 threads. (Whether that's still true or not is another matter.)
| mgk wrote: |
| Of course, whilst this scenario with 200 Input nodes may work, as already pointed out, it is still not the best design... |
Maybe that's the only point that really matters. It's not the best design ... |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Dec 24, 2011 6:05 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| rekarm01 wrote: |
It's been a while since I researched this, and it was against an older version of the broker ... That will teach me to rely on my fuzzy memory before quoting numbers.
Oops. What I meant was: 1 thread per message flow + 1 thread per input node + 1 thread per additional instance, so each message flow that had an input node would consume at least 2 threads. (Whether that's still true or not is another matter.) |
I've always counted only the input nodes in a message flow + the additional instances... never added an extra thread just for the flow by itself.
So if I have the traditional MQInput->MQOutput, then it's exactly 1 thread if there are exactly 0 additional instances, and otherwise 1+additional instances.
If I have two MQInput nodes->MQOutput, then it's at most 2*(1+additional instances). Although under no load, it will still only be 2 threads. |
|
| Back to top |
|
|
| WBIMaestro |
| Posted: Tue Dec 27, 2011 1:04 pm Post subject: |
|
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
| the problem could be solved very easily if the putting application can be modified to set the MQRFH2 header with a uniques customer id. I have suggesteed this but looks like its not something that can be immediately implemented. in the meantime 200 exec grps, one per customer with a copy of the message flow for that customer (and relevant i/p,o/p, audit,error ) queues seems to be the option to go with. anyone see any issues there? |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Dec 27, 2011 1:05 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Dec 27, 2011 1:16 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| lancelotlinc wrote: |
| Go back to the drawing board. |
 Looks like the service is largely independent from the customer in its processing. 1 input queue and 200 output queues should do it...
This could also let you reassign cost by message processed...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Dec 27, 2011 1:22 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| WBIMaestro wrote: |
| anyone see any issues there? |
Huge administrative & maintenance overheads? A reliance on all the properties your deploy script uses (and if you're thinking of deploying 200 flows by hand you're crazy) being correct so that the flow for customer 1 is not only deployed to execution group 1 but reading from the input queue that belongs to customer 1 and writing the response to the output that belongs to customer 1. Plus 200 audit and error mechanisms, unless you plan to combine all of that, which would pose it's own issues.
I shudder to think how you're going to keep that up to date.
You've still not explained why each customer needs it's own flow.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
