| Author |
Message
|
| gilc |
 Posted: Tue Jul 16, 2013 4:25 am Post subject: Collector node? Maybe a different design? |
 |
|
Apprentice
Joined: 17 Mar 2011
Posts: 32
|
Hey all
I have an issue regarding a design we have and would like to consult with you.
The sending application sends messages with X number of records that changes in every message. The receiving application has to get those records in some BULK_SIZE which can be greater\lower\equal to X (cause like I said, X changes every message).
So the requirement is to split the records to bulks and if there are records "left" (meaning, less than BULK_SIZE), they'll have to wait for more "left" records to come in the next messages (or sent as they are after a certain Timeout).
So our design is to split the the input messages to bulks of records and propagate. Whenever we have some records "left" , we send them to a Collector node (with one input) which waits for more "left" records to add into a new bulk.
And an example for better understanding:
BULK_SIZE = 5
1st message comes with 6 records.
1 bulk is sent to receiver and 1 record is waiting in Collector.
2nd message comes (in less than 10 seconds) with 2 records.
Adds up to the previous collection which after 10s sends to the receiver a bulk with 3 records.
So... my question basically is do you see any other solution instead of using Collector node? This node makes flow development a pain in the arse especially when there's an exception...
Using WMB 8.0.0.2.
Any advice will be grateful.
Many thanks,
~Gil |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Tue Jul 16, 2013 4:29 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
I would stack incoming records in a queue (or in GlobalCache). When depth reaches X, send a package out.
No muss, no fuss, works every time.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| gilc |
| Posted: Tue Jul 16, 2013 4:43 am Post subject: |
|
|
Apprentice
Joined: 17 Mar 2011
Posts: 32
|
Hey lance
First of all thanks for the reply.
The problem is that we don't always wait for X messages. We wait for X messages OR until Timeout (I probably didn't explain myself enough the first time.. sorry...).
That means we have to schedule a Get call as well. and if I understood you correctly, by "package" them means group them, so that means we need to know what's the last message in that group. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 16, 2013 4:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
This is a collection pattern.
You can take various approaches to avoid using the Collector node to accomplish a collection pattern. Database, queue, expired message handling, global cache, etc.
But I expect you'll find that they all offer tradeoffs.
I wouldn't tend to take the approach you've had, where you forward complete batches immediately, and only delay leftover records. i'd forward all records independently to collector and fetch complete sets out the other side.
But presumably that's a performance optimization made from an architectural or experiential perspective. |
|
| Back to top |
|
|
| gilc |
| Posted: Tue Jul 16, 2013 5:08 am Post subject: |
|
|
Apprentice
Joined: 17 Mar 2011
Posts: 32
|
Hey jeff
I also thought of the idea of sending each record independently to a collector but as you said it might cause a performance issue especially because the bulk size will probably be 10 and we can have messages with hundreds of records in them (and others with only one record).
Thanks for your insight, I might consider doing so nevertheless. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 16, 2013 5:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
To be clear, what you are doing to forward complete batches immediately is probably a performance optimization over the way I'd tend to do it first hand.
So I expect your solution performs better than the one I'd pull out first hand.
Except that using Collector node for all records individually will propagate complete collections independently. So as you feed records from an input message with 1000s of records, each set is completed and propagated in a separate thread.
So the performance of each approach may scale differently. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Jul 16, 2013 6:11 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| gilc wrote: |
| Hey lance First of all thanks for the reply.The problem is that we don't always wait for X messages. We wait for X messages OR until Timeout (I probably didn't explain myself enough the first time.. sorry...). That means we have to schedule a Get call as well. and if I understood you correctly, by "package" them means group them, so that means we need to know what's the last message in that group. |
You're welcome.
WMQ has a queue property called Queue service interval:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzax.doc%2Fmo10830_.htm
You can set QSI-HIGH for timeout at your desired interval, which can be tuned at runtime, especially in PROD.
Every <n> milliseconds where your queue still has msgs in it, an MQ event message is generated, which could be fed into your message flow to read the next (incomplete) batch of messages.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 16, 2013 9:35 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I had problems using the Collector Node when a variable number of events were in a collection. In my case, the range was from 1 to 18 messages and it became pretty errattic in its operation.
I redesigned the process to use a DB Table and that has been working without problem for the last 8 months and more than 1 million collections.
In other flows with a static set of inputs the collector node works fine.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| gilc |
| Posted: Tue Jul 16, 2013 11:34 pm Post subject: |
|
|
Apprentice
Joined: 17 Mar 2011
Posts: 32
|
@jeff
Duly noted.
@lance
Hmm I actually never knew about this. definitely worth a read!
@smdavies
I wish I could use DB here (if not for this solution then for other issues). The manager is VERY stubborn and sticks to the "if it worked 3 years ago, it's good enough for now" approach. I don't know who was the designer of the infrastructure back then but he's done some crazy maneuvers... sorry, just blowing off some steam... *sigh*
Thanks all! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 16, 2013 11:51 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If it worked 3 years ago, it was working under an entirely separate set of business requirements.
Or your business is the mythical business that has requirements that are well known and well understood and don't change. |
|
| Back to top |
|
|
| gilc |
| Posted: Wed Jul 17, 2013 12:06 am Post subject: |
|
|
Apprentice
Joined: 17 Mar 2011
Posts: 32
|
They work here with closed projects that are separate from one another. For every new project they copy-paste the infra from one to another and it works "fine" cause the basics are the same for each one (they're written horribly in my opinion but they do the job).
I came here to design and build a new project and I'm trying to improve the overall infra.
But that's just a little story about the business regardless to the Collection issue which is a new requirement for this specific project. |
|
| Back to top |
|
|
| goffinf |
| Posted: Wed Jul 17, 2013 12:28 am Post subject: |
|
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
| gilc wrote: |
But that's just a little story about the business regardless to the Collection issue which is a new requirement for this specific project. |
... and thus an opportunity for you to build something that isn't constrained by existing implementation patterns I would have thought ! |
|
| Back to top |
|
|
| gilc |
| Posted: Wed Jul 17, 2013 4:16 am Post subject: |
|
|
Apprentice
Joined: 17 Mar 2011
Posts: 32
|
| goffinf wrote: |
| gilc wrote: |
But that's just a little story about the business regardless to the Collection issue which is a new requirement for this specific project. |
... and thus an opportunity for you to build something that isn't constrained by existing implementation patterns I would have thought ! |
Well yeah, that's what I'm hoping to achieve and that's why I opened this thread in the first place - to get some ideas of different approaches for the issue described. |
|
| Back to top |
|
|
|
|
