ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Collector node? Maybe a different design?

Post new topic  Reply to topic
 Collector node? Maybe a different design? « View previous topic :: View next topic » 
Author Message
gilc
PostPosted: Tue Jul 16, 2013 4:25 am    Post subject: Collector node? Maybe a different design? Reply with quote

Apprentice

Joined: 17 Mar 2011
Posts: 32

Hey all

I have an issue regarding a design we have and would like to consult with you.

The sending application sends messages with X number of records that changes in every message. The receiving application has to get those records in some BULK_SIZE which can be greater\lower\equal to X (cause like I said, X changes every message).
So the requirement is to split the records to bulks and if there are records "left" (meaning, less than BULK_SIZE), they'll have to wait for more "left" records to come in the next messages (or sent as they are after a certain Timeout).

So our design is to split the the input messages to bulks of records and propagate. Whenever we have some records "left" , we send them to a Collector node (with one input) which waits for more "left" records to add into a new bulk.

And an example for better understanding:
BULK_SIZE = 5
1st message comes with 6 records.
1 bulk is sent to receiver and 1 record is waiting in Collector.
2nd message comes (in less than 10 seconds) with 2 records.
Adds up to the previous collection which after 10s sends to the receiver a bulk with 3 records.

So... my question basically is do you see any other solution instead of using Collector node? This node makes flow development a pain in the arse especially when there's an exception...

Using WMB 8.0.0.2.

Any advice will be grateful.
Many thanks,
~Gil
Back to top
View user's profile Send private message
lancelotlinc
PostPosted: Tue Jul 16, 2013 4:29 am    Post subject: Reply with quote

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

I would stack incoming records in a queue (or in GlobalCache). When depth reaches X, send a package out.

No muss, no fuss, works every time.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Back to top
View user's profile Send private message Send e-mail
gilc
PostPosted: Tue Jul 16, 2013 4:43 am    Post subject: Reply with quote

Apprentice

Joined: 17 Mar 2011
Posts: 32

Hey lance

First of all thanks for the reply.

The problem is that we don't always wait for X messages. We wait for X messages OR until Timeout (I probably didn't explain myself enough the first time.. sorry...).
That means we have to schedule a Get call as well. and if I understood you correctly, by "package" them means group them, so that means we need to know what's the last message in that group.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jul 16, 2013 4:58 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

This is a collection pattern.

You can take various approaches to avoid using the Collector node to accomplish a collection pattern. Database, queue, expired message handling, global cache, etc.

But I expect you'll find that they all offer tradeoffs.

I wouldn't tend to take the approach you've had, where you forward complete batches immediately, and only delay leftover records. i'd forward all records independently to collector and fetch complete sets out the other side.

But presumably that's a performance optimization made from an architectural or experiential perspective.
Back to top
View user's profile Send private message
gilc
PostPosted: Tue Jul 16, 2013 5:08 am    Post subject: Reply with quote

Apprentice

Joined: 17 Mar 2011
Posts: 32

Hey jeff

I also thought of the idea of sending each record independently to a collector but as you said it might cause a performance issue especially because the bulk size will probably be 10 and we can have messages with hundreds of records in them (and others with only one record).

Thanks for your insight, I might consider doing so nevertheless.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jul 16, 2013 5:46 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

To be clear, what you are doing to forward complete batches immediately is probably a performance optimization over the way I'd tend to do it first hand.

So I expect your solution performs better than the one I'd pull out first hand.

Except that using Collector node for all records individually will propagate complete collections independently. So as you feed records from an input message with 1000s of records, each set is completed and propagated in a separate thread.

So the performance of each approach may scale differently.
Back to top
View user's profile Send private message
lancelotlinc
PostPosted: Tue Jul 16, 2013 6:11 am    Post subject: Reply with quote

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

gilc wrote:
Hey lance First of all thanks for the reply.The problem is that we don't always wait for X messages. We wait for X messages OR until Timeout (I probably didn't explain myself enough the first time.. sorry...). That means we have to schedule a Get call as well. and if I understood you correctly, by "package" them means group them, so that means we need to know what's the last message in that group.


You're welcome.

WMQ has a queue property called Queue service interval:

http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=%2Fcom.ibm.mq.csqzax.doc%2Fmo10830_.htm


You can set QSI-HIGH for timeout at your desired interval, which can be tuned at runtime, especially in PROD.

Every <n> milliseconds where your queue still has msgs in it, an MQ event message is generated, which could be fed into your message flow to read the next (incomplete) batch of messages.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Back to top
View user's profile Send private message Send e-mail
smdavies99
PostPosted: Tue Jul 16, 2013 9:35 am    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

I had problems using the Collector Node when a variable number of events were in a collection. In my case, the range was from 1 to 18 messages and it became pretty errattic in its operation.

I redesigned the process to use a DB Table and that has been working without problem for the last 8 months and more than 1 million collections.

In other flows with a static set of inputs the collector node works fine.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
gilc
PostPosted: Tue Jul 16, 2013 11:34 pm    Post subject: Reply with quote

Apprentice

Joined: 17 Mar 2011
Posts: 32

@jeff
Duly noted.

@lance
Hmm I actually never knew about this. definitely worth a read!

@smdavies
I wish I could use DB here (if not for this solution then for other issues). The manager is VERY stubborn and sticks to the "if it worked 3 years ago, it's good enough for now" approach. I don't know who was the designer of the infrastructure back then but he's done some crazy maneuvers... sorry, just blowing off some steam... *sigh*

Thanks all!
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jul 16, 2013 11:51 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

If it worked 3 years ago, it was working under an entirely separate set of business requirements.

Or your business is the mythical business that has requirements that are well known and well understood and don't change.
Back to top
View user's profile Send private message
gilc
PostPosted: Wed Jul 17, 2013 12:06 am    Post subject: Reply with quote

Apprentice

Joined: 17 Mar 2011
Posts: 32

They work here with closed projects that are separate from one another. For every new project they copy-paste the infra from one to another and it works "fine" cause the basics are the same for each one (they're written horribly in my opinion but they do the job).
I came here to design and build a new project and I'm trying to improve the overall infra.

But that's just a little story about the business regardless to the Collection issue which is a new requirement for this specific project.
Back to top
View user's profile Send private message
goffinf
PostPosted: Wed Jul 17, 2013 12:28 am    Post subject: Reply with quote

Chevalier

Joined: 05 Nov 2005
Posts: 401

gilc wrote:

But that's just a little story about the business regardless to the Collection issue which is a new requirement for this specific project.


... and thus an opportunity for you to build something that isn't constrained by existing implementation patterns I would have thought !
Back to top
View user's profile Send private message
gilc
PostPosted: Wed Jul 17, 2013 4:16 am    Post subject: Reply with quote

Apprentice

Joined: 17 Mar 2011
Posts: 32

goffinf wrote:
gilc wrote:

But that's just a little story about the business regardless to the Collection issue which is a new requirement for this specific project.


... and thus an opportunity for you to build something that isn't constrained by existing implementation patterns I would have thought !


Well yeah, that's what I'm hoping to achieve and that's why I opened this thread in the first place - to get some ideas of different approaches for the issue described.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Collector node? Maybe a different design?
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.