| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Collector Node design Question |
« View previous topic :: View next topic » |
| Author |
Message
|
| next |
 Posted: Fri May 04, 2012 7:25 am Post subject: Collector Node design Question |
 |
|
Voyager
Joined: 02 May 2010
Posts: 75
|
Hi,
We are facing an issue with the collector node.
The input for the flow is from a file and we have to use the same input for sending output to two target systems. One with a file and one with MQ as outputs.
We had put the messages to two queues after reading the delimited records from the file.
We had designed the second flow to read the messages from the queue and collate them using a collector node for writing it as a Single message with multiple records for the MQ output. We had set a timeout as we wont know how many records can come in the file.
We have used the same pattern for multiple flows reading multiple files.
The issue is when the message number crosses 6000, the flow just hangs and there is no output from the expire terminal of the collector node.
Is this is an issue with the collector node ? Has anyone experienced such issues ? 
Can Collector node handle more than 6000 messages ?
Last edited by next on Fri May 04, 2012 7:35 am; edited 1 time in total |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri May 04, 2012 7:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I would turn on service trace and collect it when you send the 5999th message and again when you send the 6000th message.
And then open a PMR. |
|
| Back to top |
|
|
| next |
| Posted: Fri May 04, 2012 7:38 am Post subject: |
|
|
Voyager
Joined: 02 May 2010
Posts: 75
|
Thanks mqjeff,
Yes i will try to do that, it is not exactly happening at 6000th message, it is in that range of 6200~. I will try to get the exact number.
In theory, would it be fine to use the collector node ? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri May 04, 2012 7:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| next wrote: |
| In theory, would it be fine to use the collector node ? |
Yes, in general.
In specific, it may not be set up to handle that many messages in a timely, well behaved fashion.
In specific, you may simply be using up the available memory for the EG process, since it has to hold all those messages at once. |
|
| Back to top |
|
|
| next |
| Posted: Fri May 04, 2012 10:16 am Post subject: |
|
|
Voyager
Joined: 02 May 2010
Posts: 75
|
It uses MQ Queues for holding the messages right ?
Would it help if i try to increase the Java heap size of the EG ? |
|
| Back to top |
|
|
| goffinf |
| Posted: Sat May 05, 2012 2:39 am Post subject: |
|
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
| next wrote: |
| It uses MQ Queues for holding the messages right ? |
Yes SYSTEM.BROKER.EDA.EVENTS and SYSTEM.BROKER.EDA.COLLECTIONS
| next wrote: |
| Would it help if i try to increase the Java heap size of the EG ? |
Possibly, but ...
We were using the Collector for creating analysis of flow usage. The number of messages in any given collection definitely exceeded the 6000 where you are experiencing a problem but we did eventually have to redesign this process WITHOUT using Collectors. The main reason was to do with achieving sufficient thru-put, that is, matching the rate at which messages were arriving with the propagation of collections from the Collector. This is a common(ish) issue but to some extent is exacerbated by the slightly peculiar nature of Collector nodes insofar as threading (I'm not sure it's actually possible to influence the number of thread available on the output side of the Collector other than running more than one of them in a flow, which brings it's own constraints around correlation of course).
Collector nodes certainly appeared to work reliably up to some thru-put thresholds but determining exactly the point at which they would fail with the sort of behaviour you are seeing we found to be very difficult to pin down. I suspect (but don't know with any certainty) that they are effected by other resource usage such as JVM heap (for it's internal maps of collections and associated correlation ids) as you have mentioned and of course the Collector isn't the only thing consuming that resource. I did try for a long time with various arrangements of flows to manage thru-put, tried multiple Collectors, increasing the queue depths to maximum values, but all ultimately proved unsuccessful in terms of providing the predictable and reliable behaviour we were looking for, so in the end we abandoned Collectors for this particular use case (I'm sure they are fine in other circumstances).
Part of the testing we conducted was looking both in this forum and in APARs and fix pack changes for various releases of Broker. You will find quite a few that relate to the Collector node, some of which refer to memory usage, threading and the speed of propagation.
Sorry if this sounds negative, it's not my intention at all to put you off using Collectors (I would certainly consider them for other profiles), I'm just trying to give you a sense of what we found.
HTHs
Fraser. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat May 05, 2012 4:00 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I've used a Flow with a Collector node to process Broker Event messages.
In my testing, I let the number of messages build up to well over 400K before starting the flow.
Naturally I made sure that the SYSTEM.BROKER.... queues had their MAXDEPTH values adusted accordingly.
I encounter no problems like the ones you are seeing.
This was on 6.1.0.3 , 6.1.0.7 and 7.0.0.1 all on Linux or Solaris
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
