| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Creating 5000 MQ messages |
« View previous topic :: View next topic » |
| Author |
Message
|
| goffinf |
 Posted: Thu Jun 05, 2008 12:18 pm Post subject: Creating 5000 MQ messages |
 |
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
I have a flow that receives a message similar to this :-
<response>
<lowerBound>1000</lowerBound>
<upperBound>6000</upperBound>
</response>
The flow is then supposed to create messages on an MQ queue in a set format, one for each number in the range. As you can see the range can be as big as 5000.
I put together a simple flow for this just looping around a label node that has an attached MQOutput.
It works just fine on small'ish ranges (250 or so) but when I test it with 1000 the broker crashes.
I should also say that this processing isn't especially time critical, so if it takes a minute or two to complete, that should be OK.
I don't have the exact trace to hand (I'll post it tommorrow when I get back into the office), but I was wondering if :-
a. this a sensible way to approach this problem
b. there is a better approach that someone can suggest
c. whether anyone can suggest why the crash occurs (I appreciate that without the trace thats asking a bit much but ... maybe its a memory thing ?
Thanks
Fraser. |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Jun 05, 2008 10:14 pm Post subject: I did this a while ago |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
The case was where a single message would generate 10,000 messages.
The 10,000 messages were actually stored in a database bt there was still 10,000 MQ messages generated at the end of the day.
Te reason your 'broker crashed' ( well it was probably just the Execution Group DataFlowEngine that died) was that the EG process grew and grew and eventually ran out of address space as All the work you were trying to get done was being done under a single unit of work. This has to be kept in such a state so that if (for example) an exception was generated, the whole schebang would get rolled back in a coherent way.
Here is how I got the 10K output Messages
1) the receiving flow looked at the total number of messages that it needed to generate and then divided it by a value read from a DB table ( so that it could be changed on the fly if needed). 1 was added to the total and then the original message was sent to a SECOND flow (using Propagate) repeatedly until the 'total' was reached.
2) The second flow would read a message and also the 'count' value from the database and read the original messaged from the Database Table and after formatting it send it on its way ( again via the PROPAGATE esql function). This was done until the number of messages specified in the DB table had been sent and then that thread would be ended.
3) The second flow would read the next message from its input queue and the process of sending up to 10K messages would continue.
This way of generating <10K messages allows a much reduced load on the Execution Group tht runs the message flow that actually does final messages generation.
We set the limit to 500 and this seems to work fine and has been in production for 6 months now.
Now this approach may well have to be modified considerably for your situation but here is ONE way to achieve the generation of many MQ Messages from ONE input message.
There are no doubt other ways to do this.
Steve D
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| goffinf |
| Posted: Fri Jun 06, 2008 12:44 am Post subject: |
|
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
| Thanks Steve, that was really helpful, I'll give this pattern a try and post back my results. |
|
| Back to top |
|
|
| mgk |
| Posted: Fri Jun 06, 2008 12:53 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
I agree with everything Steve says, and this is a good way to do what you want to do. However, I think a likely reason the broker "crashed" in your case is because you were "just looping around a label node". And creating a loop in a flow is not good, as (on average) only a few hundred nodes can be called in sequence in a single flow. Therefore, using PROPAGATE as Steve says is a much better way to go to avoid loops in your flow.
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
