| Author |
Message
|
| smdavies99 |
 Posted: Tue Aug 16, 2016 11:03 am Post subject: |
 |
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Using a DB to hold the data is IMHO the best way forward.
Then the ordering or missing data can be handled once the last message has arrived.
It will also allow you to
- avoid the MQGet loop entirely. Each message is a single unit of work
- avoid the collector node
- mix message groups
- 2 or 2000 messages in a group it does not matter.
- Have a periodic clean up operation just in case there are is no END of Group message.
- Being able to monitor orphaned data.
I'd go further and split the processing up into two flows.
The first part to do the collection and storage of the db.
The second is only triggered when an End of group message is received. This reads the entire group of messages from the DB and processes them
The DB table would be keyed on the GroupId.
There are other ways for do this naturally but the above is how I'd go about doing it.
YMMV
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Aug 16, 2016 11:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| when a message has ended a collection. |
Would that still be considered part of the collection?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 16, 2016 11:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| when a message has ended a collection. |
Would that still be considered part of the collection? |
I'm sure the KC has something to say on this... 
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| siva reddy |
| Posted: Tue Aug 16, 2016 11:43 am Post subject: |
|
|
Novice
Joined: 16 Jul 2016
Posts: 11
|
vitor/ smdavies
say i have configured collector node to timeout after 5 mins, expecting that all the 2000 plus records of the group are processed by the mq input node.
Now even in a case where we have got only 9 records as a group, the collector node has to still wait for 5 mins even after receiving all the messages.
so , i am thinking which way --- go with the collector node but cannot avoid wait time for small group of messages
- or go with DB approach , though it involves additional processing of inserting and fetching records.
I will do a PoC and validate which option gives the best performance overall |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 16, 2016 11:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| siva reddy wrote: |
| Now even in a case where we have got only 9 records as a group, the collector node has to still wait for 5 mins even after receiving all the messages. |
Yes. If you configure it to timeout.
You don't need to.
You just need to know how to identify a message as belonging to a certain collection, and when a message indicates that the collection is complete.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 16, 2016 11:55 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| siva reddy wrote: |
| say i have configured collector node to timeout after 5 mins, expecting that all the 2000 plus records of the group are processed by the mq input node. |
What kind of system have you got where it takes the MQInput node 5 minutes to read 2000 messages? 5 seconds should be closer to the mark, 10 seconds for caution.
| siva reddy wrote: |
| Now even in a case where we have got only 9 records as a group, the collector node has to still wait for 5 mins even after receiving all the messages. |
Or you could adopt the ingenious suggestion of my most worthy associate. If it works. Which you could prove with a PoC.
| siva reddy wrote: |
| so , i am thinking which way --- go with the collector node but cannot avoid wait time for small group of messages |
5 minutes? 5 minutes? How much processing have you got between the MQInput node and the Collector that's slowing it down so much????
| siva reddy wrote: |
| I will do a PoC and validate which option gives the best performance overall |
It's more a question of do you want the database and the associated administration / additional point of failure than performance. While it's fairly unlikely the network link to the db will fail, you need to accept that if it does your flow is immobilized.
Now that's probably an acceptable risk in most organizations, but someone still needs to accept it, based on the business criticality of what this flow does.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| siva reddy |
| Posted: Tue Aug 16, 2016 9:19 pm Post subject: |
|
|
Novice
Joined: 16 Jul 2016
Posts: 11
|
Hi Vitor,
We may not take 5 minutes to process 2000 records, i just gave a indicative number though it was on a higher side, there is no processing done except extracting the payload and consolidating all the messages that are sent as part of the group.
I will validate this through PoC and define the timeout to be configured on the collector node, i would agree with your point that DB brings in additional point of failure. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Aug 16, 2016 10:01 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| siva reddy wrote: |
I will validate this through PoC and define the timeout to be configured on the collector node, i would agree with your point that DB brings in additional point of failure. |
In my experience I have found the DB route to be the most reliable. The flow I described in a previous post was originally developed using a collector node and has very much the same use case as you describe. Initially, it rand for a month and then just stopped working. Onlt a server reboot (this was on V7.0.0.4) would get it working again.
I spent hours and hours looking for reasons why it stopped working but failed. So I re-designed the flow to use a DB instead of a collector node. Took me a couple of days while I was snowed in , in Jordan. Yes it snows there...
Since it was deployed, it has run almost faultlessly. The only changes I had to make were minor ones when we moved the system to IIB 9.0.0.3.
My flow puts the messages (that are in the same logical group) into the DB and uses a 30 second timeout. Once 30 secs have passed since the last message was added to the group, the extract flow is triggered. Thankfully, we have a field right at the start of the message that allows the minimum of parsing of the input message that is perfect for the DB key.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Aug 17, 2016 4:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| siva reddy wrote: |
| We may not take 5 minutes to process 2000 records, i just gave a indicative number though it was on a higher side, there is no processing done except extracting the payload and consolidating all the messages that are sent as part of the group. |
There's no "it may not take 5 minutes". If all your flow does is read the message off the queue and pass the payload down to the Collector node then you should be getting much higher dequeue rates than that, and if you're not you need to be investigating why.
You don't need user code to "extract the payload" of an MQ message and if you think you do you should review your design.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 17, 2016 5:19 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Once more for those watching from the bleachers (or five years from now).
A collection node can have more than one input.
The collection node itself needs some method to identify what collection a given message belongs, when it is sent to any input terminal.
Each input can have different conditions on when that input is "complete".
A collection is deemed complete when one condition on one input terminal says the collection is complete.
One condition can accept an infinite number of messages. This condition can expiry.
One condition can accept exactly one message.
There is a direct mapping between these rules and the rules of how MQ message groups are built.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
