| Author |
Message
|
| csongebalazs |
 Posted: Wed Nov 11, 2009 4:56 am Post subject: message group handling |
 |
|
Voyager
Joined: 30 Jan 2004
Posts: 78
|
Hi,
I need some advice, how can i solve the following problem with WMB.
I have to process input messages which contains group id, seq number and total number of messages in usr part of the MQRFH2 header. The total number always different and the incoming order of messages not equal the seq number in their header.
The message processing flow must be run more then 1 instances.
I have to detect that situation, when the last message arrived.
I think, the collection node is not perfect for this purpose, because i cannot set collection input quantity in runtime (based on header total number info).
If i write messages into database, and check the number of recorded messages in case of every message incoming, can it be bad in case of paralell processing?
Or is it any built in node, who can handle this situation?
Regards
Balázs |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Wed Nov 11, 2009 5:51 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Take a look at using the MQMD (version 3) This has all the featurs you need for grouping messages including a 'Last Message of Group'.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Nov 11, 2009 7:28 am Post subject: Re: message group handling |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| csongebalazs wrote: |
| I have to detect that situation, when the last message arrived. |
This is known as message affinity, and there are endless posts on why this is a bad thing. One of the many reasons for this is it's very hard to run parallel processing (like more than one instance) on such a model.
| csongebalazs wrote: |
| I think, the collection node is not perfect for this purpose, because i cannot set collection input quantity in runtime (based on header total number info). |
I think you're right
| csongebalazs wrote: |
| If i write messages into database, and check the number of recorded messages in case of every message incoming, can it be bad in case of paralell processing? |
It can and will.
I'm also a bit unclear on your model. If you've got all the messages stored in a database, then confirmed that the last message of a group has arrived, are you then able to process the messages out of sequence?
| csongebalazs wrote: |
| Or is it any built in node, who can handle this situation? |
As my associate correctly points out, the MQInput node can handle this but that relies on these details being stored in the MQMD (where IBM have put the group id and other details). So you'd need the application which puts the message to modify it's behaviour, or a pre-screening flow to move the details from RFH2 to MQMD.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 11, 2009 7:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| MB v7 has Sequence and Resequence nodes that may be of assistance. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Nov 11, 2009 7:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| MB v7 has Sequence and Resequence nodes that may be of assistance. |
Again I must research these shiny new features.... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| csongebalazs |
| Posted: Thu Nov 12, 2009 1:49 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2004
Posts: 78
|
| smdavies99 wrote: |
| Take a look at using the MQMD (version 3) This has all the featurs you need for grouping messages including a 'Last Message of Group'. |
MQMD grouping facility is used to reunite big messages which was splitted to many parts in the sender side. Am I right? If yes, this is not my solution, because i receive independent messages which has a common purpose.
| Vitor wrote: |
| I'm also a bit unclear on your model. If you've got all the messages stored in a database, then confirmed that the last message of a group has arrived, are you then able to process the messages out of sequence? |
I have to process all messages 1 by 1. They can be processed different way, depends on its content, etc., but all recorded to a database. The arrival of the last message initiate futher procesing steps, a report generation, which related to all received message, not the last or actually received.
So this is the background 
Anyway, the problem must be independent this background. How can detect last message during paralell processing...
| mqjeff wrote: |
| MB v7 has Sequence and Resequence nodes that may be of assistance. |
I will check it, but the system environment is in WMB6.1 verion in our customer, and i think they will not change to brand new WMB7.0 in the short term future. So i have to use WMB6.1 and ORACLE 10 Databse capabilities (without gateway). |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 12, 2009 7:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| csongebalazs wrote: |
| MQMD grouping facility is used to reunite big messages which was splitted to many parts in the sender side. Am I right? |
No, that's segmentation. Message grouping does what you need.
| csongebalazs wrote: |
| Anyway, the problem must be independent this background. How can detect last message during paralell processing... |
If the putting application invokes message grouping, then the messages won't be made available by the queue manager until the last message arrives. Now AFAIK the entire group has to be read by the same getting application and I'm guessing the same is true for an MQInput node - opposing views welcomed here. So you're be looking for a pre-processing flow to fan the group out into parallel instances.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Nov 12, 2009 7:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Message grouping would merely ensure that the whole set of messages was available, and also ensure that no processing of a given group could occur until that was true.
The MQInput node isn't going to magically aggregate or collect them, and I'm not entirely sure it will return them in order - but it might do at least that.
I had thought that collector could be used to collect until you sent it a LAST_MSG_IN_GROUP, but perhaps that's not the case? I know you need at least one fixed count element, which should be the LAST_MSG_IN_GROUP signal.
But getting back to the original requirement.
Is the need to process them in order? Or merely to detect that the last message has arrived? Is the need to process no messages UNTIL the last message arrives?
You should still be able to take an MQ Grouping/collector approach, in that you feed messages as they come through into a queue with a modified set of MQMD flags based on the business level data, and then loop over an MQGet node when you know you've found the last of the set. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 12, 2009 7:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| The MQInput node isn't going to magically aggregate or collect them, and I'm not entirely sure it will return them in order - but it might do at least that. |
Surely it'll only return those members of the group? And you can request logical order I think?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Nov 12, 2009 8:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| The MQInput node isn't going to magically aggregate or collect them, and I'm not entirely sure it will return them in order - but it might do at least that. |
Surely it'll only return those members of the group? And you can request logical order I think? |
Yes.
But not "at once". It's going to start a new flow instance for each individual message. So you won't get all of the messages in the same flow instance. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 12, 2009 8:03 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| It's going to start a new flow instance for each individual message. So you won't get all of the messages in the same flow instance. |
Ah.
But this seems to be what the original poster is looking for (all the messages of a group processed in parallel once the last message has arrived) so this could be the answer.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| csongebalazs |
| Posted: Fri Nov 13, 2009 1:05 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2004
Posts: 78
|
| mqjeff wrote: |
Message grouping would merely ensure that the whole set of messages was available, and also ensure that no processing of a given group could occur until that was true.
The MQInput node isn't going to magically aggregate or collect them, and I'm not entirely sure it will return them in order - but it might do at least that.
I had thought that collector could be used to collect until you sent it a LAST_MSG_IN_GROUP, but perhaps that's not the case? I know you need at least one fixed count element, which should be the LAST_MSG_IN_GROUP signal.
But getting back to the original requirement.
Is the need to process them in order? Or merely to detect that the last message has arrived? Is the need to process no messages UNTIL the last message arrives?
You should still be able to take an MQ Grouping/collector approach, in that you feed messages as they come through into a queue with a modified set of MQMD flags based on the business level data, and then loop over an MQGet node when you know you've found the last of the set. |
During parallel message processing doesn't meter the order. It contains only data validation and storing to DB. In usr part of the MQRFH2 contains sequence info, which recorded too, so in case of report generation, data can be selected in right order.
So, if i understand right, during parallel processing, i have to set MQMD grouping info, and put all messages out a temp queue, than make a new flow, which MQ input queue waiting all messages, then read those messages and throw away all except last, which will start the common processing. Right?
In WMB documentation they says the following:
To ensure that the message flow that processes group messages does not wait for unavailable messages:
Avoid deploying additional instances of a flow that retrieves group messages
So parallel message processing flow will run additional instances, but that one which contains group waiter MQinput node, will work alone.
-------------------------
I tried to use WMB7 resequence node, and it can be a good solution to solve this problem too in WMB7 environment. (But i have to work in 6.1 ).
-------------------------
Thanks for help. |
|
| Back to top |
|
|
|
|
