| Author |
Message
|
| siva reddy |
 Posted: Sat Aug 13, 2016 11:15 pm Post subject: Facing issues when processing group mesaages |
 |
|
Novice
Joined: 16 Jul 2016
Posts: 11
|
Hi All,
I have a requirement to process group messages based on group id.
Intially i thought to use collector node followed by mq input node which will trigger the flow once all the group messages arrived.
But to use collector node we have to specify when it is supposed to trigger, either based on number of messages nor time interval. I am not sure on both the criterias.
So i thought of going with mq get node to pull all the messages from the queue once flow is triggeed by mq input node (triggers flow once it receives all the group messages) but mq get node is not able to pick up the first message as it is pointing to second message.
Message 1: already picked up ny mq input node.
Message 2: expecting that mq get node will pick it up.
Message 3: mq get node picking up third message
So i thougt of using another mq node which will browse the first messgae using mqbrowse option in mqgmo so that the next coming mq get node will read that message and will repeat this process till the end of the last message.
But using one more mq get node seems like perfomance overhead.
Can you please suggest me if is there any best approach to achieve this.  |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Sun Aug 14, 2016 5:57 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
When the flow is idle it has ZERO information about the GroupID and/or the number of messages in the Group.
When the first message (or group of messages) is written (and committed) to the Queue the Flow wakes up and reads the first message in the group.
This allows the flow to get the GroupId
Then the MQGET node will read the next message in the group based upon the GroupID that was present in the MQMD/Message Properties of the first message.
The Flow has to store the message data somewhere until all messages have been read. (I'd probably put them in a DB Table with the Primary Key being the Timestamp and the GroupID.
Then it reads the next message etc until the Last Message is Encountered.
Then all the read messages can be processed.
There really is no need to browse anything. Just accept that the first message in the group has been read by the MQINPUT Node that starts the thread.
IF you know that the number of messages in a group will be fairly small then you should not run into any major issues.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| siva reddy |
| Posted: Tue Aug 16, 2016 12:54 am Post subject: |
|
|
Novice
Joined: 16 Jul 2016
Posts: 11
|
As i have mentioned earlier, the number of messages sent in a group is not constant and will differ , hence we cannot use the number of messages as an option to loop through the MQget node.
We are trying to process the messages are below -
MQInput - Compute1- MQGet1 - Compute2
MQInput node is configured to send the group messages only after receiving the last message, compute node takes the first message as a trigger and loops through the MQGET node to pull the remaining messages till the last message indicator is populated.
we are facing challange handling the cursor, as the first message is sent through MQIinput node the cursor is pointed to second message, and hence when the control comes to MQget node we are only able to get the third message. As we are doing a read and not browse we cannot alter the cursor to point to previous message.
Any thoughts on how this can be handled in a better way ?
Regards,
Siva |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 16, 2016 2:38 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| siva reddy wrote: |
As i have mentioned earlier, the number of messages sent in a group is not constant and will differ , hence we cannot use the number of messages as an option to loop through the MQget node.
|
And why not? You can count and you can check for last in group right?
So you will know if you have all the messages in the group.
Each message in the group should also have an order number that defines its position in the group. If the total number of messages received for the group does not match the order number of the last message in group you are in trouble...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Aug 16, 2016 3:45 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
IF we are really talking about MQ message groups then every message will have an indicator that tells you if the message you have read was the least message or if you have to read another one.
This is HOW MQ was designed to handle message groups.
If there is some other indicator that is contained in the message then you will have to roll your own method of finding the last message in the group.
This post might help
http://www.mqseries.net/phpBB2/viewtopic.php?t=1705&sid=07ed682db5df3e305d2261b462d08483
This post dates from 2002 so this feature has been around for a long time. It does work.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 16, 2016 3:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Also, if you need to handle the contents of the group as a whole - if it's an XML document for ex - you will need to store the contents of each message so that it lasts between instances of the flow.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| siva reddy |
| Posted: Tue Aug 16, 2016 7:55 am Post subject: |
|
|
Novice
Joined: 16 Jul 2016
Posts: 11
|
Thanks for the reply,
As part of the message i am getting
1. Group ID
2. Message Sequence Number (messge position in the group)
3. Last Message Indicator ( helps to identify the last mesage in the group)
My message flow;
MQInput - Compute1- MQGet1 - Compute2
In compute1 i am running a loop till the last message of the group using Last message indicator.
Mq input node already picked up the messge with message sequence number 1.
I am expecting that mq get1 will pick up the message with message sequence number 2.
But it is pulling the message with message sequence number 3
Can we make some changes to mq get1 to pull the messages from message sequense number 2 onwards. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 16, 2016 8:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| siva reddy wrote: |
| But it is pulling the message with message sequence number 3 |
Which means something has pulled message sequence number 2.
| siva reddy wrote: |
| Can we make some changes to mq get1 to pull the messages from message sequense number 2 onwards. |
No. There's no difference between the MQGet node and an MQGet API call. You can't specify which message it's going to pick except in terms of match criteria (like group id). It will always pick the first available message that matches, possibly the highest priority available message that matches if the queue is set to deliver in priority order not FIFO.
If the MQGet node is giving you message 3, then for some reason message 2 is not available when it asks. Possible reasons include (but are not limited to):
- another instance of your flow
- another application reading that queue
- message 2 not being committed by the sending application
- message 3 has a higher priority than message 2
- etc
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| siva reddy |
| Posted: Tue Aug 16, 2016 9:38 am Post subject: |
|
|
Novice
Joined: 16 Jul 2016
Posts: 11
|
Thanks for the response vitor.
The queue is only used by the IIB interface , no other application reads the messages from the queue.
Sending application sends all the messages in the group with the same priority (rather no difference in the message priority) all the messages are sent in the same way and hence there should not be any issue about message not being committed.
We have only single instance of the flow running.
What are the best practices when we have to consolidate messages using group id (in this case we cannot use collector node) , specially looping over the MQGET node. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 16, 2016 9:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Stop the flow.
Have the app send a group.
Confirm that all of the messages are in the group and are committed.
Confirm that the messages on the queue are "in the right order".
Adjust your solution to not rely on messages being "in the right order".
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 16, 2016 9:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| siva reddy wrote: |
| TWhat are the best practices when we have to consolidate messages using group id (in this case we cannot use collector node) , specially looping over the MQGET node. |
The best practice for looping over the MQGet node is don't. It's accepted to be the fastest & most reliable way to run the EG out of memory, causing it to crash (each iteration adds to the stack, which eventually blows).
I'd also question your assertion that you can't use the Collector node. I accept you don't know the number of messages in a group, but you do know that the MQInput node will not start passing them down until the group is complete (because you will have set the All Messages Available option). Thus you can set the Collector to assemble by group id, and consider the group complete after a short timeout interval.
Alternatively you can read the messages with an MQInput node, store them in a database / global cache / other semi-permanent storage with the group id as a key and retrieve the block of messages when you detect the Last Message in Group indicator.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Aug 16, 2016 10:13 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
[quote="Vitor"]
| siva reddy wrote: |
The best practice for looping over the MQGet node is don't. It's accepted to be the fastest & most reliable way to run the EG out of memory, causing it to crash (each iteration adds to the stack, which eventually blows).
|
While I agree with my colleague on this, if you know that the number of messages in the group is going to be less than 100 then with 64bit EG's, you can loop over the MQGet node providing that each message is not that big.
So, you have to develop a solution that will use some way of preserving the read dats until you get all the messages in the group.
I have created one flow that processes some 400K+ messages per day that has in effect a message grouping (but not MQ groups). This data is stored in a DB until the last message is read. Then the data is read from the DB and processed.
It has been running for 3+ years now and has never had an issue. The size of the groups varies between 1 and 56 (highest recorded count so far).
There are other ways to do this including but not limited to the Collector Node.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| siva reddy |
| Posted: Tue Aug 16, 2016 10:22 am Post subject: |
|
|
Novice
Joined: 16 Jul 2016
Posts: 11
|
Agree with you Vitor,
looping MQget is a very bad practice and can bring down the execution group if not handled properly.
Now coming to the point of using collector node, its always the first option to use but there will be significant wait time.
Say there are a group of 9 records , since we dont have the exact number of records we only have the option of configuring for a some time for collector node to wait and process the messages.
Some times there can be about 2000 records in a group, every messages is not more than 1 KB. So in this case we have to configure collector node to wait for all the 2000 records or max records expected. this would mean every time the collector record has to wait for the configured time even if all the messages have reached. this would reduce the overall performance
i think we have to live with it if we have to use collector node, or do you see other optimal approach using collector node
the alternate and better approach would be storing the records in DB and extracting them all using the group id as the key. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 16, 2016 10:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| siva reddy wrote: |
| Some times there can be about 2000 records in a group, every messages is not more than 1 KB. So in this case we have to configure collector node to wait for all the 2000 records or max records expected. this would mean every time the collector record has to wait for the configured time even if all the messages have reached. this would reduce the overall performance |
Exactly how long would the Collector have to wait for the MQInput node to read 2000 messages off the queue? Given that it won't start reading them (and starting the collection timer) until the group is complete? And if you're talking about the time delay between record 1 and record 2000 arriving on the queue, there's no way to eliminate that at the receiving end given that you need to process the entire group.
| siva reddy wrote: |
| the alternate and better approach would be storing the records in DB and extracting them all using the group id as the key. |
As I indicate above. Assumes access to a database, which the Collector solution does not require.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 16, 2016 11:00 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The collector node only needs to know that a message has started a new collection ,what collection a message belongs to, and when a message has ended a collection.
None of these require the MQInput node that feeds the collection to wait. None of these require that the Collection node wait.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
