| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| message sequence problem, "Order mode" no effect |
« View previous topic :: View next topic » |
| Author |
Message
|
| arkenstone |
 Posted: Fri Mar 20, 2009 12:51 pm Post subject: message sequence problem, "Order mode" no effect |
 |
|
Newbie
Joined: 08 Jul 2008
Posts: 8
|
Hi,
Here is my flow architecture:
FLOW 1:
MQinput (Q1) --> Computenode1 --> MQoutput (Q2)
FLOW 2:
MQinput (Q2) --> Computenode2 containing very heavy logic --> MQoutput (Q3)
In Q1, i'm receiving XML messages with this structure (simplified):
<data>
<user>a user ID</user>
<sequence>a sequence number</sequence>
</data>
Several messages with different user IDs can be put in Q1 in a very short delay (in the worst case at the same time).
I have a business rule which says that, for a given user ID, output message sequence should be the same as input sequence. Unfortunately, it is not negotiable.
A second important constraint is that we execute a very time consuming operation in the FLOW 2 from the message content (PDF generation), and regarding the amount of message to be processed each day (thousands) we need to have several instance of the FLOW 2.
So what i've tried to do is:
- In FLOW 1 Computenode1, I replace the MQMD.UserIdentifier of the incoming message by the value contained in the <user> tag (padded with spaces to fill 12 characters, the length of UserIdentifier).
- In FLOW 2, in the MQinput node, I have set the "Order mode" to "By User ID".
Regarding the IBM doc (http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r0m0/index.jsp?topic=/com.ibm.etools.mft.eb.doc/ac00350_.htm), that should perfectly fit my needs:
If you set Order Mode to By User ID, the node ensures that messages from a specific user (identified by the UserIdentifier field in the MQMD) are processed in guaranteed order. A second message from one user is not processed by an instance of the message flow if a previous message from this user is currently being processed by another instance of the message flow.
The problem is: despite of my node configuration, messages still arrive in random order in the output queue (Q3), like if my "Order mode = By user ID" was ignored.
Additional info:
- I have no additional instances for FLOW 1
- I have 15 additional instances for FLOW 2
- my test set contains 5 messages for 3 different users (15 messages in all)
- I have tried "Order mode = By Queue Order" in the MQinput of Q2, and the result is the same => messages are in random order in Q3
- I have checked messages put in Q2 by FLOW 1, and the UserIdentifier is set correctly (set from the value contained in XML message)
- my 15 messages are put in Q1 in less than a second, and the time consuming logic in FLOW 2 takes about 3-4 seconds for each message
Another weird observation:
My 15 messages pass through the FLOW 1 in few milliseconds. When they reach Q2 they are all absorbed by the different instances of FLOW 2 directly (I would have expected that FLOW 2 waits to have finished processing the first message for a given user ID before absorbing the next one, and so that only 3 instances of FLOW 2 could be active at the same time, as I have only 3 different users in my test set).
Am I missing something ?
Thanks.
Regards |
|
| Back to top |
|
 |
| rekarm01 |
| Posted: Fri Mar 20, 2009 2:52 pm Post subject: Re: message sequence problem, "Order mode" no effe |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| arkenstone wrote: |
So what i've tried to do is:- In FLOW 1 Computenode1, I replace the MQMD.UserIdentifier of the incoming message by the value contained in the <user> tag (padded with spaces to fill 12 characters, the length of UserIdentifier).
- In FLOW 2, in the MQinput node, I have set the "Order mode" to "By User ID".
|
The FLOW 1 MQOutput node "Message Context" property should be either "Set Identity" or "Set All". Consult the WMQ and WMB documentation for details. To verify that FLOW 1 sets the MQMD headers properly, stop the FLOW 2 message flow, and browse its input queue.
Not to split hairs, but that URL is for the WebSphere Event Broker, not for the WebSphere Message Broker.
| arkenstone wrote: |
- I have tried "Order mode = By Queue Order" in the MQinput of Q2, and the result is the same => messages are in random order in Q3
- I have checked messages put in Q2 by FLOW 1, and the UserIdentifier is set correctly (set from the value contained in XML message)
|
Is either FLOW 1 or FLOW 2 deployed to multiple execution groups? |
|
| Back to top |
|
|
| arkenstone |
| Posted: Fri Mar 20, 2009 3:15 pm Post subject: Re: message sequence problem, "Order mode" no effe |
|
|
Newbie
Joined: 08 Jul 2008
Posts: 8
|
| rekarm01 wrote: |
| arkenstone wrote: |
So what i've tried to do is:- In FLOW 1 Computenode1, I replace the MQMD.UserIdentifier of the incoming message by the value contained in the <user> tag (padded with spaces to fill 12 characters, the length of UserIdentifier).
- In FLOW 2, in the MQinput node, I have set the "Order mode" to "By User ID".
|
The FLOW 1 MQOutput node "Message Context" property should be either "Set Identity" or "Set All". Consult the WMQ and WMB documentation for details. To verify that FLOW 1 sets the MQMD headers properly, stop the FLOW 2 message flow, and browse its input queue.
|
yes, i dit it, and i checked the messages
| arkenstone wrote: |
| - I have checked messages put in Q2 by FLOW 1, and the UserIdentifier is set correctly (set from the value contained in XML message) |
Touché. The url for Message Broker (the content is the same):
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r0m0/index.jsp?topic=/com.ibm.etools.mft.doc/ac00350_.htm
| rekarm01 wrote: |
| arkenstone wrote: |
- I have tried "Order mode = By Queue Order" in the MQinput of Q2, and the result is the same => messages are in random order in Q3
- I have checked messages put in Q2 by FLOW 1, and the UserIdentifier is set correctly (set from the value contained in XML message)
|
Is either FLOW 1 or FLOW 2 deployed to multiple execution groups? |
I tried both in the same bar file and same Execution Group, and in 2 different bars and 2 different Execution Group => result is the same. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Mar 20, 2009 6:21 pm Post subject: Re: message sequence problem, "Order mode" no effe |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| arkenstone wrote: |
| rekarm01 wrote: |
| Is either FLOW 1 or FLOW 2 deployed to multiple execution groups? |
I tried both in the same bar file and same Execution Group, and in 2 different bars and 2 different Execution Group => result is the same. |
To clarify:- Is FLOW 1 deployed to more than 1 execution group?
- Is FLOW 2 deployed to more than 1 execution group?
The answer to both questions should be "no".
Some other questions:- Which version of broker is running?
- Are there any other message flows reading from Q1 or Q2?
- Does FLOW 2 work as expected with no additional instances?
|
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Mar 20, 2009 10:58 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
You seem to have some sort of Message Affinity (wrt messages with a particular id)
This is generally a bad idea for Broker work. If you search on this in this forum, you will soon get the idea why it is a bad idea.
If you want to guarantee that a set of messages is processed in a particular order you should look at grouping the messages (At an WMQ Level) then you can read them by group ID.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| arkenstone |
| Posted: Sat Mar 21, 2009 2:46 am Post subject: Re: message sequence problem, "Order mode" no effe |
|
|
Newbie
Joined: 08 Jul 2008
Posts: 8
|
| rekarm01 wrote: |
To clarify:- Is FLOW 1 deployed to more than 1 execution group?
- Is FLOW 2 deployed to more than 1 execution group?
The answer to both questions should be "no".
|
The answer to both is no.
First I tried with both message flows in the same EG.
Then I tried with FLOW 1 in first EG, and FLOW 2 in a second EG.
| rekarm01 wrote: |
Some other questions:- Which version of broker is running?
- Are there any other message flows reading from Q1 or Q2?
- Does FLOW 2 work as expected with no additional instances?
|
I think it's version 6.1, i could ask the infrastructure guys to be sure.
No other message flow is reading from Q1 or Q2.
FLOW 2 works fine with no additional instances (messages are treated one by one and so the sequencing is kept), but for performance reasons, i need at least about 10 instances in parallel.
Last edited by arkenstone on Sat Mar 21, 2009 3:15 am; edited 1 time in total |
|
| Back to top |
|
|
| arkenstone |
| Posted: Sat Mar 21, 2009 3:13 am Post subject: |
|
|
Newbie
Joined: 08 Jul 2008
Posts: 8
|
| smdavies99 wrote: |
You seem to have some sort of Message Affinity (wrt messages with a particular id)
This is generally a bad idea for Broker work. If you search on this in this forum, you will soon get the idea why it is a bad idea.
If you want to guarantee that a set of messages is processed in a particular order you should look at grouping the messages (At an WMQ Level) then you can read them by group ID. |
Yes, sort of, but I don't really have a choice unfortunately. This design involves a lot of other components upstream. Then changing the design would have a lot of consequences.
And grouping messages is not applicable regarding the business constraints.
Anyway, for my point of view, the IBM documentation is clear, what is described there seem to fit exactly my needs. So why it doesn't behave as expected ?
I found topics on this forum with guys having the same constraint as me and for whom the "Order mode = By User ID" seems to work as expected :
http://www.mqseries.net/phpBB2/viewtopic.php?t=29095
http://www.mqseries.net/phpBB2/viewtopic.php?t=36377
http://www.mqseries.net/phpBB2/viewtopic.php?t=31334
What Jeff the "Grand Poobah" is describing in his last post of the last URL is exactly what I need...but for some reason, my flow doesn't behave as expected.
what i expect supposing the content of messages in Q2 is the following:
1: User1 / sequence 1 => FLOW 2 instance 1
2: User2 / sequence 1 => FLOW 2 instance 2
3: User3 / sequence 1 => FLOW 2 instance 3
4: User1 / sequence 2
5: User2 / sequence 2
6: User3 / sequence 2
7: User1 / sequence 3
8: User2 / sequence 3
9: User3 / sequence 3
And then, each instance waits to finish processing his message before picking the next one with the same userID. Thus, as i have messages of only 3 different users, i can't have more than 3 instances active at the same time.
now, what seems to happen despite of the "Order mode = By user ID":
1: User1 / sequence 1 => FLOW 2 instance 1
2: User2 / sequence 1 => FLOW 2 instance 2
3: User3 / sequence 1 => FLOW 2 instance 3
4: User1 / sequence 2 => FLOW 2 instance 4
5: User2 / sequence 2 => FLOW 2 instance 5
6: User3 / sequence 2 => FLOW 2 instance 6
7: User1 / sequence 3 => FLOW 2 instance 7
8: User2 / sequence 3 => FLOW 2 instance 8
9: User3 / sequence 3 => FLOW 2 instance 9
and as a consequence, messages are outputed in random order, and not the order in Q2. |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Mar 23, 2009 2:26 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
This is targeted at the next v6.1 fixpack:
IC51113 - MQINPUT NODE DOES NOT HONOR "BY QUEUE ORDER" WITH ADDITIONAL INSTANCES
If you need it urgently, you could raise a PMR and request it
This was found and fixed in v6.0, so it should already be fixed in v6.1:
V6.0 CSD09 and V6.1 : IC57560 IN-ORDER DELIVERY OF MESSAGES NOT PRESERVED |
|
| Back to top |
|
|
| arkenstone |
| Posted: Mon Mar 23, 2009 4:57 am Post subject: |
|
|
Newbie
Joined: 08 Jul 2008
Posts: 8
|
Sometimes this Broker makes me mad...this morning I started a new Flow project, i've copy-pasted my code from the old to the new project, configured my nodes exactly the same way ("Order mode = By User ID")...and now it's working as expected.
Looks like something was corrupted in the old project...
Hi Kimbert, actually my problem was not with the "By Queue order" but with the "By User ID". I tried with the "By Queue Order" just to see how the flow would behave.
Anyway, thanks to all folks who have tried to help me. |
|
| Back to top |
|
|
| lamtbs |
| Posted: Sun May 24, 2009 1:15 am Post subject: |
|
|
Acolyte
Joined: 07 Nov 2005
Posts: 57
Location: Hong Kong
|
Hi,
may I know what platform you are running on?
Tried it on my Window Broker 6.1.0.3, it's NOT running in sequence as expected, restarted broker, rebuild workspace, restarted Windows, still doesn't work.
Then I found this APAR - IC57560
IN-ORDER DELIVERY OF MESSAGES NOT PRESERVED WHILE USING THE OPTION ORDER MODE ='BY USER ID'
seems it has not been working, wonder how you made it work??
Besides the Order Mode = By User ID, have you set any other parameter to make it work??
Thanks! |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
