| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Using MQBROWSE with message sequencing |
« View previous topic :: View next topic » |
| Author |
Message
|
| mqjeff |
 Posted: Thu Jul 15, 2010 11:44 am Post subject: |
 |
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Bruce - I think the situation is this.
***********
A set of applications have been written that produce messages in an ordered sequence, and this sequence is logically necessary to process the messages.
All of these applications have been running on the same qmgr.
A new application that also sends messages in logical sequence has been added that is now exchanging messages over MQ channels, from a remote queue manager.
This necessarily introduces some possibility that messages will arrive out of sequence.
AaronMenchions is looking to create a mechanism to resequence messages, more or less as they arrive, without changing the sending or receiving applications.
**********
Introducing additional queues won't do anything to affect the fact that MQ channels can deliver messages out of sequence occasionally.
Particularly on zed, where queues can be indexed by correlation id, a better solution than using MQBROWSE to read a queue would be to ensure that the sending application puts the sequence number into the correlation ID, and that the sequence numbers are atomically increasing. The "front porch" application can then get "the next message" by incrementing the correlation ID of the last message. No need to browse.
A better solution than the above is to use an intermediate staging table to hold a message that is NOT the next message. I.e. read each message, check to see if it has "the next" sequence number, if so, forward it. if not, stick it in the table, and then check to see if the table DOES hold "the next" message.
Using MQBROWSE will rapidly lead to deep queues if you're not careful, and is potentially more intensive and error prone.
The ideal solution is to rework the whole mess so that there is no sequence dependency in the first place. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Jul 15, 2010 1:50 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| mqjeff wrote: |
| The ideal solution is to rework the whole mess so that there is no sequence dependency in the first place. |
 As is the application is not scalable. Taking out the sequence dependency makes the application scalable and allows for better throughput. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jul 15, 2010 2:02 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Our senior support even contacted IBM, and they just said "we do not guarentee message order when you cross Q manager". |
Nor can IBM ensure message order when multiple apps concurrently put messages to the same queue (interleaving).
You mentioned lost packets. Is that the issue? Are you experiencing network failures? If so, then you have TCP/IP stack issues - again, not an WMQ issue at all.
Read the APR and APG looking for "message order". This is very well documented.
[edit] And it's not that ibm/wmq cannot guarantee message order; rather, message order can be guaranteed - but with exceptions (like messages in/out of UofW's, messages that ended up in a DLQ, ...).
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sumit |
| Posted: Fri Jul 16, 2010 2:18 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| AaronMenchions wrote: |
| "we are operating just fine, it's the middle man that's messing up". |
Isn't that the most common comment that we come accross!!
btw, I have seen such scenarios where in sequence of messages (in the very similar way as yours) needs to be retained.
May I know how are you receiving messages on your "front porch" queue? Is there just one sender qmgr which is sending messages, or there are number of intermediate qmgrs?
Can you check and confirm if you are using clustered scenario or svr-rqstr type configuration?
Below are the steps that I would recommend (though I am not very good with Mainframe, but these are normal MQ settings)
1. Ensure that you are using persistence messages. - to prevent message loss.
2. Ensure that the get application is not losing any message during GET operation.
3. Check that the message priority is same for similar messages i.e. same for all FRED and same for all BARNEY (also, they are not mutually exhastive)
4. Specify a fixed path for message route. (Are you using load balancing, though doesn't look like)
_________________
Regards
Sumit |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Fri Jul 16, 2010 5:20 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
| sumit wrote: |
May I know how are you receiving messages on your "front porch" queue? Is there just one sender qmgr which is sending messages, or there are number of intermediate qmgrs? |
I'll answer the questions that I know. 
the front porch receives messages from at least 2 qmgrs, of which one apparently is on another machine. This has always been the case however, and it was only since last fall that messages started to arrive out of sequence regularly from the off-machine qmgr.
| Quote: |
| 1. Ensure that you are using persistence messages. - to prevent message loss. |
All of our messages we use here are persistant I beleive, at least for the queue I'm talking about.
| Quote: |
| 2. Ensure that the get application is not losing any message during GET operation. |
Losing messages is very rare. As i posted in antoher reply, the only time it's happened for this queue was when one of the queue configurations was defined incorrectly. So, normal day to day stuff never goes missing..... *knock on wood*
| Quote: |
| 3. Check that the message priority is same for similar messages i.e. same for all FRED and same for all BARNEY (also, they are not mutually exhastive) |
For our scenario, there is no relationship between messages from FRED or BARNEY, just that messages from FRED are 4 messages per UoW and they need to be in sequence. Within FRED, all messages are set to the same priority. That was one of the first things we checked when the problem surfaced in the fall.
4.
| Quote: |
| Specify a fixed path for message route. (Are you using load balancing, though doesn't look like |
)
Yes, we use load balancing here. I'm not up on the techincalities of it all, but we have multiple "SYS's" on the mainframe, and also jobs can run in one of several CICS regions.
I hope our solution will work. I'm sure if we had time all over again, the MQ stuff would be set up far differntly, but the foundation was laid 13 or 14 years ago, which is far before my time. it's not easy to change things, especially in the government when so many transactions rely on it! |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jul 16, 2010 5:39 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Losing messages is very rare. As i posted in antoher reply, the only time it's happened for this queue was when one of the queue configurations was defined incorrectly |
Exactly what does this mean? How was it defined incorrectly? MAXDEPTH too low, so messages ended up in the DLQ? Some other attribute?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Fri Jul 16, 2010 5:52 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
| bruce2359 wrote: |
| Quote: |
| Losing messages is very rare. As i posted in antoher reply, the only time it's happened for this queue was when one of the queue configurations was defined incorrectly |
Exactly what does this mean? How was it defined incorrectly? MAXDEPTH too low, so messages ended up in the DLQ? Some other attribute? |
Some new queues were created for the release of our system in April. When they were created, the support group that handles this here made a mistake in the Storage Class name, which I think made it impossible to send messages to that queue.
Our server handles most types of errors, doing rollbacks and such, but it was never coded for a case of an incorrect config of this type. So, the message was sent to a queue that didn't "work", and the few messages were lost.
We have since patched our server to include this type of check in the future, so hopefully messages shouldn't be lost....
Last edited by AaronMenchions on Fri Jul 16, 2010 5:56 am; edited 1 time in total |
|
| Back to top |
|
|
| sumit |
| Posted: Fri Jul 16, 2010 5:54 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| AaronMenchions wrote: |
| Yes, we use load balancing here. I'm not up on the techincalities of it all, but we have multiple "SYS's" on the mainframe, and also jobs can run in one of several CICS regions. |
Does it mean that your applicaiton(s) look for a sequence of messages (i.e. 1,2,3,4,5,etc) in all the instances of front porch queue?
With load balancing in place, MQ will share the message load amongst all the available 'front porch' queues. Hence, it's required that your app(s) looks for message in all the available queues.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jul 16, 2010 5:59 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| a mistake in the Storage Class name, which I think made it impossible to send messages to that queue. |
I gather that this got into PRODuction without being TESTed and QAd.
Yes, invalid storage class (STGCLASS) attributes will cause puts by MCA to fail.
So, some of the messages ended up in the DLQ? This error should have been captured in the CHIN syslog.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Fri Jul 16, 2010 6:07 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
There is just one front porch queue for this application. There are more then one applications that put to this front porch queue.
Maybe "behind the scenes" this is load balanced and replicated by the system? but if so, the same is true for the GETting server too.
It's invisible to us, anyway, though maybe it helps explain how stuff is arriving out of sequence if one version of the queue is running faster and gets it's message beofre another? |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Fri Jul 16, 2010 6:10 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
| bruce2359 wrote: |
| Quote: |
| a mistake in the Storage Class name, which I think made it impossible to send messages to that queue. |
I gather that this got into PRODuction without being TESTed and QAd.
Yes, invalid storage class (STGCLASS) attributes will cause puts by MCA to fail.
So, some of the messages ended up in the DLQ? This error should have been captured in the CHIN syslog. |
Well, the new setup was tested in other envrionments, and since things were defined correctly there, it all worked. This problem happened during our pre-release sign-on (thankfully) so very limited damage was done.
We checked the DLQ and nothing was put there. I will admit I haven't heard of the CHIN syslog however? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jul 16, 2010 6:12 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| though maybe it helps explain how stuff is arriving out of sequence if one version of the queue is running faster and gets it's message beofre another? |
Queues don't have velocity.
The is ample documentation on doing problem-determine on so-called "missing or lost messages".
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sumit |
| Posted: Fri Jul 16, 2010 6:28 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| AaronMenchions wrote: |
| There is just one front porch queue for this application. |
Given that you have just one destination queue and probable load balancing (refereing to MQ load balancing here, acheived by cluster), you may try defining a fixed path for your message so that all messages use ONE AND ONLY ONE path to reach destination.
Use defbind(open) at sender with target to a fixed destination queues at all the intermediate qmgrs.
The intention is to tell all the messages to follow only THIS path and no else.
With load balancing in picture, different mesages can take different route (as per algo or network availability) which may at the end is realised as 'out of sequence' messages.
You would like to try this on your test env. first.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jul 16, 2010 6:37 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
It seems that the OP is faced with message affinity, where one or more messages comprise a transaction.
Message affinity requires that all related messages arrive at the same queue, in order (in this case), and are not interleaved with other messages.
Among other restrictions, message affinity precludes workload distribution.
Again, this is all wonderfully documented in such titles as "when message order must be preserved."
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sumit |
| Posted: Fri Jul 16, 2010 6:49 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Thanks bruce2359, I'll look into the mannuals for message affinity
However, it makes me think when you say-
| bruce2359 wrote: |
| and are not interleaved with other messages. |
Whereas one of the examples that OP gave had-
| Quote: |
| So, for example, say the queue has this: FRED1,2,4,BARNEY1,2,3,4FRED5,6,7,8. |
Reading it further makes me feel like, it's fine to have mixed mesages from different app as far as messages from same app are in sequence.
I also believe that OP mentioned 2 problems-
1. Missing message (where message 3 was missing)
2. Out of sequence message.
Did I miss something? 
_________________
Regards
Sumit |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
