| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Using MQBROWSE with message sequencing |
« View previous topic :: View next topic » |
| Author |
Message
|
| AaronMenchions |
 Posted: Wed Jul 14, 2010 11:25 am Post subject: Using MQBROWSE with message sequencing |
 |
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
I'll admit up front I'm not the developer, but rather the boss, so I'll do my best to explain the situation.
We currently have a front porch queue that recieves messages from mulitple applications, and these messages are sequenced per application.
For example, we may get messages from FRED that are sequenced 1,2,3,4,5,6,7,8 as well as messages from BARNEY that are sequenced 1,2,3,4,5. The messages are read from the front porch by our server running in CICS, and put onto sub-queues. When the messages are put on the front porch queue, we have a DB2 table that is upated with the sequence numbers.
We are trying to solve a problem where things can arrive out of squence. I.E. messages from FRED may come in 1,2,4,3,5,6,7,8. Our current system in production dies on message four, giving us an out of squence error and shutting down the queue. We then intervene manually, reorder the message, etc.
We, on the advice of our support group that maintains MQ, are trying to implelment a solution using MQBROWSE and all that it entails.
We have it working if a message is out of order, but not if a message is missing entirely.
So, for example, say the queue has this: FRED1,2,4,BARNEY1,2,3,4FRED5,6,7,8.
So, we are missing the message from FRED that's number 3.
As it stands right now, we BROWSE and correctly process the first two messages from FRED, but since number 4 is out of sequence, we go looking for number 3. Well, since it's not on the queue at all, what will happen is that from message 4 onwards for all of FREDs messages, we put them on the error queue because the FRED application is now "out of sequence" because we "expect" the next number to arrive is #3, but it never comes, so every FRED message we check after that is not equal to 3, so it becomes an error.
However, messages from BARNEY will still be processed, as it's not missing any packets and is still able to be sequenced properly.
Has anyone ever had to deal with this type of situation? I hope I have explained it properly, but probably not. 
Another question for anyone who's used MQ BROWSE. What is the max search depth you should do? Or have you just let it search to the end of the queue? I'm just wondering if you didn't set a max search depth, would an MQBROWSE call reach the end of a queue even if messages were being put on it? Or could you run into a scenario where your search runs forever because the messages are added on faster then the BROWSE looks?
Thanks for any help or links somone can provide! |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Wed Jul 14, 2010 12:21 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
What you are referring to are "message groups," where one or more physical messages are all part of a message group.
In addition to MessageId and CorrelId fields in the message descriptor (MQMD), there is a GroupId field that uniquely identifies the message group.
At MQPUT time, the app that creates the message group needs to specify that this message is part (or not) of the message group. The consuming application needs to specify to the qmgr that it wants only messages part of the group.
Message groups are a programming construct. Take a look at the WMQ Application Programming Reference and WMQ Application Programming Guide.
[edit] And you have explained it perfectly. It is common that messages belonging to different applications (or instances of the same application) can coexist in the same queue.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jul 14, 2010 12:23 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You really shouldn't be leaving the messages on the queue in the first place.
There are better design options to solve this problem of sequencing messages - for example stick them into a database table.
But there's very little you can do to solve the issue of a missing message. If the whole sequence can't be processed if one of the items is missing, then the best you can do is wait a while until you decide that the message either has come or will never come, and then take action as appropriate.
But you should also consider what the proper place for this kind of business logic is. Does it belong on the front porch? or in the back end application?
You should also strongly consider avoiding using sequence dependent messages in the first place. Messages should be as stateless as possible - each message should represent an atomic and independent piece of data that can be processed. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jul 14, 2010 12:34 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
The issue of missing messages is solved with message groups.
That said, the entire issue of how to ensure that for each request sent, a reply is returned, has been discussed here at length.
WMQ is a transport (a pipe). WMQ will not duplicate or lose messages (with a variety of disclaimers). However, WMQ cannot enforce good behavior of applications.
It is possible that the requesting app sends request messages 1,2,3,4,5; and that the replying app chooses (through defective code) to only send replies 1,3,5. It is possible that the requesting app (badly coded) could send duplicate request messages, OR omit request messages, OR have a request message end up on a dead-letter-queue. It is possible that the replying app can send duplicate replies, OR omit replies, OR replies can end up in a dead-letter-queue.
WMQ offers to developers COA (Confirm on Arrival) and COD (Confirm on Delivery) are ReportOptions messages. These are of questionable (dubious) value (see the discussions here).
Well-behaved requesting app needs to keep track of request messages it has sent, and the replies it received. Well-behaved replying apps need to keep track of request messages it received, and the replies it sent back. This is an ideal use for a DB on both ends of the request-reply scenario.
[edit] Both requesting and replying apps need to be able to tolerate duplicate/missing messages.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jul 14, 2010 12:44 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bruce2359 wrote: |
| The issue of missing messages is solved with message groups. |
No.
The issue of missing messages is at best shuffled around with message groups. If the missing message is *never* sent, there's nothing you can do to detect this with groups, as long as you are asking to wait for the group to be complete.
Or if the sending application is going to be sending an ever increasing sequence number, where message 100001 came right after message 10000, and they both came many many days after message 1, then you've got an application that is sitting and waiting for the message group to complete for many many days.
Careful design of these sequences can allow them to be broken into groups, and then the receiving application can at least be sure that if the complete group is actually sent, then the receiver will not start processing any part of the group until the last member is sent.
There's a LOT of computational danger here, including many many cases where you run smack into the Halting Problem. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jul 14, 2010 1:01 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Okie, dokie.
If a "group" is a set of related messages, AND they span hours, days, or weeks, then the application design sucks - for the traditional request-reply scenario.
My posts usually presume that the application isn't coded to duplicate or lose messages; but I've seen too many of them coded exactly this way.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Wed Jul 14, 2010 6:04 pm Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
Grouping was discussed originally, but thrown out the window as it would impact the other applications who are putting to the queue. This is a government organziation, so getting other folks to do work can be onerous. 
It appears the reason that one particular application is arriving out of sequence is that their messages originate on an MQ manager on a different machine, then get sent to ours. Most other applications that put to the front porch are on the same machine as us, so the messages tend to arrive in sequence.
So, the missing packet scenario I described is usually not happening, just the out of sequencing, which we potentially have fixed. But, the missing packet wrinkle is something we want to deal with too.
As for putting the messages on a table right away, that isn't happening with these applications, and i'm not sure it will, as redesigns are hard to pull off. And this sucker was designed ages ago by other developers!
Same with sequencing. It's annoying, though we are told it's "necessary".
That being said, one solution to our problem may be gotten by doing this:
FRED sends messages 1,2,4 and BARNEY sends 1,2,3,4. As the messages are put to the front porch, our DB2 table that tracks sequencing gets 7 rows added, one for each packet.
However, during the processing, 1 is fine, 2 is fine, but then while expecting FRED3, it gets Fred 4. One suggestion was to BROWSE for message FRED3, and if it didn't find it, increment the number in the Sequence table, thus having MQ get back on track, and process FRED4 now that it's back in synch.
This means the sub queue for FRED has messages 1,2, and 4 on it. I found out late in the afternoon, the application that GETS from the sub queue actually has their own process to detect missing packets, so they can handle it from there.
The main goal is to ensure a smooth flow on the front porch, as several applications PUT to it, and if it gets shut off, stuff gets backed up.
I'm not sure if this solution makes sense.... it seems it would work, especially now that I know the sub queue processing has error handling for missing packets..... (I'm still learning all this MQ stuff as the boss!) |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jul 14, 2010 6:42 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Grouping was discussed originally, but thrown out the window as it would impact the other applications who are putting to the queue. |
What impact? What specifically?
With z, you have lots of central storage (and, therefore, virtual storage), cpu horsepower, and I/O bandwidth to easily manage huge messages with lots of applications concurrently browsing/getting/putting the same queue.
Lots of performance tuning is possible, too. Your sysprogs can easily expand queue manager bufferspace to accommodate more messages in-memory. Are you Parallel Sysplex?
| Quote: |
| So, the missing packet scenario... |
I'm presuming that you mean missing messages, rather than missing packets, since packets is one of those TCP/IP terms.
| Quote: |
| The main goal is to ensure a smooth flow on the front porch, as several applications PUT to it, and if it gets shut off ... |
What gets shut off?
| Quote: |
| ...stuff gets backed up. |
I'm guessing this means that messages are arriving faster than they can be consumed, yes? And that your consuming app design does NOT allow for multiple consuming app instances?
| Quote: |
| (I'm still learning all this MQ stuff as the boss!) |
You are not the first, nor will you be the last.
Please tell me a little about yourself. Are you a mainframe guy? Are your MQ folks from the midrange platforms?
May I strongly suggest that you attend IBM's one-day WM100 A Technical Introduction to WebSphere MQ. It is offered in classrooms and concurrently video-conference style (from your desktop at work or home).
http://www-304.ibm.com/jct03001c/services/learning/ites.wss/us/en?pageType=course_search&sortBy=5&searchType=1&sortDirection=9&includeNotScheduled=15&rowStart=0&rowsToReturn=20&maxSearchResults=200&language=en&country=us&searchString=wm100
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jul 15, 2010 4:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I wasn't talking about changing anything other than the front porch application.
It can take every message that comes in and stick it in a table, and then pull out messages in order.
The real question with an entirely missing message is who is supposed to make a decision that the message is missing, and who is supposed to be responsible for resolving it? The people who run the application that failed to send the message? The people that run the front porch application? Or the people that run the back end application?
This isn't really a technical question - it's a business decision. |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Thu Jul 15, 2010 4:42 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
| bruce2359 wrote: |
| Quote: |
| Grouping was discussed originally, but thrown out the window as it would impact the other applications who are putting to the queue. |
What impact? What specifically?
With z, you have lots of central storage (and, therefore, virtual storage), cpu horsepower, and I/O bandwidth to easily manage huge messages with lots of applications concurrently browsing/getting/putting the same queue. |
I believe the impact would be to applications putting to the front porch, as they would have to change their putter programs to use grouping, no?
| Quote: |
| The main goal is to ensure a smooth flow on the front porch, as several applications PUT to it, and if it gets shut off ... |
| Quote: |
| ...stuff gets backed up. |
The application reading from the front porch shuts off the queue when they detect an out of sequence situation. We then have to manually move messages around ourselves, from our error queue to resequence the messages, and turn the queue back on for further processing. The data in the messages are used to create time-sensitive letters, so the problems cannot sit overnight and wait until the next day, else the communications become stale dated.
| Quote: |
| Please tell me a little about yourself. Are you a mainframe guy? Are your MQ folks from the midrange platforms? |
We're mainframe. CICS, COBOL, DB/2.... z/0s...... all the fun stuff.
This is a good idea. I never thought about taking the intro course myself... it would probably help me out a lot. Thanks for the suggestion. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 15, 2010 4:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| AaronMenchions wrote: |
| I believe the impact would be to applications putting to the front porch, as they would have to change their putter programs to use grouping, no? |
Only if those application messages need to get mixed up in this sequence problem. If they don't use sequence and are independent of the sequenced messages then no.
| AaronMenchions wrote: |
| We're mainframe. CICS, COBOL, DB/2.... z/0s...... all the fun stuff. |
Yum.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Thu Jul 15, 2010 7:59 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
Everyone uses sequencing that puts to this front porch queue. Annoying, yes! lol.
Here's a question on a possible design for our solution. We are thinking to leave the strategy for a "missing message" as it is currently. That is, if system FRED ends up with a missing message, then all of the following FRED messages will get dumped onto our error queue for FRED.
This is how it is today, and we would leave the same thing with a BROWSE solution.
So, if the front porch queue has a mix of messages from FRED and BARNEY, and FRED is missing one, our server will go BROWSING for FRED, and if it comes across the BARNEY messages and they are ok, it will process them at the same time as browsing for FRED.
Once we reach the end of the queue and realize FRED is missing a message, we set an error flag and then all subsuqent messages from FRED get put into the error queue, while processing for BARNEY continues.
My only question is this. Is a BROWSE faster on cpu then a PUT? i.e. if there are 100 messages on the queue, and we start to BROWSE them while another application is still PUTting to the queue, will the BROWSE ever reach the "end" of the queue? I.E finsih before the PUTs might be completed? |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 15, 2010 8:25 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| AaronMenchions wrote: |
| My only question is this. Is a BROWSE faster on cpu then a PUT? i.e. if there are 100 messages on the queue, and we start to BROWSE them while another application is still PUTting to the queue, will the BROWSE ever reach the "end" of the queue? I.E finsih before the PUTs might be completed? |
Typically it is faster. A message that's been put might not even be available until the UoW in which it resides has been commited if syncopoint is in use. Likewise if the message is persistent it needs to be logged. Also in this example if there are a number of messages held on the queue they're likely to be pushed out to disc.
Given all that, browse is a hideously ineffecient and cpu expensive operation so on a queue of any depth I'd hate to say.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jul 15, 2010 11:29 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| I believe the impact would be to applications putting to the front porch, as they would have to change their putter programs to use grouping, no? |
Are you seeking a "no changes, but fix the problem" solution?
If messages can contain zero, one or more, components, AND the producing application and consuming application don't or won't know which and how many messages will be required to complete the transaction (or whatever you want to call it), then I don't see any method of ensuring the correct count and sequence.
Short of rewriting the applications, you might want to rethink the design. Is this a new application? Or have changes been made that now have FRED and BARNEY messages, where before, it was just FRED messages?
Is it possible that FRED messages can go to a FRED queue, while BARNEY messages go to a BARNEY queue. This will, at least, separate FRED and BARNEY. One instance of the consuming app can open the FRED queue, while a second instance of the exact same consuming application could open the BARNEY queue.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| AaronMenchions |
| Posted: Thu Jul 15, 2010 11:38 am Post subject: |
|
|
Newbie
Joined: 14 Jul 2010
Posts: 9
|
See, my team is a third party to this whole process. My area provides the programs that GET from this front porch queue, and PUT to the sub queues. The applications that PUT to the front porch and the application that reads from the sub queues are not my area.
As for what has changed, no one can answer that actually. Before October of last year we had very very few messages arriving out of sequence on the front porch queue. Since then, we have at least one or two instances a week of it occuring.
We have been told by the group that supports MQ itself on the mainframe that no network changes have been made to "all of a sudden" cause this problem. We tried to find out why it was occuring, to no avail.
Our senior support even contacted IBM, and they just said "we do not guarentee message order when you cross Q manager".
So, we had to figure out how to do this. Since this is a federal government shop, the attitude from both the PUTing app and the GETing app is "we are operating just fine, it's the middle man that's messing up".
Since my team is responsible for the "middle man", we are the ones who have to come up with the solution, hence changing from a straight GETing server to a BROWSE/GET one.
Anyway, it seems like our plan should cover most scenarios, and the likelihood of a message going "missing" is actually remote. It's happened once in the last x amount of years, and that was due to a queue config problem.
So, hopefully testing wont' reveal any other hidden suprises! |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2, 3 Next |
Page 1 of 3 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
