| Author |
Message
|
| sdrazi |
 Posted: Wed Oct 28, 2009 2:46 pm Post subject: Some messages not getting processed from the queue |
 |
|
Apprentice
Joined: 17 Apr 2007
Posts: 34
|
Hello..
We have a datastage process that puts messages to a Queue and a .NET application processes the message from that queue.
We see that some messages in the queue are not getting picked up by the .NET application (about 0.2%).
The .NET application is a continuously running application looking for messages in the queue and uses the MQGMO_WAIT Or MQGMO_BROWSE_NEXT options to browse the message.
After processing the message it uses the MQGMO_WAIT Or MQGMO_MSG_UNDER_CURSOR to do a destructive read on the message.
We were processing all the messages when the datastage process had the input as table when writing the message.
When the process changed from reading a table to reading a bunch of flat files for the same data we started noticing these messages being stuck in the queue.
If we restart the .NET application all the messages that are on the queue get processed.
Can I please get some help figuring out what is going on and reason for this change behaviour when the only the processing upstream changed from input as a table to a bunch of files. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Oct 28, 2009 2:58 pm Post subject: Re: Some messages not getting processed from the queue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sdrazi wrote: |
The .NET application is a continuously running application looking for messages in the queue and uses the MQGMO_WAIT Or MQGMO_BROWSE_NEXT options to browse the message.
After processing the message it uses the MQGMO_WAIT Or MQGMO_MSG_UNDER_CURSOR to do a destructive read on the message. |
Why do this? Why not just read the message?
| sdrazi wrote: |
| Can I please get some help figuring out what is going on and reason for this change behaviour when the only the processing upstream changed from input as a table to a bunch of files. |
You don't say what in your code resets the cursor to the top of the queue so I'd imagine that under some combination of circumstances the cursor is moving past them. This is another good reason (aside from the fact it's hideously inefficient) for avoiding cursors.
Another possibility is there's a bug in your code where the message id's not being reset and this is causing the cursor to skip.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Oct 28, 2009 4:14 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
Messages in queues are delivered out of the queue FIFO or by PRIORITY (and FIFO within priority). BROWSE_NEXT moves forward in the queue. Messages that arrive after the one you've browsed will be behind the cursor, so browse next will not find them.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 28, 2009 6:05 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
Do not browse and do a destructive get ...
Use UOW management and Syncpoint.
Do a destructive GET and commit after having processed the message.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sdrazi |
| Posted: Thu Oct 29, 2009 7:21 am Post subject: |
|
|
Apprentice
Joined: 17 Apr 2007
Posts: 34
|
thank you for your help...
This is the code for browse.....
RetrievedMsg = New MQMessage
gmo = New MQGetMessageOptions
gmo.Options = MQC.MQGMO_WAIT Or MQC.MQGMO_BROWSE_FIRST
gmo.MatchOptions = MQC.MQMO_NONE
gmo.WaitInterval = 10000
gmo.Options = MQC.MQGMO_WAIT Or MQC.MQGMO_BROWSE_NEXT
txtMqError = ""
Try
inputqueue.Get(RetrievedMsg, gmo)
Catch ex As MQException
txtMqError = ex.Message
End Try
After processing the retreived message...we do a destructive read like this...
gmo.Options = MQC.MQGMO_WAIT Or QC.MQGMO_MSG_UNDER_CURSOR
inputqueue.Get(RetrievedMsg, gmo)
| Quote: |
You don't say what in your code resets the cursor to the top of the queue so I'd imagine that under some combination of circumstances the cursor is moving past them. This is another good reason (aside from the fact it's hideously inefficient) for avoiding cursors. |
I do not understand the cursor concept much so what option do I use to reset the cursor...this is an application continuously looking for messages....
| Quote: |
Another possibility is there's a bug in your code where the message id's not being reset and this is causing the cursor to skip. |
Is this with the datastage app that is writing to the queue?
How can we process the message and then destroy it without doing a browse...(or without using cursors?)
so can we use UOW and syncpoint even though there is only one message in the UOW?
Does this have performance issues?
| Quote: |
| Messages that arrive after the one you've browsed will be behind the cursor, so browse next will not find them. |
apologies but I do not understand this completely..can I get more explanation on this?[/quote] |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 29, 2009 8:13 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
So, what is the business requirement to browse (look at) a message before you get (destructively read) a message? Is there something about the message content that the application looks at to determine if it's worth getting?
The same could be accomplished with get-in-syncpoint, look at the message content, if acceptable content then commit; if not acceptable content, then backout.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sdrazi |
| Posted: Thu Oct 29, 2009 8:37 am Post subject: |
|
|
Apprentice
Joined: 17 Apr 2007
Posts: 34
|
If the application terminates after getting the message we dont want to loose the message...
so is sync point less expensive that cursors and guarantees the retreival of message even in conditions (still unknown) that have some of the messages sitting on the queue and not getting consumed while most of the messages are getting consumed? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 29, 2009 9:12 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| If the application terminates after getting the message we don't want to loose the message... |
Refer to the WMQ App Programming Ref. and WMQ App Programming Guide. If the app terminates and the UofW does not complete, it will be backed out.
| Quote: |
| so is sync point less expensive that cursors and guarantees the retreival of message even in conditions (still unknown) that have some of the messages sitting on the queue and not getting consumed while most of the messages are getting consumed? |
Syncpoints (UofWs) ensure consistency within a transaction. Cost? Less than multiple MQI calls - gets with browse, followed by destructive gets.
As to the browse cursor: At first browse, the browse cursor is pointing to the top of the queue. Browse next moves the cursor to the 2nd message in the queue. When the next message is put to the queue, it goes to the top of the queue, but the browse cursor is already past the newer message(s). This likely explains why some messages are not being consumed. Again, refer to the WMQ App manuals for details on how to manage the cursor.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sdrazi |
| Posted: Thu Oct 29, 2009 10:43 am Post subject: |
|
|
Apprentice
Joined: 17 Apr 2007
Posts: 34
|
thanks for the help ...
so if the application putting the message is very fast when compared to the application consuming it...with browse cursors there is a possibility that messages will not be recognised. So the datastage job is probably putting in messages much faster from the flat files than from the table.
How is the message id used in consuming the message? Is there a possibily the new process is putting in duplicate message ids on the queue and hence the .NET App does not recognise this? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 29, 2009 12:16 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
The relative speed of requesting and replying applications is NOT the real issue you face; nor is it necessarily the place to look for a solution.
Is it possible for you to have multiple instances of the consuming application running, each getting messages from the same queue?
The WMQ APR and APG suggest that the requesting application ask the qmgr to create a unique MsgID in the MQND. When the replying app creates the reply message, it will move the inbound MsgId to the CorrelId of reply message MQMD.
After the requesting app puts the request msg to the queue, it does a get with wait on the reply-to-queue, waiting for a match of MsgId and CorrelId.
This is the traditional request-reply scenario. This allows for multiple requesting apps and multiple replying apps.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sdrazi |
| Posted: Thu Oct 29, 2009 1:34 pm Post subject: |
|
|
Apprentice
Joined: 17 Apr 2007
Posts: 34
|
We have just one instance of the application reading from the queue.
This is not a request response scenario...but a simple SEND and Forget scenario.
| Quote: |
The relative speed of requesting and replying applications is NOT the real issue you face; nor is it necessarily the place to look for a solution.
|
So is using UOW and Syncpoint to consume the message the solution.
Is the issue only on the consuming end?
We processed millions of messages from the table to queue via datastage without any getting stuck for about 5 days.
As soon as the datastage processing shifted to multiple flat files as the input we started seeing these messages being stuck which is making us think that something at the putting end changed which is causing this scenario. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 29, 2009 3:08 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| making us think that something at the putting end changed which is causing this scenario. |
It is quite likely that the increase in volume of messages, PLUS the odd coding of browse cursor and browse-then-get, that got you to this problem.
If browsing is required (instead of using syncpoint and MQCMIT/MQBACK), then set the browse cursor back to the top of the queue before the next get.
If browsing is not required, get rid of it.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sdrazi |
| Posted: Wed Nov 04, 2009 1:33 pm Post subject: |
|
|
Apprentice
Joined: 17 Apr 2007
Posts: 34
|
Thanks for the leads on the issue...
We found out from the ETL guys the difference in the datastage process from files as opposed to table was that datastage was committing all data in the file at one time (since they wanted to commit all data from file at once or rollback) when compared to the committing one row at a time from the table which was skipping the messages.
We plan on trying the SyncPoint MQCMIT logic as a permenant solution but as a hotfix we reset the cursor to the top of the queue whenever the program is done processing a batch of messages in the queue. |
|
| Back to top |
|
|
|
|
