| Author |
Message
|
| bartek_n |
 Posted: Wed Sep 28, 2005 9:56 am Post subject: How to get the uncommited messages |
 |
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
Hio all!
Currently we've got a problem with tracking down the reasons and the point of failure in 'information flow' which bases on MQSeries queue. We have big java applications which communicate through the queue. And somehow, one message has dissapeared... I can only guess what happened and which system should be blamed...
Anyway, I can see that (basing on the current depth of the queue) there are few old uncommited messages there. I suppose there might be the one.
So, the question is - how can I get this messages from the queue?
What tool should be used - as far as I know there is no way to read them using Java API. But maybe some administration tools?
And sesond question - is it possible to configure queue manager that way, that it would log every message incoming/outgoing from the queue? Just msgId and time would do the trick...
Thanks in advance!
Bartek |
|
| Back to top |
|
 |
| wschutz |
| Posted: Wed Sep 28, 2005 10:31 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
There is no way to read uncommitted messages (unless you're reading them from the same process that created them).
Did these messages come across a channel or local puts? Are you saying that the "putting" big java application still has the unit of work open? Or has that application gone away? If it crashed or disconnected from mq the queue manager will handle the uncommitted messages properly.
The only way to log all messages is to use an API crossing exit to somehow log the messages yourself.
_________________
-wayne |
|
| Back to top |
|
|
| bartek_n |
| Posted: Wed Sep 28, 2005 11:31 am Post subject: |
|
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
| Quote: |
| Did these messages come across a channel or local puts? Are you saying that the "putting" big java application still has the unit of work open? Or has that application gone away? If it crashed or disconnected from mq the queue manager will handle the uncommitted messages properly. |
The application has been restarted few times and these uncommited msgs are still there. I am not sure what is the exact reason such messages are left in the queue. We are using XA transaction spanning MQ and Oracle access, so there are many possiblities something goes wrong. Usually, such msgs are removed when transaction finishes, but in some cases (I guess when no commit nor rollback has been invoked and process dies for some reason) they stay.
So, I don't know why queue manager didn't handle this...
| Quote: |
| The only way to log all messages is to use an API crossing exit to somehow log the messages yourself. |
Yeah  My logs say that msg has been sent, guys from the other side say that they never received it, because they don't see any statement in their logs... |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Sep 28, 2005 3:44 pm Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| Quote: |
| We are using XA transaction spanning MQ and Oracle access |
Is MQ the transaction manager (or is something else, like websphere?) There's XA protocols about resource managers determining the outcome of indoubt transactions when they can reconnect to the transaction manager.
_________________
-wayne |
|
| Back to top |
|
|
| bartek_n |
| Posted: Wed Sep 28, 2005 3:54 pm Post subject: |
|
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
| Quote: |
| Is MQ the transaction manager (or is something else, like websphere?) There's XA protocols about resource managers determining the outcome of indoubt transactions when they can reconnect to the transaction manager. |
Our app is running on ATG Dynamo application server and we use its transaction manager. What do you mean - that TM should be configured specially to handle such situations? Because so far, it doesn't...
Maybe also important is that it happends from time to time that we find the lock in Oracle db held by in-doubt transaction, which seems not to be resolved automatically by Oracle's RECO process. And also, there are no serious problems with network, unclean servers' restarts etc. which would result in some unnormal behaviour. |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Sep 28, 2005 4:31 pm Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
So ATG is the transaction manager and MQ and Oracle are resource managers. What I'm saying is, that if there is an indoubt MQ transaction, MQ askes ATG what the outcome of the transaction was. It sounds like there might be some protocol error, leaving indoubt messages.
I don't know anything about configuring ATG. afaik, ATG isn't listed as a supported TM.
But in any case, do you see any FDC files in the errors directory?
_________________
-wayne |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Sep 28, 2005 8:10 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
We have the same problem.
You can't trust the ATG ecommerce transaction manager to be fully XA Compliant specifically if you are not using the jta transaction manager and are using the "FakeXADriver" which I have been told does the commit to the DB on the "prepare commit".
We are getting 2 types of uncommitted messages
a) "ghost" messages. They are uncommitted and will disappear as soon as the qmgr is recycled.
b) uncommitted messages. They have received a "prepare commit" but never got the commit or rollback. You can force commit them using dspmqtrn and rsvmqtrn.
You should have a number of FDC files with a specific Probe id (from memory 4xxxx010 ... XAE_PROTO_ERR)
At the same time you may have long running transactions getting rolled back (see qmgr logs) even though you do not have any long running transaction. This is a side effect of the messages on MQ still waiting on a definitive commit or rollback even though the session is done.(~ above 1% of the messages)
At least I know we are no longer suffering alone.
In our case we have cases where we get a close.XA being sent before an end.XA is being sent. (XA Protocol violation). We are still checking our different options and no the 10g driver against the 9i Db did not do it for us....
We have an open ticket with ATG about this problem...
Enjoy the pain   |
|
| Back to top |
|
|
| bartek_n |
| Posted: Thu Sep 29, 2005 1:30 am Post subject: |
|
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
| Quote: |
We have the same problem.
You can't trust the ATG ecommerce transaction manager to be fully XA Compliant specifically if you are not using the jta transaction manager and are using the "FakeXADriver" which I have been told does the commit to the DB on the "prepare commit". |
I managed not to use the "FakeXADriver" - we got separate datasource using oracle.jdbc.xa.client.OracleXADataSource. However, currently (this app is 'inherited' from the previous developer/maintainer) we got 2 repositories taking part in XA transactions - one (more important) is using real XA drivers and second is still using "FakeXADriver". I got feeling this can be source of problems, but switching whole huge app (already running in live env.) right now is a bit scaring - but it will have to be done, anyway.
| Quote: |
We are getting 2 types of uncommitted messages
a) "ghost" messages. They are uncommitted and will disappear as soon as the qmgr is recycled.
b) uncommitted messages. They have received a "prepare commit" but never got the commit or rollback. You can force commit them using dspmqtrn and rsvmqtrn. |
May you please give more light how to use these tools? is it possible then to somehow peek into these msgs header/body and see what are they from? As I understand, if manuall commit is forced, they will show up as a normal msgs in the queue?
| Quote: |
You should have a number of FDC files with a specific Probe id (from memory 4xxxx010 ... XAE_PROTO_ERR)
|
There are plenty of files in 'errors' folder, but none of them has info about such error. However, I'm not the admin of these MQSeries instance and I have no clue how it is configured...
| Quote: |
At the same time you may have long running transactions getting rolled back (see qmgr logs) even though you do not have any long running transaction. This is a side effect of the messages on MQ still waiting on a definitive commit or rollback even though the session is done.(~ above 1% of the messages) |
I will try to verify logs (as soon as I find them...) - but it seems that this is the problem here...
| Quote: |
At least I know we are no longer suffering alone. |
I'm glad I could help! 
| Quote: |
In our case we have cases where we get a close.XA being sent before an end.XA is being sent. (XA Protocol violation). We are still checking our different options and no the 10g driver against the 9i Db did not do it for us.... |
| Quote: |
We have an open ticket with ATG about this problem...
Enjoy the pain |
Yeah that's a lot of joy, indeed
You have reported this problem to ATG - so I guess that you've some sample code when this problem has been isolated, right? Would it be possible to give a hint how to construct such minimal snippet?
Bit thanks for reply!
Bartek |
|
| Back to top |
|
|
| wschutz |
| Posted: Thu Sep 29, 2005 2:22 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| Quote: |
| You can force commit them using dspmqtrn and rsvmqtrn. |
dspmqtrn is used to display in-doubt transactions. It gives you a list of transactions IDs. You can use rsvmqtrn to commit or to backout those transactions.
If the indoubt messages are a result of MQPUTs, then the backout makes those msessages go-away. If indobuts are a result og MQGETs, then the backout makes them messages re-appear. 
_________________
-wayne |
|
| Back to top |
|
|
| bartek_n |
| Posted: Thu Sep 29, 2005 2:48 am Post subject: |
|
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
| Quote: |
dspmqtrn is used to display in-doubt transactions. It gives you a list of transactions IDs. You can use rsvmqtrn to commit or to backout those transactions.
If the indoubt messages are a result of MQPUTs, then the backout makes those msessages go-away. If indobuts are a result og MQGETs, then the backout makes them messages re-appear |
Thanks! I am just digging into docs about these tools.
Bartek |
|
| Back to top |
|
|
| bartek_n |
| Posted: Thu Sep 29, 2005 3:05 am Post subject: |
|
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
I've tried to use the dspmqtrn command and I got output like that:
| Code: |
AMQ7056: Transaction number 0,925213.
XID: formatID 4871251, gtrid_length 52, bqual_length 46
gtrid [FF5F5700FFFF57FF6461622D746573742D6170703230332E696E742E6469726162612E64652C5431313237343638323139333430]
bqual [6461622D746573742D6170703230332E696E742E6469726162612E64652C54313132373436383231393334302C02]
AMQ7056: Transaction number 0,0.
XID: formatID 4871251, gtrid_length 52, bqual_length 46
gtrid [7FFF1700FFFF4BFF6461622D746573742D6170703230332E696E742E6469726162612E64652C5431313234393834333935373934]
bqual [6461622D746573742D6170703230332E696E742E6469726162612E64652C54313132343938343339353739342C02]
AMQ7056: Transaction number 0,1.
XID: formatID 4871251, gtrid_length 52, bqual_length 46
gtrid [FFFF1700FFFF4BFF6461622D746573742D6170703230332E696E742E6469726162612E64652C5431313234393834333935373934]
bqual [6461622D746573742D6170703230332E696E742E6469726162612E64652C54313132343938343339353739342C02]
|
and according to documentation it should be like:
| Code: |
AMQ7107: Resource manager 0 is MQSeries.
AMQ7107: Resource manager 1 is Oracle MQBankDB
AMQ7107: Resource manager 2 is Oracle MQFeeDB
AMQ7056: Transaction number 0,1.
XID: formatID 5067085, gtrid_length 12, bqual_length 4
gtrid [3291A5060000201374657374]
bqual [00000001]
AMQ7105: Resource manager 0 has committed.
AMQ7104: Resource manager 1 has prepared.
AMQ7104: Resource manager 2 has prepared. |
Why there is no info displayed about other resource managers? Is it different situtation because MQSeries is not a transaction manager but resource manager?
And more - is it possible to match such entry with the lock in Oracle (held by in-doubt transaction)? |
|
| Back to top |
|
|
| wschutz |
| Posted: Thu Sep 29, 2005 6:06 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| Quote: |
Is it different situtation because MQSeries is not a transaction manager but resource manager?
|
Yes, if MQ isn't the TM, it doesn't know about the RM's. Is this causing you a problem? Can't you still use rsvmqtrn to correct the indoubt uow's?
_________________
-wayne |
|
| Back to top |
|
|
| bartek_n |
| Posted: Thu Sep 29, 2005 6:26 am Post subject: |
|
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
| Quote: |
| Yes, if MQ isn't the TM, it doesn't know about the RM's. Is this causing you a problem? |
I hope not, just wandering why mine otput was different.
| Quote: |
| Can't you still use rsvmqtrn to correct the indoubt uow's? |
I guess I could, but I have to wait before trying this until the system scanning this queue will be down and when some admin will invoke this command (I don't have rights  to do this).
As I understood, it is possible that after running it some of the 'ghosts' could show up and I don't want the listening system to pick it up, before I get to this msgs.
Big thanks for help!
Bartek |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Sep 29, 2005 12:56 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Thats the right approach.
We create a .sh file using
| Code: |
| dspmtrn | grep AMQ | sed |sed > forcecommit.sh |
The output is right.
rsvmqtrm will need to have something like:
| Code: |
| rsvmqtrn -m QMGR -c 0,925213 |
Have you tried yet using the JTA transaction manager supplied by ATG ?
| Quote: |
You have reported this problem to ATG - so I guess that you've some sample code when this problem has been isolated, right? Would it be possible to give a hint how to construct such minimal snippet?
|
To reproduce the problem we need to stress test in our qual environment.
The problem appears in between 1 to 2 % of the messages we shot through. There does not seem to be a specific pattern to it and when you try to isolate it transaction by transaction it seems to always work as designed ....
Enjoy |
|
| Back to top |
|
|
| bartek_n |
| Posted: Thu Oct 06, 2005 3:02 am Post subject: |
|
|
Novice
Joined: 28 Sep 2005
Posts: 10
|
|
| Back to top |
|
|
|
|
