| Author |
Message
|
| tczielke |
 Posted: Thu Jan 08, 2015 2:15 pm Post subject: |
 |
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
If you take one of these queues (i.e. MY.Q1) that have these uncommitted messages, and do the following:
ex:
dis conn(*) all type(handle) where (OBJNAME EQ MY.Q1)
and if connections are returned (i.e. the connection 54A2D054202B0902 was returned), and you run the following command for these connections:
ex:
dis conn(54A2D054202B0902) all
do you see any connections that have a UOWSTATE value that is not NONE? |
|
| Back to top |
|
 |
| rconn2 |
| Posted: Thu Jan 08, 2015 2:51 pm Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
| I'll try and post back tomorrow am. Thanks for helping!! |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jan 09, 2015 7:17 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The problem might originate from WAS.
Make sure you have set up the right credentials for WAS to commit or rollback pending transactions at restart.
The transaction does not need to be in a prepared (commit) state.
If the prepare commit was never issued does it show in the dspmqtrn? Is the transaction considered to be in doubt? The message will still show in the queue depth....
Any of these "ghost" messages should either disappear on bounce of qmgr or show up in dspmqtrn...
I would also advise you to check with your WAS admin as this was a known problem depending on WAS & MQ versions.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rconn2 |
| Posted: Fri Jan 09, 2015 8:42 am Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
| tczielke wrote: |
| do you see any connections that have a UOWSTATE value that is not NONE? |
All the connections for the local queue (with 131 uncommitted) show NONE; there's a connection for the transmission queue (with 227 uncommitted) and it shows ACTIVE. |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Jan 09, 2015 8:50 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
| For the connection on the XMITQ with the UOWSTATE(ACTIVE), the other fields on the DIS CONN ALL will help you see how long the UOW has been active and what pid/tid owns that connection. This will help you at least (potentially) to tie it back to the appropriate application and application team to investigate this issue. |
|
| Back to top |
|
|
| rconn2 |
| Posted: Fri Jan 09, 2015 9:00 am Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
| fjb_saper wrote: |
| If the prepare commit was never issued does it show in the dspmqtrn? Is the transaction considered to be in doubt? |
dspmqtrn -m <qmgr> returns nothing.
The status of the client channel used by the application doesn't show anything in column "in doubt status" for any of the connections.
For the server channel used by the transmission queue, the status of the only connection shows "not in doubt" under column "in doubt status".
Interestingly, I can't browse the uncommitted messages on the local queue. When I try to browse (with a depth of 131 uncommitted messages), no messages show up in MQ Explorer to browse.
I haven't tried a bounce of the queue manager as this is a prod system and I'll need a time-window to do so. In any case, if the uncommitted messages get cleared, it'd only be temporary as they'd soon accumulate again.
I would expect that a busted transaction -- where the connection gets broken -- would get rolled back automatically. I'm the MQ admin, but also do some WAS admin (but I'm not a SME for WAS). Is there something I might examine in the WAS console? Or is this something that will be in app code?
Thanks guys for your help... I want to pursue this till it gets fixed. |
|
| Back to top |
|
|
| rconn2 |
| Posted: Fri Jan 09, 2015 9:16 am Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
| tczielke wrote: |
| For the connection on the XMITQ with the UOWSTATE(ACTIVE), the other fields on the DIS CONN ALL will help you see how long the UOW has been active and what pid/tid owns that connection. This will help you at least (potentially) to tie it back to the appropriate application and application team to investigate this issue. |
The conname shows the IP and Listening port of the destination queue manager for the Server channel that corresponds with the xmitq. The connection has just been active since today (the connection will stop and restart automatically as configured to do so). |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Jan 09, 2015 9:47 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
The pid/tid of that connection will probably the amqrmppa process. It is cryptic and may require some root access to run some of the lsof commands, but if this is a Unix/Linux platform, you can tie that back to the pid/tid of the application that is on the CLNTCONN end by doing the following:
1. From the DIS CONN ALL info, note the pid/tid (i.e. 1000.2) for that connection.
2. Run amqrdbgm (channel cache program) and do the c option. It will list out the pid.tids for channels that are active. Note the socket that your 1000.2 is using (i.e. Socket(5) )
3. Do a "lsof -p 1000" (or the UNIX platform equivalent i.e. pfiles) and find file descriptor 5. It will look something like the following:
amqrmppa 8894 mqm 5u IPv4 7969342 0t0 TCP SERVER1.hewitt.com:ibm-mqseries->SERVER1.hewitt.com:35615 (ESTABLISHED)
From this example, the application on the CLNTCONN end is using ephemeral port 35615.
4. Now do the appropiate lsof command on port 35615 and find what process is using that port. Here is an example:
mqm@SERVER1$ lsof -i TCP:35615
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
amqrmppa 8894 mqm 5u IPv4 7969342 0t0 TCP SERVER1.hewitt.com:ibm-mqseries->SERVER1.hewitt.com:35615 (ESTABLISHED)
amqsputc 28390 mqm 3u IPv4 7969498 0t0 TCP SERVER1.hewitt.com:35615->SERVER1.hewitt.com:ibm-mqseries (ESTABLISHED)
So pid 28390 is the application that is connected to our SVRCONN channel. |
|
| Back to top |
|
|
| rconn2 |
| Posted: Fri Jan 09, 2015 11:02 am Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
Sorry, I meant it's a Sender channel (not a SVRCONN channel). The conname is showing the destination queue manager's IP and port. This is regarding the uncommitted's on the XMITQ.
The UOW's on the connections for the application's dedicated SVRCONN channel, for the local queue, all show NONE. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 09, 2015 11:10 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What is the status of the receiver channel on the other side?
What happens to the uncommited messages if you stop the sender channel? |
|
| Back to top |
|
|
| rconn2 |
| Posted: Sat Jan 10, 2015 4:39 am Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
The application was brought down this am for a maintenance release. The app dedicated Client channel was inactive. There were still uncommitted messages on the local queue. I was unable to browse these messages.
When I stopped the Sender channel (I didn't stop the Rcvr -- read your email too late), the Xmitq still showed 215 uncommitted messages. When I allowed gets and tried to browse, there were no messages to browse. When I tried to Clear the queue, there was the message that there were uncommitted puts and they wouldn't clear.
I restarted the Queue manager and all the uncommitted messages finally disappeared. I expect they'll slowly build up again this next week. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Jan 10, 2015 6:13 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| rconn2 wrote: |
| The application was brought down this am for a maintenance release. The app dedicated Client channel was inactive. There were still uncommitted messages on the local queue. I was unable to browse these messages. |
I'm a bit confused. Please use more precise terms.
Transmission queues have nothing to do with Client channels. Transmission queues are used with SENDER and SERVER channels at one end of the channel. SENDER and SERVER channels connect to RECEIVER or REQUESTER channels at the other (receiving) end of the channel.
Local queue? Do you mean the transmission queue? Or some other local queue?
Why were you unable to browse the messages? Defective keyboard? Are you not authorized to access the local queue? Did you get some kind of error?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rconn2 |
| Posted: Sat Jan 10, 2015 8:30 pm Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
There were uncommitted messages on a regular local queue and also on a transmission queue.
Regarding the regular local queue, the Svrconn client channel used by the application was stopped (became inactive) when the application was brought down this am. This had no effect on the uncommitted messages, which remained. When I tried to browse the messages with MQ Explorer, no messages showed up to be browsed. Yet, the queue showed a depth. The queue Status showed the same number of messages as the depth.
Regarding the Transmission queue, the Sender channel was stopped (I didn't stop the Receiver channel... not sure of its state). There were 215 uncommitted messages and a queue depth of 215. I allowed gets on this queue, but when I tried to browse, again as with the regular local queue, no messages showed up. When i chose to Clear the queue, I received the error message that there were "uncommitted puts".
When I restarted the queue manager, all these uncommitted messages on both the transmission queue and regular local queue went away. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jan 11, 2015 2:22 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
This is the case for messages put under transactional control but never hit the prepare commit point. If the connection gets interrupted and the transaction manager has no way of reinstating pending transactions, these messages become orphaned and are know as ghost messages.... (no way to force a commit or rollback). The sometimes go away if you can find the lost connection on the queue manager and force it closed. They do go away when you bounce the qmgr.
Be careful about the way the application works. If it is relying on the WAS TM to complete the transaction, it might believe the transaction was successful when clearly it was not. A User managed transaction might be more precise here...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rconn2 |
| Posted: Sun Jan 11, 2015 1:08 pm Post subject: |
|
|
Voyager
Joined: 09 Aug 2007
Posts: 79
Location: MD, USA
|
Thanks. That's exactly how I described them -- orphans and ghost messages  . And, they did go away on a qmgr restart.
Is there any way other than a bounce to get rid of them? How about a time to live setting... say set for 100 hours or somesuch? It'd be good to have a way to rollback messages not committed within a designated time. |
|
| Back to top |
|
|
|
|
