| Author |
Message
|
| ruggy1976 |
 Posted: Wed Jun 15, 2011 2:00 am Post subject: SVRCONN in hung state. Messages don't reach the destination. |
 |
|
Novice
Joined: 15 Jun 2011
Posts: 11
|
Hi all....
Looking at APAR "IC40975: CURDEPTH OF QUEUE NEVER REACHES ZERO DUE A GHOST MESSAGE" the problem seems to be fixed through fix U497506 already included in MQ vers. 6.0.0.0.
However this problem persists in MQ vers. 6.0.2.9 which is installed on Aix 5.3.0.0.
1) Running the following command this is our result.
display queue(PREHISMSMQ02.XMITQ)
1 : display queue(PREHISMSMQ02.XMITQ)
AMQ8409: Display Queue details.
QUEUE(PREHISMSMQ02.XMITQ) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2011-05-30)
ALTTIME(10.53.04) BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLWLPRTY(0)
CLWLRANK(0) CLWLUSEQ(QMGR)
CRDATE(2009-05-11) CRTIME(16.25.09)
CURDEPTH(16) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR(XmitQ for PREHIS0102A Normal) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(1)
MAXDEPTH(5000000) MAXMSGL(4194304)
MONQ(QMGR) MSGDLVSQ(PRIORITY)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS(1) PROCESS( )
PUT(ENABLED) QDEPTHHI(80)
QDEPTHLO(20) QDPHIEV(DISABLED)
QDPLOEV(DISABLED) QDPMAXEV(ENABLED)
QSVCIEV(NONE) QSVCINT(999999999)
RETINTVL(999999999) SCOPE(QMGR)
SHARE STATQ(QMGR)
TRIGDATA( ) TRIGDPTH(1)
TRIGMPRI(0) TRIGTYPE(FIRST)
USAGE(XMITQ)
end
-- As you can see the CURDEPTH is 16.
2) Also, I've ran the following command
premq01a:/usr/mqm/bin:>/opt/mqm/samp/bin/amqsbcg PREHISMSMQ02.XMITQ QMCERT1
AMQSBCG0 - starts here
**********************
MQOPEN - 'PREHISMSMQ02.XMITQ'
No more messages
MQCLOSE
-- As you can see NO messages are in the queue.
3) And to conclude this command
dspmqtrn -e -i -m QMCERT1
There are no prepared transactions.
-- Any in-doubt transactions for QMCERT1 QManager
We've already experienced this issue and in that occasion we stopped relative QManager "resolving" definitely. I think this is not the best solution at all.
Can you please help me to fix this issue?
Thanks in advance for helping me on this
Fabio R.
Last edited by ruggy1976 on Thu Jun 16, 2011 6:05 am; edited 1 time in total |
|
| Back to top |
|
 |
| ruggy1976 |
| Posted: Wed Jun 15, 2011 2:01 am Post subject: |
|
|
Novice
Joined: 15 Jun 2011
Posts: 11
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 15, 2011 2:15 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Well, MQ v6 is still in support, so you can open an PMR.
I wonder though if you gave the correct set of options to dspmqtrn. It showed "no prepared transactions", which might mean "no XA transactions".
So that's the only thing I'd do before opening a PMR, is double-check that different options to dspmqtrm didn't give a different result.
I'd also look at DIS QSTATUS and DIS CONN to see if there were open handles on the queue at all. |
|
| Back to top |
|
|
| ruggy1976 |
| Posted: Wed Jun 15, 2011 2:39 am Post subject: |
|
|
Novice
Joined: 15 Jun 2011
Posts: 11
|
Hi there...
First of all THX a lot for quick reply. MUCH APPRECIATED 
As you correctly suggested I've performed other 2 tests.
1) premq01a:/home/mqscr:>cd /opt/scripts/mqm/mh04/
premq01a:/opt/scripts/mqm/mh04:>./xmqqstat.sh -e -h -t -m QMCERT1 -q PREHISMSMQ02.XMITQ
Xmqqstat v1.1 - Developed by Oliver Fisse (IBM)
Connected to queue manager 'QMCERT1'
PLATFORM(UNIX) LEVEL(600) CCSID(819)
MAXHANDS(512) MAXMSGL(4194304) MAXPRTY(9) MAXUMSGS(10000) MONQ(OFF)
Processing LOCAL queue 'PREHISMSMQ02.XMITQ'
DESC(XmitQ for PREHIS0102A Normal)
CRDATE(2009-05-11) CRTIME(16.25.09) ALTDATE(2011-05-30) ALTTIME(10.53.04)
CLUSTER() CLUSNL() DEFBIND(OPEN)
BOTHRESH(0) BOQNAME()
MONQ(QMGR) USAGE(XMIT) NOTRIGGER
Dumping 1 handle(s)...
PID TID AT CHL/APPL TAG/CONN USER ID B INP I O S
------------------------------------------------------------------------------
106652 2328 USER CHQMCERT1HIS2 mqm N EXC Y N N
Server 2006\system\q2qgw.exe
10.207.149.142
Time MxML MxQD G P OIC OUC MDC MEC UNC CQD PQF TQF TQE QOM OQTS OQTL
-------------------------------------------------------------------------------------------------------------------
12:27:04 4194304 5000000 E E 1 0 99 102 0 0 0.00% -0.00s -0.00s
Disconnected from queue manager 'QMCERT1'
Xmqqstat v1.1 ended.
-- Xmqqstat is a useful tool which helps to understand few info about a queue. Looking at the result I see
Dumping 1 handle(s)...
PID TID AT CHL/APPL TAG/CONN USER ID B INP I O S
------------------------------------------------------------------------------
106652 2328 USER CHQMCERT1HIS2 mqm N EXC Y N N
Server 2006\system\q2qgw.exe
10.207.149.142
2) I've also run the DIS QSTATUS
DIS QSTATUS(PREHISMSMQ02.XMITQ)
6 : DIS QSTATUS(PREHISMSMQ02.XMITQ)
AMQ8450: Display queue status details.
QUEUE(PREHISMSMQ02.XMITQ) TYPE(QUEUE)
CURDEPTH(0) IPPROCS(1)
LGETDATE( ) LGETTIME( )
LPUTDATE( ) LPUTTIME( )
MONQ(OFF) MSGAGE( )
OPPROCS(0) QTIME( , )
UNCOM(NO)
-- The CURDEPTH is 0
To be sincere I'm groping in the dark ))))))
Thanks for your support
HAVE A NICE DAY
Fabio R. |
|
| Back to top |
|
|
| ruggy1976 |
| Posted: Wed Jun 15, 2011 2:56 am Post subject: |
|
|
Novice
Joined: 15 Jun 2011
Posts: 11
|
Hi again....
Referring to your note:
I wonder though if you gave the correct set of options to dspmqtrn. It showed "no prepared transactions", which might mean "no XA transactions".
So that's the only thing I'd do before opening a PMR, is double-check that different options to dspmqtrm didn't give a different result.
I've noticed this link
http://www-01.ibm.com/support/docview.wss?uid=swg1IC64115&wv=1
It seems that the COMMAND DSPMQTRN ERRONEOUSLY RETURNS ERROR CODE 71 (UNEXPECTED ERROR) WHEN THERE ARE NO PREPARED TRANSACTIONS.
I think this is the case, isn't it?
Bye
Fabio R. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 15, 2011 3:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So the ipprocs says that someone has the queue open for input, and Oliver's little script shows that it is someone known as q2qgw.exe. So I'd check this program to see if it uses transactions at all.
I agree that the APAR IC64415 may apply here as well. But you didn't capture the actual RETURN CODE from dspmqtrm, just the text it produced. so dspmqtrm may be properly returning 102 and thus the APAR won't apply.
I would confirm as well that DIS QUEUE and DIS QSTATUS show the same CURDEPTH - or if they show different CURRDEPTH in the "same" time, then that's useful as well. |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Jun 15, 2011 3:25 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| mqjeff wrote: |
| So the ipprocs says that someone has the queue open for input, and Oliver's little script shows that it is someone known as q2qgw.exe. So I'd check this program to see if it uses transactions at all. |
Why would any application, other than an MCA, hold an XMITQ open for input? That said, q2qgw.exe is the MSMQ-MQSeries Bridge Service so probably MSDTC is in the mix somewhere.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 15, 2011 3:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| exerk wrote: |
| Why would any application, other than an MCA, hold an XMITQ open for input? |
Oh hey, lookit that. It's an XMITQ.
| exerk wrote: |
| That said, q2qgw.exe is the MSMQ-MQSeries Bridge Service so probably MSDTC is in the mix somewhere. |
excellent! |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 15, 2011 1:16 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I have seen the same type of ghost messages on the xmitq in following case:
- message was put for a remote qmgr => hence on xmitq within a UOW
- UOW never was committed
- UOW never got the prepare for commit
The result is that you do have a message that counts in the curdepth but is not in a state where it can either be force committed or rolled back.
The message is not in a gettable state.
The connection putting the message may have been cut ...
The only way I have seen those messages go away is when the client fully disconnected (java app) or when the qmgr was bounced...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jun 15, 2011 2:01 pm Post subject: Re: APAR IC40975 not fixed in MQ vers 6 |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| ruggy1976 wrote: |
| -- As you can see the CURDEPTH is 16. |
Check the channel status for the channel serving that XMITQ. Possibly the channel is indoubt or is in some hung state? |
|
| Back to top |
|
|
| ruggy1976 |
| Posted: Wed Jun 15, 2011 11:32 pm Post subject: |
|
|
Novice
Joined: 15 Jun 2011
Posts: 11
|
Hi there....
As you correctly noticed the channel serving that XMITQ, for some reasons, was in hung state.
I've stopped and restarted the relative channel and the messages reached the destination...
Thanks for support and quick responses
APPRECIATED
HAVE A NICE DAY
Fabio |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Jun 16, 2011 12:30 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| ruggy1976 wrote: |
| I've stopped and restarted the relative channel and the messages reached the destination... |
Out of curiosity, what was the channel you stopped/started?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| ruggy1976 |
| Posted: Thu Jun 16, 2011 1:07 am Post subject: |
|
|
Novice
Joined: 15 Jun 2011
Posts: 11
|
Hi... The channel type is SVRCONN.
Regards
Fabio R. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Jun 16, 2011 1:18 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Client -> SVRCONN -> QREMOTE -> XMITQ -> MSMQ pipe
Is the above correct?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| ruggy1976 |
| Posted: Thu Jun 16, 2011 1:25 am Post subject: |
|
|
Novice
Joined: 15 Jun 2011
Posts: 11
|
|
| Back to top |
|
|
|
|
