ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportWebSphere can't connect to MQ after several days

Post new topicReply to topic
WebSphere can't connect to MQ after several days View previous topic :: View next topic
Author Message
cmckenna
PostPosted: Mon Mar 22, 2021 5:48 am Post subject: WebSphere can't connect to MQ after several days Reply with quote

Newbie

Joined: 22 Mar 2021
Posts: 6

I am currently supporting a very old application running on WebSphere version 7.0.0.19 and using MQ version 7.0.1.2.

After 4 to 5 days of the application running just fine, we suddenly start getting the following errors:

[3/19/21 19:14:03:158 EDT] 0000001c SibMessage W [:] CWSJY0003W: JMSCC3036: An exception has been delivered to the connections exception listener: '
Message : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ1107: A problem with this connection has occurred. This exception has been delivered due to an exception listener being registered on this connection. Use the linked exception to determine the cause of this error.
Class : class com.ibm.msg.client.jms.DetailedIllegalStateException


Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE').
Class : class com.ibm.mq.MQException

If we recycle WAS and MQ, the application is fine again for another 4 to 5 days before the cycle repeats.

I have checked the connection pools during the issue and there are plenty of free connections so I don't believe we have a leak.

Also, WAS and MQ are running on the same server and there is NO SSL involved yet the reason code seems to indicate SSL.

I also grabbed hunderds of MB of websphere trace logs from the latest incident and see nothing in those traces that is any more helpful than the above mentioned stack trace.
Back to top
View user's profile Send private message Send e-mail
exerk
PostPosted: Mon Mar 22, 2021 8:18 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6224

There are a few hits for exception code JMSWMQ1107, such as IC68653 which seems to be the most relevant to you.

Which method of connecting WAS to the queue manager are you using? As you mention SSL I'm assuming client rather than bindings mode, but would like it confirmed (or not).
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
cmckenna
PostPosted: Mon Mar 22, 2021 8:51 am Post subject: Client mode Reply with quote

Newbie

Joined: 22 Mar 2021
Posts: 6

Client mode is correct
Back to top
View user's profile Send private message Send e-mail
exerk
PostPosted: Mon Mar 22, 2021 9:04 am Post subject: Re: Client mode Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6224

cmckenna wrote:
Client mode is correct

As they are on the same server, why?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
cmckenna
PostPosted: Mon Mar 22, 2021 9:08 am Post subject: Reply with quote

Newbie

Joined: 22 Mar 2021
Posts: 6

Wish I had a good answer for that. This app is about 12 years old and I inherited it when I joined this new company a couple months ago. I have bee trying to get as much detail about the design and implementation but the developers and architects are long gone
Back to top
View user's profile Send private message Send e-mail
exerk
PostPosted: Mon Mar 22, 2021 9:15 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6224

cmckenna wrote:
Wish I had a good answer for that. This app is about 12 years old and I inherited it when I joined this new company a couple months ago. I have bee trying to get as much detail about the design and implementation but the developers and architects are long gone

I feel your pain...

1. How long before you inherited this did it start, i.e. does this particular system have a history of this, or is it recent?

2. What recent changes have been made to the system, e.g. OS, if any?

3. Has changing it to bindings mode been floated with management?

I have a suspicion that a decision was made long ago to use client mode 'in case' the resource usage of the server got to the point where MQ got 'moved' to a different server, but that's speculation on my part.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
cmckenna
PostPosted: Mon Mar 22, 2021 9:46 am Post subject: Reply with quote

Newbie

Joined: 22 Mar 2021
Posts: 6

again, its difficult to ascertain all the details around this.... here is what I have been told....

the issue according to some, started in early fall 2020.

They had migrated this app to AIX prior to my arrival although no one remembers the date this was done.

As far as I know, no discussion has been had regarding binding mode.

Sorry I can not be of more help. I have a ton of WAS experience going back to version 3 but I fall short with MQ and JMS messaging. Everything I supported in the past was apps talking to actual external Message Brokers with a dedicated team of MQ experts to handle the support on that side.

My new place is a small shop that does not have a lot of expertise....
Back to top
View user's profile Send private message Send e-mail
exerk
PostPosted: Mon Mar 22, 2021 10:12 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6224

Anything in the queue manager logs correlating with the time of connection failure?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
mvic
PostPosted: Wed Mar 24, 2021 9:44 am Post subject: Re: WebSphere can't connect to MQ after several days Reply with quote

Jedi

Joined: 09 Mar 2004
Posts: 2080

cmckenna wrote:
WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE')


Just in case you didn't see this already:

https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.tro.doc/q041290_.htm

There seem to be a few reasons you could get this condition. Possibly your QM's listener for the port the client wants to use is not running?
Back to top
View user's profile Send private message
cmckenna
PostPosted: Wed Mar 24, 2021 10:16 am Post subject: Reply with quote

Newbie

Joined: 22 Mar 2021
Posts: 6

Thanks for the link. I did make some changes to the teams playbook for when this happens again for gathering additional diagnostics and validation that in fact the qmanager is up and functional or not. Just waiting for the next occurance now.... based on history, whold be today or tomorrow
Back to top
View user's profile Send private message Send e-mail
cmckenna
PostPosted: Tue Mar 30, 2021 10:07 am Post subject: Reply with quote

Newbie

Joined: 22 Mar 2021
Posts: 6

Managed to capture some more information.... when the server starts is in its tailspin, there are dozens upon dozens of connections to the queue manage that are stuck in CLOSE_WAIT and FIN_WAIT_2 for over 10 minutes,

I validated that the listener is running during this and that it can be connected to (used telnet to port 1414) and dspmq shows the queue manager status as running.

A netstats on a health server never shows any connections in CLOSE_WAIT or TIME_WAIT_2
Back to top
View user's profile Send private message Send e-mail
RogerLacroix
PostPosted: Thu Apr 01, 2021 3:12 pm Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3195
Location: London, ON Canada

Sounds like the code is not closing the connection when it is done. Some developers think that if/when the code stops using the connection object that MQ will automatically close and disconnect from the queue manager. That is a false assumption. The code MUST close/disconnect the connection.

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportWebSphere can't connect to MQ after several days
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.