| Author |
Message
|
| cmckenna |
 Posted: Mon Mar 22, 2021 5:48 am Post subject: WebSphere can't connect to MQ after several days |
 |
|
Newbie
Joined: 22 Mar 2021
Posts: 6
|
I am currently supporting a very old application running on WebSphere version 7.0.0.19 and using MQ version 7.0.1.2.
After 4 to 5 days of the application running just fine, we suddenly start getting the following errors:
[3/19/21 19:14:03:158 EDT] 0000001c SibMessage W [:] CWSJY0003W: JMSCC3036: An exception has been delivered to the connections exception listener: '
Message : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ1107: A problem with this connection has occurred. This exception has been delivered due to an exception listener being registered on this connection. Use the linked exception to determine the cause of this error.
Class : class com.ibm.msg.client.jms.DetailedIllegalStateException
Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE').
Class : class com.ibm.mq.MQException
If we recycle WAS and MQ, the application is fine again for another 4 to 5 days before the cycle repeats.
I have checked the connection pools during the issue and there are plenty of free connections so I don't believe we have a leak.
Also, WAS and MQ are running on the same server and there is NO SSL involved yet the reason code seems to indicate SSL.
I also grabbed hunderds of MB of websphere trace logs from the latest incident and see nothing in those traces that is any more helpful than the above mentioned stack trace. |
|
| Back to top |
|
 |
| exerk |
| Posted: Mon Mar 22, 2021 8:18 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
There are a few hits for exception code JMSWMQ1107, such as IC68653 which seems to be the most relevant to you.
Which method of connecting WAS to the queue manager are you using? As you mention SSL I'm assuming client rather than bindings mode, but would like it confirmed (or not).
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| cmckenna |
| Posted: Mon Mar 22, 2021 8:51 am Post subject: Client mode |
|
|
Newbie
Joined: 22 Mar 2021
Posts: 6
|
|
| Back to top |
|
|
| exerk |
| Posted: Mon Mar 22, 2021 9:04 am Post subject: Re: Client mode |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| cmckenna wrote: |
| Client mode is correct |
As they are on the same server, why?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| cmckenna |
| Posted: Mon Mar 22, 2021 9:08 am Post subject: |
|
|
Newbie
Joined: 22 Mar 2021
Posts: 6
|
| Wish I had a good answer for that. This app is about 12 years old and I inherited it when I joined this new company a couple months ago. I have bee trying to get as much detail about the design and implementation but the developers and architects are long gone |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Mar 22, 2021 9:15 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| cmckenna wrote: |
| Wish I had a good answer for that. This app is about 12 years old and I inherited it when I joined this new company a couple months ago. I have bee trying to get as much detail about the design and implementation but the developers and architects are long gone |
I feel your pain...
1. How long before you inherited this did it start, i.e. does this particular system have a history of this, or is it recent?
2. What recent changes have been made to the system, e.g. OS, if any?
3. Has changing it to bindings mode been floated with management?
I have a suspicion that a decision was made long ago to use client mode 'in case' the resource usage of the server got to the point where MQ got 'moved' to a different server, but that's speculation on my part.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| cmckenna |
| Posted: Mon Mar 22, 2021 9:46 am Post subject: |
|
|
Newbie
Joined: 22 Mar 2021
Posts: 6
|
again, its difficult to ascertain all the details around this.... here is what I have been told....
the issue according to some, started in early fall 2020.
They had migrated this app to AIX prior to my arrival although no one remembers the date this was done.
As far as I know, no discussion has been had regarding binding mode.
Sorry I can not be of more help. I have a ton of WAS experience going back to version 3 but I fall short with MQ and JMS messaging. Everything I supported in the past was apps talking to actual external Message Brokers with a dedicated team of MQ experts to handle the support on that side.
My new place is a small shop that does not have a lot of expertise.... |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Mar 22, 2021 10:12 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Anything in the queue manager logs correlating with the time of connection failure?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Mar 24, 2021 9:44 am Post subject: Re: WebSphere can't connect to MQ after several days |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
|
| Back to top |
|
|
| cmckenna |
| Posted: Wed Mar 24, 2021 10:16 am Post subject: |
|
|
Newbie
Joined: 22 Mar 2021
Posts: 6
|
| Thanks for the link. I did make some changes to the teams playbook for when this happens again for gathering additional diagnostics and validation that in fact the qmanager is up and functional or not. Just waiting for the next occurance now.... based on history, whold be today or tomorrow |
|
| Back to top |
|
|
| cmckenna |
| Posted: Tue Mar 30, 2021 10:07 am Post subject: |
|
|
Newbie
Joined: 22 Mar 2021
Posts: 6
|
Managed to capture some more information.... when the server starts is in its tailspin, there are dozens upon dozens of connections to the queue manage that are stuck in CLOSE_WAIT and FIN_WAIT_2 for over 10 minutes,
I validated that the listener is running during this and that it can be connected to (used telnet to port 1414) and dspmq shows the queue manager status as running.
A netstats on a health server never shows any connections in CLOSE_WAIT or TIME_WAIT_2 |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Thu Apr 01, 2021 3:12 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Sounds like the code is not closing the connection when it is done. Some developers think that if/when the code stops using the connection object that MQ will automatically close and disconnect from the queue manager. That is a false assumption. The code MUST close/disconnect the connection.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
|
|
