ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » com.ibm.mq.MQException: MQJE001: Reason 2009

Post new topic  Reply to topic Goto page 1, 2  Next
 com.ibm.mq.MQException: MQJE001: Reason 2009 « View previous topic :: View next topic » 
Author Message
vicks_mq
PostPosted: Tue Oct 03, 2017 1:47 pm    Post subject: com.ibm.mq.MQException: MQJE001: Reason 2009 Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

our JBOSS application using MQ Resource Adaptor is getting the following error once a while when making MQ transactions.
com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2009

We checked the below IBM MQ support page for the root cause of this problem.
MQ connection is terminating with error code 2009


1. A firewall that is terminating the connection (YES, we had a major network firewall change in the last few months)
2. An IOException that causes the socket to be closed
3. An explicit action to cause the socket to be closed by one end (no explicit action is being done)
4. The queue manager is offline (that is not the case)
5. The maximum number of channels allowed by the queue manager are open (not the case as we have allowed upto 5,000 channels and during the error time we see only 10 of them)
6. A configuration problem in the Queue Connection Factory (QCF)
(no changes has been occurred in QCF),.

We asked our network firewall team to check if there is any drop connection between JBOSS server and MQ server but they are not able to find any such drops.

so we are kind of lost on what should be our next steps to resolve this issue.
Any help would be highly appreciated.
Back to top
View user's profile Send private message
tczielke
PostPosted: Tue Oct 03, 2017 1:54 pm    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

If I was the MQ admin here, I would go to the JBOSS application developers and ask them why their application is not recovering properly from a 2009 (broken connection) error. This is a requirement of any properly written application to handle errors, recover properly, etc. Don't let the developers dupe you into thinking this is a network issue . . .
_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
vicks_mq
PostPosted: Tue Oct 03, 2017 2:14 pm    Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

Quote:
If I was the MQ admin here, I would go to the JBOSS application developers and ask them why their application is not recovering properly from a 2009 (broken connection) error. This is a requirement of any properly written application to handle errors, recover properly, etc. Don't let the developers dupe you into thinking this is a network issue . . .


we have 2 applications which has reported this error, and one of them is successfully recovering from this error bu they still concerned why this error is showing in their logs. and they expect MQ admin to provide resolutions.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Oct 03, 2017 2:16 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2491
Location: Melbourne, Australia

It could be a transient network issue, or an issue with the TCP stack / OS at either end. For some reason, the TCP socket session between the client (MQ Resource adapter) and server (MQ queue manager) is dropping out. MQ cannot recover this internally. The app needs to recover its state, reconnect to MQ, and continue processing. If this happens often, you will need to investigate the MQ error logs and engage a network specialist.
_________________
Glenn
Back to top
View user's profile Send private message
vicks_mq
PostPosted: Tue Oct 03, 2017 3:24 pm    Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

Quote:
It could be a transient network issue, or an issue with the TCP stack / OS at either end. For some reason, the TCP socket session between the client (MQ Resource adapter) and server (MQ queue manager) is dropping out. MQ cannot recover this internally. The app needs to recover its state, reconnect to MQ, and continue processing. If this happens often, you will need to investigate the MQ error logs and engage a network specialist.


we are not getting written anything in our MQ logs or QMGR logs, but the applications are reporting 2009 error.
Back to top
View user's profile Send private message
vicks_mq
PostPosted: Tue Oct 03, 2017 4:22 pm    Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

Quote:
It could be a transient network issue, or an issue with the TCP stack / OS at either end. For some reason, the TCP socket session between the client (MQ Resource adapter) and server (MQ queue manager) is dropping out. MQ cannot recover this internally. The app needs to recover its state, reconnect to MQ, and continue processing. If this happens often, you will need to investigate the MQ error logs and engage a network specialist.


Could this issue be related to Server Connection Channel? Like changing any configuration parameters in SVRCONN channel will help in any way?
Back to top
View user's profile Send private message
vicks_mq
PostPosted: Tue Oct 03, 2017 5:11 pm    Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

we have the parameter DiSCINT on our "SERVER CONNECTION CHANNEL" set to 0, could that be a problem?
Back to top
View user's profile Send private message
vicks_mq
PostPosted: Tue Oct 03, 2017 7:11 pm    Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

that might not be a problem as per IBM documentation

You can specify any number of seconds from zero through 999 999 where a value of zero means no disconnect; wait indefinitely.


For server-connection channels using the TCP protocol, the interval represents the client inactivity disconnect value, specified in seconds.
If a server-connection has received no communication from its partner client for this duration, it terminates the connection. The server-connection inactivity interval applies under the following circumstances:
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 04, 2017 4:44 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vicks_mq wrote:
we have the parameter DiSCINT on our "SERVER CONNECTION CHANNEL" set to 0, could that be a problem?


It means that in the event of a transient network problem (such as my worth associate describes) the connection will "fail" with a 2009 (connection broken) error because you've told it to stay connected forever.

For this reason it's unusual to use zero as a value - why did you select this? What's the reasoning / use case?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Oct 04, 2017 5:12 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

Actually zero used to be the norm in MQ v6 before the DISCINT setting was available for svrconn channels.

Which is why migrated and new svrconn channels have that value in MQ v7 (and later) by default. It's therefore not unusual for a server connection channel.

Having a lowish DISCINT value can cause MQRC 2009 issues because MQ will drop the channel for inactivity, when the app might still be up.

Something fairly large, but not indefinite might be a good value to use.

As said elsewhere, apps should be coded to reconnect on 2009 (but use a sleep delay first to avoid a high speed loop). MQ client auto-reconnect is one (somewhat flawed way) to do this.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 04, 2017 5:46 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

zpat wrote:
As said elsewhere, apps should be coded to reconnect on 2009 (but use a sleep delay first to avoid a high speed loop).




But also have some kind of threshold so that they're not trying to reconnect forever. At some point they need to throw an error along the lines of:

"Look, the connection dropped, I've been trying it and it's just not happening so now you need to use those opposable thumbs and demonstrate that you're the dominant life form on this planet. For now."
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Oct 04, 2017 6:15 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9392
Location: US: west coast, almost. Otherwise, enroute.

... and then issue the uninformative error message 'try again later'.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 04, 2017 6:21 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

bruce2359 wrote:
... and then issue the uninformative error message 'try again later'.





Or "an error has occurred".

Or "Application function failed"

My personal favorite - "some bad stuff went down". Which did actually help with problem resolution, indicating that an early development code version had escaped into production.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Wed Oct 04, 2017 2:41 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2491
Location: Melbourne, Australia

vicks_mq wrote:
we are not getting written anything in our MQ logs or QMGR logs, but the applications are reporting 2009 error.

If an active SVRCONN connection drops out, there should be something in the qmgr's error logs, because it is an unexpected failure. eg.

Code:
----- amqrccca.c : 1066 -------------------------------------------------------
10/05/17 00:16:47 - Process(30277694.3508483) User(mqm) Program(amqrmppa)
                    Host(xxxxx) Installation(Installation1)
                    VRMF(7.5.0.6) QMgr(MQPxxxxx)

AMQ9208: Error on receive from host 10.62.2.22 (10.62.2.22).

EXPLANATION:
An error occurred receiving data from 10.62.2.22 (10.62.2.22) over TCP/IP. This
may be due to a communications failure.
ACTION:
The return code from the TCP/IP read() call was 73 (X'49'). Record these values
and tell the systems administrator.
----- amqccita.c : 4062 -------------------------------------------------------
10/05/17 00:16:47 - Process(30277694.3508483) User(mqm) Program(amqrmppa)
                    Host(xxxxx) Installation(Installation1)
                    VRMF(7.5.0.6) QMgr(MQPxxxxx)

AMQ9999: Channel 'MYAPP.SVRCONN' to host '10.62.2.22' ended abnormally.

EXPLANATION:
The channel program running under process ID 30277694 for channel
'MYAPPP.SVRCONN' ended abnormally. The host name is '10.62.2.22'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.

_________________
Glenn
Back to top
View user's profile Send private message
vicks_mq
PostPosted: Thu Oct 05, 2017 6:00 am    Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

Hi Glenn,

we sometimes have the similar error in our QMGR logs for the SVRCONN channel in questions but this error timing is not matching what has been recorded by application log when 2009 error occurs .

so we are not sure if this error message in our QMGR logs relates to application getting 2009 error.

Quote:
----- amqccita.c : 4116 -------------------------------------------------------
10/04/2017 06:47:54 PM - Process(7174.93491) User(mqm) Program(amqrmppa)
Host(debug-mq.host.com) Installation(Installation1)
VRMF(8.0.0.2) QMgr(PRDQMGR1)

AMQ9209: Connection to host 'jboss-hostname (11.123.431.12)' for channel
'ABC.XYZPRD' closed.

EXPLANATION:
An error occurred receiving data from 'jboss-hostname (11.123.431.99)' over
TCP/IP. The connection to the remote host has unexpectedly terminated.

The channel name is 'ABC.XYZPRD''; in some cases it cannot be determined and
so is shown as '????'.
ACTION:
Tell the systems administrator.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » com.ibm.mq.MQException: MQJE001: Reason 2009
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.