MQSeries.net :: View topic - dreaded 2009 reason code

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Java / JMS » dreaded 2009 reason code

Goto page 1, 2 Next

dreaded 2009 reason code

« View previous topic :: View next topic »

Author

Message

aks

Posted: Wed Aug 31, 2005 5:24 pm Post subject: dreaded 2009 reason code

Voyager

Joined: 19 Jul 2002
Posts: 84

I've searched the forums for 2009 errors and read countless entries on why this problem occurs and how to fix it, but haven't had any luck myself in this instance.

We recently migrated our entire Windows MQ 5.3 CSD 7/WebSphere 5.0.2 apps to AIX 5.3 running WebSphere App Server 6.0.1 and MQ 5.3 CSD 11, so a lot of variables have changed.

We have configured the WebSphere MQ queue connection factory and queue destinations to be the same as Windows, but still get a 2009 when calling the QueueSender.send method.

Here is what I have done so far:
1) MaxChannels and MaxActiveChannels set to 400 on AIX qmgr. Although when a netstat -an | grep 1414 |wc -l command is issued, the number is always around 10
2) Qmgr, hostname, channel and port all correct with the correct case in the names in the queue destination config in WAS
3) User id and password same in WAS config as what queue manager runs under
4) Coped the same com.ibm.mq.jar and com.ibm.mqjms.jar from mq (/usr/mqm/java/lib) to WAS (/apps/ibm/was/v6.0/appserver/lib/WMQ/java/lib) and restarted WAS instance - no difference in using jars either from mq or WAS

The error in my /var/mqm/errors/AMQERR01.log has this

Code:

09/01/05 10:21:06
AMQ6184: An internal WebSphere MQ error has occurred on queue manager QMBKDIR1.

EXPLANATION:
An error has been detected, and the WebSphere MQ error recording routine has
been called. The failing process is process 1294480.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier, and to save the generated output files. Contact your IBM support
center. Do not discard these files until the problem has been resolved.

The process 1294480 is /usr/mqm/bin/amqrmppa -m QMBKDIR1
There is an FDC file, but I can't decipher it (I can attach if it would help)

Web apps that make client connections using base MQI work OK - it is just this JMS MDB app that is failing.

Is there anything else I can check or try?
I am getting quite frustrated with it!

Thanks
Alan

jefflowrey

Posted: Thu Sep 01, 2005 3:57 am Post subject:

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

The FDC file will have a header block that starts it. You can post only that header block, and it may help us.
_________________
I am *not* the model of the modern major general.

csmith28

Posted: Thu Sep 01, 2005 9:41 am Post subject:

Grand Master

Joined: 15 Jul 2003
Posts: 1196
Location: Arizona

Code:

echo "dis chs(*)" | runmqsc | grep RUNNING | wc -l

Will give you an accurate count of RUNNING Channels though I don't think the number of channels active channels is related to this issue. If it were you would see entries in the /var/mqm/qmgrs/MQMGR/errors/AMQERR0*.LOG's.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial.

aks

Posted: Thu Sep 01, 2005 1:57 pm Post subject:

Voyager

Joined: 19 Jul 2002
Posts: 84

FDC header block is shown below.
I do not know if my problem is related, but the probe ID of RM046002 looks similar to what is generated by a problem described by IC46238: CHANNEL TERMINATES ON CLIENT CONNECTED TO SERVER AT 5.3FP10 OR HIGHER WITH PROBE RM046002 FDCS - INVALID DATA FORMAT
(http://www-1.ibm.com/support/docview.wss?uid=swg1IC46238)

This was introduced in Fix pack 10, but supposedly fixed in Fix Pack 11.
We intend to roll back Fix pack 11 today and try again, to rule this out
Thanks
Alan

Code:

csmith28

Posted: Thu Sep 01, 2005 2:23 pm Post subject:

Grand Master

Joined: 15 Jul 2003
Posts: 1196
Location: Arizona

OK, good luck.

Let us know how things turn out.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial.

hopsala

Posted: Thu Sep 01, 2005 2:31 pm Post subject:

Guardian

Joined: 24 Sep 2004
Posts: 960

aks wrote:

We recently migrated our entire Windows MQ 5.3 CSD 7/WebSphere 5.0.2 apps to AIX 5.3 running WebSphere App Server 6.0.1 and MQ 5.3 CSD 11, so a lot of variables have changed.

Well, if the whole CSD-rollback thing doesn't work (I don't imagine it will, but it's definitely worth a try) maybe you should try reverting to the last state to pinpoint the prob - is it in MQ or WAS; what I mean is:
1. Try the code on WebSphere 5.0.2 with MQ 5.3 CSD 7
2. Try the code on WebSphere 5.0.2 with MQ 5.3 CSD 11
3. Try the code on WebSphere App Server 6.0.1 with MQ 5.3 CSD 7
4. Try the code on WebSphere App Server 6.0.1 with MQ 5.3 CSD 11

I hope you had kept the previous environemnt up, if you didn't, there's a valuable lesson here

Anyhow, paste the code, so we can see what you're trying to do...

aks

Posted: Thu Sep 01, 2005 11:04 pm Post subject:

Voyager

Joined: 19 Jul 2002
Posts: 84

Fortunately (or unfortunately) when we rolled back to base MQ 5.3 on AIX and also Fix Pack 2, the 2009 problem went away, and the messages are flowing freely. No other configuration settings were changed.

I suppose I should report this to IBM, but I will have to reapply csd 11 to reproduce this, though AIX is a breeze for this sort of work - we completely uninstalled MQ but left the config under our /var/mqm directories, reinstalled it and everything worked again without any further configuration. The queues and channels and process definitions were completely intact. It certainly is a different story when dealing with the Windows version

Thanks for all your help
Alan

PeterPotkay

Posted: Fri Sep 16, 2005 7:26 am Post subject:

Poobah

Joined: 15 May 2001
Posts: 7722

Any resolution for RM046002?

We just upgraded our Windows 2000 MQ Queue Managers in DEV and QA to 5.3 CSD11, and keep seeing the below, while the JMS clients keep reporting 2009s. The IBM site says CSD11 fixes RM046002.

Code:

_________________
Peter Potkay
Keep Calm and MQ On

PeterPotkay

Posted: Thu Sep 22, 2005 5:23 pm Post subject:

Poobah

Joined: 15 May 2001
Posts: 7722

Looks this is a problem when the Queue Manager is at CSD11 and the Java Client is at CSD8 or CSD9.

From IBM:

Code:

The fixpack 11 adds a check to the server to defend the server against
malformed data flows which can cause heap corruption and outages.
Which maintenance levels the MQ clients are at? Unfortunately the
Java client does provide these malformed flows. Due to this the client
get 2009 error while trying to connect to the server. This is a
client
problem and hence need the client to upgrade to the fixpack 11. By
staying back at fixpack 06, this FDC will not be generated, but run the
risk of heap corruption which the customer was experienced.
.
Hence, we suggest, MQ clients also should be upgraded to fixpack 11.
Perhaps the clients should move forward to 11, keeping the server back
to fixpack 6, and later upgrading the server to 11 when all the clients
are at safe level. If the problem appears at this level, please
provide a
detailed trace from both server and the client taken at the same time
along with FDCs and the errorlogs.

Joy. Hunting down all the apps that are at CSD8 or CSD9 and telling them they need to upgrade ASAP, so I can go to CSD11 on my QMs in Production.
_________________
Peter Potkay
Keep Calm and MQ On

PeterPotkay

Posted: Thu Sep 29, 2005 3:02 pm Post subject:

Poobah

Joined: 15 May 2001
Posts: 7722

Thought I'd share the info related to this JMS / Java bug from the PMR.

Code:

#1. Do we have to apply IC47335 to the QM at CSD11, or only the MQ
Clients that are Java/JMS at CSD11?

#2. This link implies there is a problem with JMS / Java MQ clients at
CSD8, CSD9, CSD10 AND CSD11. Is this true? The only way to be safe is
to apply CSD11 with IC47335?

#3. Are clients at CSD7 or below safe?

#4. I have applied CSD11 to my desktop QM. I then applied IC47335. I
can now just zip up that whole dir, and give it to JMS clients, and
they will upgraded to the point where all the known bugs are taken care
of?

#5. Do we have any idea of what the implications are for JNDI objects
that were administered to LDAP using the CSD8/CSD9 service packs and
running in runtime and on the QM with CSD11?

#6. Does CSD 12 fix all this? If yes, when is it coming out? I have to
build 30+ MQ servers in the next 2 months, and need to plan accordingly.

#7. Is this bug present in MQ 6.0.0?

Here is JAVA Level 3's response:

Update: Investigating.
#1 Ifix IC47335 is related to the jar files used by JMS and Java
applications.
It must be applied on any machine where the JMS ana Java
application runs.

#2 CSD08 and CSD09 had pervasive problems when sing TCPIP
connectivity. These were resolved by CSD10 and CSD11.
The ifix files for CSD10 and CSD11 are to address other
lesser problems found since.
It is always adviseable to be at the latest level of service
because it is easier to supply service at the latest level.

#3 Clients at CSD07 and earlier will not be susceptible to the
TCPIP problems of CSD08 and CSD09.

#4 Yes.

#5 There are no known issues of JNDI objects defined at
CSD08/CSD09 level used at runtime CSD11.

#6 The contents of CSD12 have not been finalized so we cannot
be specific about which fixes will be included.
A CSD must undergo a testing procedure and fixes may be
added or removed during this process.

#7 IC47355 is not applicable to V6.

_________________
Peter Potkay
Keep Calm and MQ On

jefflowrey

Posted: Thu Sep 29, 2005 4:58 pm Post subject:

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

Lemme make sure I understand.

As long as I'm at CSD8 everywhere, I won't have your problem? It's only if I upgrade to CSD11 on my servers, that I need to upgrade everything else to CSD11?
_________________
I am *not* the model of the modern major general.

PeterPotkay

Posted: Thu Sep 29, 2005 6:13 pm Post subject:

Poobah

Joined: 15 May 2001
Posts: 7722

So I asked them that earlier on. "What gives? Since when does upgrading a QM cause MQClients to stop working?" IBM's response:

Quote:

The fixpack 11 adds a check to the server to defend the server against
malformed data flows which can cause heap corruption and outages.
Which maintenance levels the MQ clients are at? Unfortunately the
Java client does provide these malformed flows. Due to this the client
get 2009 error while trying to connect to the server. This is a
client
problem and hence need the client to upgrade to the fixpack 11. By
staying back at fixpack 08, this FDC will not be generated, but run the
risk of heap corruption which the customer was experienced.
.
Hence, we suggest, MQ clients also should be upgraded to fixpack 11.
Perhaps the clients should move forward to 11, keeping the server back
to fixpack 8, and later upgrading the server to 11 when all the clients
are at safe level. If the problem appears at this level, please
provide a
detailed trace from both server and the client taken at the same time
along with FDCs and the errorlogs.

Something else I learned? Look at this manifest of the com.ibm.mqjms.jar, and tell me what MQ version you think it is?

Code:

Implementation-Version: "5.306 - j5306-10-050413"

5.3 CSD6? Uh, no. CSD10! The -10- is the clue.
How about this one?

Code:

Implementation-Version: "5.300 - j530-L021008"

??? Who knows? All we know is its older than 5.3 CSD8.
_________________
Peter Potkay
Keep Calm and MQ On

jefflowrey

Posted: Thu Sep 29, 2005 6:17 pm Post subject:

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

So, basically, I should plan to move everyone to v6 sooner than a 5.3 CSD later than 8.
_________________
I am *not* the model of the modern major general.

PeterPotkay

Posted: Thu Sep 29, 2005 6:33 pm Post subject:

Poobah

Joined: 15 May 2001
Posts: 7722

Who knows. After this fiasco and finding the endmqm bug in 6.0 this week, its all a crap shoot. Don't touch anything unless you have to, and only then put the latest version on and pray it doesn't come back to bite you.

New QMs I am giving 6.0. Everything existing I am leaving at 5.3 CSD11 until at least next year, or until I forced to upgrade if some new bug is found. That's the problem with an MQClient Concentrator QM design. Constantly new clients are coming aboard, and sometimes the new clients have new bugs.
_________________
Peter Potkay
Keep Calm and MQ On

gstephen

Posted: Fri Oct 14, 2005 11:25 am Post subject:

Newbie

Joined: 16 Jun 2005
Posts: 7
Location: Toronto, Canada

PeterPotkay wrote:

Does this mean that if I put CSD11 in one QM in a cluster that I also have to upgrade the other QMs in the cluster?

Display posts from previous:

Goto page 1, 2 Next

Page 1 of 2

MQSeries.net Forum Index » IBM MQ Java / JMS » dreaded 2009 reason code

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP