Author |
Message
|
CoreySanders |
Posted: Thu Apr 10, 2008 5:56 am Post subject: MQRC_UNEXPECTED_ERROR (2195) During get with wait |
|
|
Novice
Joined: 20 Feb 2008 Posts: 11
|
I have code that is running a loop that does a get with wait for 60 seconds, time out, get with wait, etc. We are letting this application run idle; no data. Every few days or so I am seeing MQRC_UNEXPECTED_ERROR (2195) being thrown by the get. This can persist for as little as 20 seconds or as long as 3 hours. If I am patient, the errors go away on their own.
I've been over the documentation in as much detail as I can. There isn't much on 2195, but the general feel I get is that 2195 it most likely caused due to a system resource issue. I had the sysadmins check the recommended kernel settings on both the client and servers systems. We check out.
The documentation also says that in the event a 2195 occurs, information will be written to the MQ logs on the server and an FDC file will be generated in /var/mqm/errors. I am getting neither piece of information. According to the docs, this can happen if the number of files handles has been exhausted. I wrote a little script using lsof that will list all the user owned processes and count the file descriptors. The last time the 2195 occured, I ran my script and we were nowhere near the system limit. The client application had 116 open files. No process on the server side had more than 64 open files.
We are going to open a ticket with IBM support. But I wonder if anyone here has any suggestions on how to further debug this issue. Environment details are listed below.
MQ Server: 5.306 on Sun Solaris 9
MQ Client : Java code using MQBase APIs running under IBM JDK 1.5.0_11 on Redhat Enterprise Linux AS 4 |
|
Back to top |
|
 |
jefflowrey |
Posted: Thu Apr 10, 2008 6:02 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
Wherever there is a client connection, you have to consider network issues too. So if there's a firewall in between or etc, you should look at the logs for that.
If you aren't seeing *any* FDCs, and nothing in either /var/mqm/errors or /var/mqm/qmgrs/<qm>/errors... and nothing in /var/mqm/errors on the client side... This leads a lot of credence to it being a network error. But there should be something in *one* of those places. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
CoreySanders |
Posted: Thu Apr 10, 2008 6:24 am Post subject: |
|
|
Novice
Joined: 20 Feb 2008 Posts: 11
|
We don't have a /var/mqm/errors directory on the client side. Our normal procedure for client "installation" is to lay down our java code and then copy the necessary MQ jars from the server. Is this incorrect? Should I be doing a full MQ client install on the client servers? We've been fine in the past without doing a client install, but I can see how that might be incorrect.
There is also some question about whether the server is logging correctly or not. I have the sysadmins investigating that issue. Our log files haven't changed since 2/28. It seems to me that something should have been written to the logs in a given month. In production, we ususally see channel start and stop events frequently in the logs. Any ideas why the logs might not be getting updated?
Thanks for your help. |
|
Back to top |
|
 |
jefflowrey |
Posted: Thu Apr 10, 2008 6:28 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
It's better to apply the full client, because then you know what level the JAR files your app are using is - rather than having every Java app potentially have it's own and different version of the JARs.
If the log files aren't getting written to (the error files, that is...) then it's possible that the file system is full or the files have reached a ulimit. Both of those should have caused FDCs... unless somehow the FDCs couldn't be written... but in that case the qmgr should have entirely stopped. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
PeterPotkay |
Posted: Thu Apr 10, 2008 7:28 am Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
Jeff is not wrong in saying you need the full MQ client for Java.
I am not wrong in saying you can just use the jars and nothing else.
There is no official IBM position on this. Yet. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
CoreySanders |
Posted: Thu Apr 10, 2008 7:34 am Post subject: |
|
|
Novice
Joined: 20 Feb 2008 Posts: 11
|
I just learned that our version of MQ is end of life, so we can't get support on this issue. We are going to try to update to fix pack 14 and see if that fixes the issue. I do notice that a couple of 2195 issues were fixed between 6 and 14. Hopefully that will help.
We are also pursuing an upgrade to the next version of MQ (6.0). Any tips on gotchas we might encounter due to the upgrade would be appreciated. We are a little nervous as to how large a change an upgrade will be. |
|
Back to top |
|
 |
jefflowrey |
Posted: Thu Apr 10, 2008 7:48 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
That's interesting. I thought everyone knew that MQ v5.3 was out of support.
And, again, this is one reason why you should be using the full MQ client on your machines. You will know exactly what version and level is deployed.
With your setup, you have to rebuild and repackage all of your applications. If you were pointing to the client install... you wouldn't.
It's also recommended to keep up with maintenance on a six month cycle - or not any less often than you make major releases of your application.
This goes a long way towards reducing the fear of change and giving you a firm understanding of the impact of the change. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
Nigelg |
Posted: Thu Apr 10, 2008 1:46 pm Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004 Posts: 1046
|
The Java client returns 2195 if its network connection is interrupted, rather than a 2009. _________________ MQSeries.net helps those who help themselves.. |
|
Back to top |
|
 |
CoreySanders |
Posted: Mon Apr 14, 2008 9:32 am Post subject: |
|
|
Novice
Joined: 20 Feb 2008 Posts: 11
|
Nigelg wrote: |
The Java client returns 2195 if its network connection is interrupted, rather than a 2009. |
I have been trying to reproduce the 2195 due to network interruption in my development environment and have been unsuccessful. I either get a 2059 or a 2009 error depending on how and when/where I interrupt the networking. I've tried pulling the cable (2009), using iptables to DROP/REJECT traffic (connection just hangs), and using a proxy that I shutoff in the middle of the wait (2009). Are there specific conditions under which the 2195 can occur due to network reasons? Any suggestions on how to reproduce this behavior?
Thanks. |
|
Back to top |
|
 |
|