| Author |
Message
|
| queuetip |
 Posted: Wed Feb 15, 2006 10:54 am Post subject: Listener Conflict Causes QMgr To Hang In Windows |
 |
|
Acolyte
Joined: 03 Feb 2005
Posts: 67
|
Here's the scenario that is driving me crazy...queue manager called HITD comes up and is listening to port 1465. Four hours later it goes down - assuming another program is trying to use the same port. How can I prevent this?
Here is the queue manager error log message:
2/15/2006 08:01:55
AMQ9218: The TCP/IP listener program could not bind to port number 1465.
EXPLANATION:
An attempt to bind the TCP/IP socket to the listener port was unsuccessful.
ACTION:
The failure could be due to another program using the same port number. The
return code from the 'bind' call for port 1465 was 10013. Record these values
and tell the systems administrator.
----- amqclita.c : 570 --------------------------------------------------------
Now, in the MQSeries error log (higher level)...you can see it retries.
2/15/2006 08:01:55
AMQ7882: Attempting to recover HITD/Listener WebSphere MQ service.
EXPLANATION:
The WebSphere MQ service has detected that HITD/Listener has failed, and is
attempting to restart it.
ACTION:
No Action Required.
-------------------------------------------------------------------------------
2/15/2006 08:01:55
AMQ7880: Error code 0 starting HITD/Listener WebSphere MQ service.
EXPLANATION:
The service was unable to start HITD/Listener. The error message reported was
as follows: The process has terminated with return code 20.
ACTION:
Use WebSphere MQ Services to investigate why the service could not begin. If
recovery for this service is active, MQ will attempt to recover.
-------------------------------------------------------------------------------
2/15/2006 08:02:26
AMQ7883: HITD/Listener WebSphere MQ service started from recovery.
EXPLANATION:
The WebSphere MQ service has successfully recovered HITD/Listener.
ACTION:
No Action Required.
-------------------------------------------------------------------------------
I have read in a prior post that sometimes the admins are trying to start the queue manager twice during reboot...since the queue manager had been up and running 4 hours I doubt this is the problem. Thus I am thinking another program is truly going for that port. 
Is there a way to identify the program that tried to use the same port? Any other creative ideas?
Thanks! |
|
| Back to top |
|
 |
| wschutz |
| Posted: Wed Feb 15, 2006 11:17 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
Two thoughts from this related post:
http://www.mqseries.net/phpBB2/viewtopic.php?t=19061
(1) are there any FFST (.FDC files) generated around that time? If so, can you paste the header portion;
(2) try running "runmqlsr" in the foreground in a command window.
btw...what OS is this? (EDIT: yes, windows, of course...) I suspect the problem isn't that some "other" program is using the port, I would think its related to runmqlsr aborting. You need to figure out why that is happening ...
_________________
-wayne
Last edited by wschutz on Wed Feb 15, 2006 11:25 am; edited 1 time in total |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Feb 15, 2006 11:21 am Post subject: Re: Listener Conflict Causes QMgr To Hang In Windows |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| queuetip wrote: |
EXPLANATION:
An attempt to bind the TCP/IP socket to the listener port was unsuccessful.
ACTION:
The failure could be due to another program using the same port number. The
return code from the 'bind' call for port 1465 was 10013. |
I looked up the Win32 error 10013 : "An attempt was made to access a socket in a way forbidden by its access permissions."
I can't recall ever seeing this error condition myself. Also (note to self: do some reading here) I wasn't aware access permissions could be applied to a socket.
If bind() fails, it is usually because something else in the system is already bound to the given port on the given interface. I think this isn't such a "usual" bind() failure.
Is there a .FDC file matching the time of this occurrence?
You're on Windows, but what version / service pack? What release / fix pack of MQ? Any unusual network hardware in the machine? Is it a plain ethernet adapter or something else? |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Feb 15, 2006 11:24 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
Probably not relevant enough, IMHO. That topic shows recv() failures, this is a bind() failure. |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Feb 15, 2006 11:28 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| mvic wrote: |
Probably not relevant enough, IMHO. That topic shows recv() failures, this is a bind() failure. |
Well yes, but something is causing the listener to fail in the first place.
_________________
-wayne |
|
| Back to top |
|
|
| queuetip |
| Posted: Wed Feb 15, 2006 12:09 pm Post subject: |
|
|
Acolyte
Joined: 03 Feb 2005
Posts: 67
|
| wschutz wrote: |
| (1) are there any FFST (.FDC files) generated around that time? If so, can you paste the header portion |
Unfortunately, no FDC files were produced.
| wschutz wrote: |
| try running "runmqlsr" in the foreground in a command window. |
I did this for the listener that was already running. Assuming you wanted to see if I could recreate the problem. Nope. I got this message: AMQ9255: Listener already running.
When I looked in the event log - I did not see the original errors. Just this one.
| wschutz wrote: |
| btw...what OS is this? (EDIT: yes, windows, of course...) |
IBM WebSphere MQ for Windows, Version 5.3 (haven't applied any service packs yet)...running on Windows Server 2003.
| wschutz wrote: |
| I suspect the problem isn't that some "other" program is using the port, I would think its related to runmqlsr aborting. You need to figure out why that is happening ... |
Can you provide some direction?
Thanks! |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Feb 15, 2006 12:14 pm Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| Quote: |
wschutz wrote:
try running "runmqlsr" in the foreground in a command window.
I did this for the listener that was already running. Assuming you wanted to see if I could recreate the problem. Nope. I got this message: AMQ9255: Listener already running.
When I looked in the event log - I did not see the original errors. Just this one.
|
How do you recover from this problem? Recycle MQ? Recycle the machine?
My thought was to stop the listener that comes up when MQ comes up and run a listener in the foreground in a dos window, so that four hours later you "might" see some useful information on the console. Just a thought ...
EDIT: It might also be interesting to see what "netstat -an" has to say about that port.. it can tell you if the problem is truely that some process is listening on that port...
_________________
-wayne |
|
| Back to top |
|
|
| mvic |
| Posted: Thu Feb 16, 2006 1:26 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| queuetip wrote: |
| IBM WebSphere MQ for Windows, Version 5.3 (haven't applied any service packs yet)...running on Windows Server 2003. |
If you haven't applied any CSDs to MQ 5.3, you're running code released in 2002, if I remember correctly. There have been 100s of code fixes since then. CSD12 is the latest.
(Also, if memory does not fail me, I think MQ 5.3 was released in 2002 without support for Windows 2003; this was added later).
Are there any other entries in the qmgr error logs around 2/15/2006 08:01:55 ? As Wayne points out, we need to find out if there were any precursors to the bind() failure, as these might explain the failure itself.
The 10013 is still confusing me, though. Anyone ever seen a 10013 from bind() before? Could it be that this is something only seen on Windows 2003 ? |
|
| Back to top |
|
|
| SAFraser |
| Posted: Thu Feb 16, 2006 9:56 am Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
There were definite listener problems with unpatched MQ 5.3 in Windows-- such as listeners hanging and listeners disappearing (I had it happen to me). Your problem may be different, but I sure would patch that installation as soon as possible. It was one of the early 5.3 patches that corrected the problem, and the patch really did work.
Just a side note... our standard installation procedures call for patching MQ as soon as it is installed. We don't even try to create a queue manager on an unpatched installation.
Shirley |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Feb 16, 2006 2:23 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Yup, we saw the dissappearing Listeners too, I think we were on CSD3 or CSD4 at the time. No problems once we upgraded.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 16, 2006 4:53 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PeterPotkay wrote: |
| Yup, we saw the dissappearing Listeners too, I think we were on CSD3 or CSD4 at the time. No problems once we upgraded. |
The listeners loosing the port config and such were if my memory serves me well at CSD06. Another reason to get to CSD11/12. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
