| Author |
Message
|
| rparti |
 Posted: Mon May 07, 2007 2:42 pm Post subject: Queue Manager issue |
 |
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
We have a polling service that picks up messages from a db table and pushes it to a queue. Once it "sees" the message in the db, it marks it as a "sent message".
Anyway, at some point today, the messages were not getting to the queue. We resent the messages (by setting the status in the db field to "not sent", so the polling service to pick it up again). I know that this is a bad design....
The messages still did not go though. However on restarting the queue manager, the message flow was restored.
According to my speculation the polling service is very badly designed. It is probably creating sessions or connections that it is not closing properly.
The restart of the QM killed all these "bad resources"
The 3 qmgr errors folders don't give me any hint to the problem.
Could anyone shed any light, as to how a QM restart fixed the problem.
Also apologize if I posted to the wrong Forum.
Thanks and Regards |
|
| Back to top |
|
 |
| ashu |
| Posted: Mon May 07, 2007 8:01 pm Post subject: |
|
|
Centurion
Joined: 11 Nov 2006
Posts: 132
|
Hello,
Does the polling service open a connection every time it has to put a message to the Queue? or is the connection opened only once at the start of the service? I think there is a limitation to the number of connections possible on the QMGR.If the program is opening the connection in the loop that is polling the db then there is a high chance that this number is met and all the further activity is stopped, which also explains why restarting the QMGR resolved the isseue... 
Just a guess...
Regards,
_________________
Ashu
"It is simple to be Happy but difficult to be Simple" |
|
| Back to top |
|
|
| rparti |
| Posted: Tue May 08, 2007 6:28 am Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
Thanks for the reply.
Shouldn't the error logs indicate that the connection limit is reached? I think there should be an indication somewhere. Please advise.
Thanks |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue May 08, 2007 6:30 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The error logs *somewhere* might indicate that too many connections were created.
More guaranteed, though, is that the attempt to create a connection will get back an error that says that the connection couldn't be established.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| rparti |
| Posted: Tue May 08, 2007 7:12 am Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
I thought so too. However there is no error in the logs at all. That is why I believe that there could be some other issue.
Refer to logs below. It took 12 secs to shutdown the QMGR, and in between there are a lot of "AMQ9542: Queue manager is ending." messages. Is this reflective of the number of connections that the QMGR is trying to shutdown?
What conditions would really need a QMGR restart?
This is what I see in the logs
-------------------------------------------------
One week ago 08:09:29
----- amqccita.c : 2761 -------------------------------------------------------
AMQ9208: Error on receive from host is~abc (x.x.x.x).
EXPLANATION:
An error occurred receiving data from is~abc (x.x.x.x) over TCP/IP.
This may be due to a communications failure.
ACTION:
The return code from the TCP/IP (recv) call was 10054 (X'2746'). Record these
values and tell the systems administrator.
The next entry is when we issued the shutdown command
----- amqccita.c : 2761 -------------------------------------------------------
5/7/2007 10:40:41
AMQ9411: Repository manager ended normally.
EXPLANATION:
The repository manager ended normally.
ACTION:
None.
-------------------------------------------------------------------------------
5/7/2007 10:40:42
AMQ9542: Queue manager is ending.
EXPLANATION:
The program will end because the queue manager is quiescing.
ACTION:
None.
........a lot of AMQ9542: Queue manager is ending.
5/7/2007 10:40:54
AMQ8004: WebSphere MQ queue manager 'FUSMSGRTR' ended.
EXPLANATION:
WebSphere MQ queue manager 'FUSMSGRTR' ended.
ACTION:
None. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue May 08, 2007 7:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rparti wrote: |
Refer to logs below. It took 12 secs to shutdown the QMGR, and in between there are a lot of "AMQ9542: Queue manager is ending." messages. Is this reflective of the number of connections that the QMGR is trying to shutdown?
|
Slightly pedantic, but unless you specify an "immediate" flag, the queue manager will wait for connections to be closed rather than close them down. I suspect the large number of messages is the queue manager waiting for all the applications to disconnect.
For what it's worth, a few hours ago one of our test systems hit maximum connections when some chimp decided to see if his automated performance test script worked (it did - ramped the queue manager up nicely. Shame about the rest of the users.). We had people getting 2009 errors from the GUI hand over fist, but not a sniff in the queue manager log.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rparti |
| Posted: Tue May 08, 2007 7:27 am Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
Thanks Victor.
| Vitor wrote: |
For what it's worth, a few hours ago one of our test systems hit maximum connections |
How can we tell that the maximum nubmer of connections is hit if it does not log anywhere? |
|
| Back to top |
|
|
| ashu |
| Posted: Tue May 08, 2007 7:34 am Post subject: |
|
|
Centurion
Joined: 11 Nov 2006
Posts: 132
|
As Vitor and Jeff have said...
You should get:
| Quote: |
| Reason Code 2009 (MQRC_BROKEN_CONNECTION) |
_________________
Ashu
"It is simple to be Happy but difficult to be Simple" |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue May 08, 2007 7:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rparti wrote: |
Thanks Victor.
| Vitor wrote: |
For what it's worth, a few hours ago one of our test systems hit maximum connections |
How can we tell that the maximum nubmer of connections is hit if it does not log anywhere? |
There are a number of solution. In prod we have BMC Patrol, in this instance we noticed from everyone getting 2009 there was a problem, deduced the cause from eliminating other likely 2009 causes and checked the box to verify it.
I commend the "Monitoring MQ" manual to your attention.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rparti |
| Posted: Tue May 08, 2007 7:40 am Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
|
| Back to top |
|
|
|
|
