| Author |
Message
|
| jshailes |
 Posted: Mon Jun 27, 2011 8:33 am Post subject: Should I use 1 server conn channel per application? |
 |
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
I'm having some problems in my java app whereby I lose messages because the listener thread seems to get destroyed and recreated at seemingly random times. There is a duration of between 20-60 seconds between the thread being destroyed and a new one created. It's proving very difficult to track down as no exception seems to be caught by the exception listener.
Since the problem seems to only have occurred over the past couple of weeks and something quite major that has changed recently is that I now have the application running on a live server as well as a dev box, I was wondering if there would be a problem with the 2 applications sharing the same server connection channel?
Some basic info of each instance of the application:
- 1 MQTopicConnectionFactory - Shares svr conn chl with other instances of the application, but doesn't share the port
- 3 listeners, each having different destinations.
- The WMQ server is running v7 |
|
| Back to top |
|
 |
| nandu123 |
| Posted: Mon Jun 27, 2011 9:00 am Post subject: |
|
|
Newbie
Joined: 10 Jan 2011
Posts: 5
Location: NewYork
|
| Quote: |
| I was wondering if there would be a problem with the 2 applications sharing the same server connection channel? |
This is not advisable if the security is setup based on the MCAUSER for each channel(application) connecting to MQ |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Jun 27, 2011 4:26 pm Post subject: Re: Should I use 1 server conn channel per application? |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| jshailes wrote: |
| ...I was wondering if there would be a problem with the 2 applications sharing the same server connection channel?.. |
No, there shouldn't be a problem, although its recommended to define a separate SVRCONN channel for each app. This makes it much easier to observe and control the client connections used by the apps (security, keepalive, heartbeat, maxmsgl, maxinst, stop chl, dis chs, etc). A typical standard for naming channels is <appacronym>.<qmgr>.CLIENT, although there is a 20 character length limitation on channel names.
_________________
Glenn |
|
| Back to top |
|
|
| jshailes |
| Posted: Tue Jun 28, 2011 1:15 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
Thanks very much for your input, I'll go ahead and create separate channels, although it sounds like it won't fix the problem maintaining a connection to the topic. Are there any logs that I can enable to understand whether the reason for dropping the connection is the fault of MQ?
In the back of my mind I keep wondering if when I used 'kill -9' on one of MQs processes because it was maxing out the CPU could be the cause of the problem, although I must say I feel like I'm clinging to straw on that one.
I've taken a screenshot of what happens with the listener threads in jprofiler, if anyone has any suggestions about how I could go about finding the problem I'd really appreciate it.
[img]http://tinyurl.com/mqScreenshot[/img][/img] |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jun 28, 2011 1:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| jshailes wrote: |
| In the back of my mind I keep wondering if when I used 'kill -9' on one of MQs processes because it was maxing out the CPU could be the cause of the problem, although I must say I feel like I'm clinging to straw on that one. |
NEVER kill -9 any MQ processes for any reason.
edit: except if all applications are shut down and your primary goal is to actually stop the whole queue manager.
You may or may not have caused the issue. You haven't established things like network configurations between the second application and the queue manager. There may well be a different firewall that is killing connections. |
|
| Back to top |
|
|
| jshailes |
| Posted: Tue Jun 28, 2011 2:03 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
| Quote: |
| NEVER kill -9 any MQ processes for any reason. |
but.. but it was so tempting  I ought to have listened to myself when I thought 'I really shouldn't be doing this'
| Quote: |
| You haven't established things like network configurations between the second application and the queue manager. |
I'm not quite sure I follow. There do not seem to be any differences in behaviour between the first and second application, they both drop connections. Regards network config, there is a dedicated server for MQ and 2 separate app servers.
| Quote: |
| There may well be a different firewall that is killing connections. |
There's very little in the way of security at present, no central firewall, boxes running CentOS 5 with SELinux disabled. [/quote] |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jun 28, 2011 3:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
One of the usual reasons for MQ channels to quit is that the network hardware has been configured to do things like drop connections that are open "too long". It wasn't clear that you were seeing the same behavior from both app servers - it sounded like you were only seeing it from one.
Note that I'm not talking about server hardware network configuration, I'm talking about things like switches and routers.
If your app environment is really not generating anything that resembles a cause for the listener thread to die, then MQ may have done so.
One would assume that if a listener thread was getting dropped, that it would generate at least some sort of diagnostic - if only an MQ FFST or FDC file that indicated something died horribly.
You may also have generated some FDCs when you kill -9 some processes. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Jun 28, 2011 3:55 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| jshailes wrote: |
| In the back of my mind I keep wondering if when I used 'kill -9' on one of MQs processes because it was maxing out the CPU could be the cause of the problem |
By the way.. which mq process you killed using kill -9 ?
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| jshailes |
| Posted: Tue Jun 28, 2011 4:36 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
| Quote: |
| By the way.. which mq process you killed using kill -9 ? |
I don't know unfortunately. I've just looked back though the shell history and only have a process number. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jun 28, 2011 4:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| It should have created an FDC or at least a note in the AMQERR files. |
|
| Back to top |
|
|
| jshailes |
| Posted: Tue Jun 28, 2011 5:05 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
| Quote: |
| It should have created an FDC or at least a note in the AMQERR files. |
The process was amqzlaa0.
Whilst looking for it in the logs I saw:
| Code: |
Process(3661.15) User(mqm) Program(amqzlaa0)
AMQ6709: The log for the Queue manager is full. |
Could this be why is was using 100% of the CPU? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jun 28, 2011 5:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| jshailes wrote: |
| Could this be why is was using 100% of the CPU? |
It could be. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Jun 28, 2011 5:15 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Having a lot of svrconn definitions does make CCDT maintenance more labourious.
I can see the point of naming them after apps - but I can identify apps by their MQ connection id. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jun 28, 2011 5:20 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
SVRCONNs should at least define application roles, even if not specific application instances.
Even if you have a security exit and are not relying on MCAUSER to enforce authorization roles, you should still have enough SVRCONNS so that you can properly track performance usage and connectivity. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Jun 28, 2011 6:32 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| jshailes wrote: |
| Quote: |
| It should have created an FDC or at least a note in the AMQERR files. |
The process was amqzlaa0.
Whilst looking for it in the logs I saw:
| Code: |
Process(3661.15) User(mqm) Program(amqzlaa0)
AMQ6709: The log for the Queue manager is full. |
Could this be why is was using 100% of the CPU? |
You should not be killing any process of qmgr like this.
You can check the status of cpu utilization, using 'topas' (on aix). you can approach OS Admin in order to increase the FS for mq. (Edit: Or Increase the no.of logs)
Good practice (not everyone follows that) is to have Different Logical Volumes being created for /var/mqm, /var/mqm/log/, /var/mqm/errors/, /usr/mqm/ (its part of architecture and space management)
Qmgr logs full can cause your qmgr crash !
Too many unclosed connections on the qmgr can keep the qmgr busy and if the connections are old and unused then its bad for the system as every connection occupies some part of memory.
As a separate note, You should be reading and implementing the TCP and CHANNEL stanzas of qm.ini to close the old unused connections.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
|
|
