| Author |
Message
|
| wkucard |
 Posted: Mon Jun 23, 2014 11:58 am Post subject: Connection pool limits? |
 |
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
| We are running MB 8 and MQ 7.5. We are moving a high volume of client machines over to make connections to this system for the first time, and while this is mostly working, it seems like there is some sort of hidden connection limit. We are hitting a threshold right around 170 connections whereby additional SSH connections are accepted initially but then hit a wait state. Is anyone aware of any Linux or Broker limitation/configuration that would put in place a limitation on the number of active connections at once? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Mon Jun 23, 2014 12:10 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Look at the .ini file for the qmgr in question. What are maxchannel and maxactivechannel values?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jun 23, 2014 12:17 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bruce2359 wrote: |
| Look at the .ini file for the qmgr in question. What are maxchannel and maxactivechannel values? |
Assuming these are MQ clients, and not TCPIP clients or HTTPS clients or or or or |
|
| Back to top |
|
|
| wkucard |
| Posted: Mon Jun 23, 2014 12:17 pm Post subject: |
|
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
Would it be possible that those values would not exist?
qm.ini and mqs.ini does not have an entry for that |
|
| Back to top |
|
|
| wkucard |
| Posted: Mon Jun 23, 2014 12:18 pm Post subject: |
|
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
| mqjeff wrote: |
| bruce2359 wrote: |
| Look at the .ini file for the qmgr in question. What are maxchannel and maxactivechannel values? |
Assuming these are MQ clients, and not TCPIP clients or HTTPS clients or or or or |
We are connecting direct to the broker using MQTT. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jun 23, 2014 12:24 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| wkucard wrote: |
| mqjeff wrote: |
| bruce2359 wrote: |
| Look at the .ini file for the qmgr in question. What are maxchannel and maxactivechannel values? |
Assuming these are MQ clients, and not TCPIP clients or HTTPS clients or or or or |
We are connecting direct to the broker using MQTT. |
What input nodes are the broker flows using?
What is acting as the MQTT server? |
|
| Back to top |
|
|
| wkucard |
| Posted: Mon Jun 23, 2014 1:07 pm Post subject: |
|
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
Could you help me understand what the broker flows would have to do with it? Our client machine connects to the broker server directly using MQTT on port 1883. The connection issue we are having problems with would seem to be before the flow even comes into play, since we are just talking about establishing an SSH connection here.
However, I might be overlooking something, so please help me understand how that could be related?
I am not sure what you are asking about in your second question. We are not doing anything special beyond what is part of the normal message broker install and services. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jun 23, 2014 1:19 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Ok.
What program, application, or system is listening on port 1833 on that machine?
Broker v8.0.0.x on it's own does not come with built-in MQTT connectivity. Unless they snuck something into a fixpack I didn't notice.
So either you've added the open source MQTT connector, or you've configured the MQXR Telemetry daemon, or you've done something else even more confusing and complicated.
Either broker is involved somehow, or it's not. If you tell me that all flows are only using MQ Input nodes, then I know that broker is, essentially, *not* involved. And if you tell me that they're all using FileInput nodes, I'll wonder where MQTT comes into the picture at all...
In all cases, it's relatively likely that it's *not* the queue manager that is imposing whatever connection limits you are experiencing. I believe the Telemetry daemon uses a bindings connection, and so it would only run into issues with maxhands.
But if you're expecting one queue manager to do all the work for all of your broker flows AND act as the single MQTT endpoint for all of your MQTT client applications... you should have done a *lot* of capacity planning before this, and made the Broker queue manager very very beefy and on a box that has *lots* of free memory and disk space. |
|
| Back to top |
|
|
| wkucard |
| Posted: Mon Jun 23, 2014 2:07 pm Post subject: |
|
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
Thanks for the clarification. We all have our own areas of expertise and MQ and MB is not mine!
I don't work in this environment every day and had forgotten that we are using the MQ telemetry daemon to accept connections. This, as far as I know, is just out of the box configuration. We haven't done anything special here.
We are using almost exclusively MQ Input nodes, but again I'm unsure how that comes into play. We are not experiencing any flow issues, it seems to just be initial connectivity issues from the clients trying to connect to the Broker/MQ server, whereby the initial SSH connection is made but then ushered into a time_wait state. But only after we hit a certain threshold (which seems to be about 170 clients) do we see this issue at all.
Isn't maxhands only related to the number of handles per connection? So it seems it would be unrelated to the issue here.
I don't believe we have any capacity issues doing what you describe. We are moving our MB6 load to MB8 and the MB6 environment had the exact same topology, and if anything, less resources than what our new environment is utilizing. |
|
| Back to top |
|
|
| wkucard |
| Posted: Mon Jun 23, 2014 2:20 pm Post subject: |
|
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
I suppose maxhands could come into play if what we're experiencing has nothing to do with some threshold of clients connected, but we just so happened to hit some clients now that are going past the maxhands threshold.
I found this error - is this indicative of that problem? It would seem so.
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Mon June 23 2014 14:03:46 CDT |
| UTC Time :- 1403550226.508 |
| Operating System :- Linux |
| Process Id :- 19543 |
| Product Long Name :- IBM WebSphere MQ Telemetry |
| Vendor :- IBM Corporation |
| Probe Id :- XR071009 |
| UserID :- mqm |
| Java Version :- 2.4 |
| Thread :- name=ServerWorker0 priority=5 group=main ccl=sun.misc. |
| :- Launcher$AppClassLoader@24952495 |
| Exception:1 :- com.ibm.mq.MQXRService.MQException: AMQXR0004E: MQSeri |
| :- es verb=MQSUB(String) returned cc=2(int) MQCC_FAILED r |
| :- c=2017(int) MQRC_HANDLE_NOT_AVAILABLE. |
| Message:1 :- AMQXR0004E: MQSeries verb=MQSUB(String) returned cc=2( |
| :- int) MQCC_FAILED rc=2017(int) MQRC_HANDLE_NOT_AVAILABL |
| :- E. |
| ExceptionDepth :- 1 |
| FILE_INFO :- %R%:%C%:%I% |
| Source :- |
| :- com.ibm.mq.MQXRService.MQTTServerSessionV3: |
| :- clientIdentifier=Wel_Colony98H previousState=STARTED s |
| :- [Truncated - See below for further details] |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Tue Jun 24, 2014 5:23 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
What fix level of MQ 7.5.n.n are you at?
There is a fix related to this Probe ID in 7.5.0.2. |
|
| Back to top |
|
|
| wkucard |
| Posted: Tue Jun 24, 2014 5:31 am Post subject: |
|
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
We are on 7.5.0.3, so very up to date.
On a side note, GO CARDS!! Dig the avatar. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Tue Jun 24, 2014 5:36 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
I have not used MQTT, so I will back off and let others comment.
Go Cards! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 25, 2014 7:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Yeah, that looks like you've hit a limit of maximum handles available, which is (if I recall) something like 255 in the default.
Get your MQ Administrator to read and review, very carefully, the MQTT Telemetry Performance Report, and adjust the config of the queue manager and the telemetry daemon to accommodate.
Or convince your mgmt to purchase and install a MesageSight appliance and never worry about hitting connectivity limits, or having the Telemetry daemon fall over, or worse crash the queue manager. |
|
| Back to top |
|
|
| wkucard |
| Posted: Wed Jun 25, 2014 7:29 am Post subject: |
|
|
Novice
Joined: 17 Aug 2012
Posts: 11
|
We did analyze things and adjust this limit. We are no longer seeing issues.
However, our client code has not changed from version 6 to version 8, and we verified as recently as a few days ago that problem clients were not having this issue. We are going to contact IBM support to dig into this further.
Thanks for your help as it helped us realize the true issue. |
|
| Back to top |
|
|
|
|
