| Author |
Message
|
| vinumon |
 Posted: Thu Nov 27, 2014 10:34 pm Post subject: Execution group port not reachable under loads... |
 |
|
Acolyte
Joined: 06 May 2014
Posts: 59
|
We are having Broker V8.0.0.1 with 4 EG, running on ports 7800,7801,7802,7803.
EG with port 7800 is having 13 message flows... Under performance test with message load high, we are getting port 7800 not reachable for few secs and later it comes up automatically... In a 3 hour test, there will be 1000 such occurances...
Is there any tuning parameter that I need to set at EG level....
We have kept additional instances for flows.....
Note: Similar issue we faced for Broker version V8.0.0.3... |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Nov 27, 2014 11:14 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Find the bottleneck, remove it! 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Nov 28, 2014 12:26 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I would avoid using the default port of 7800, too much chance of someone creating another broker and it trying to use that port as well.
As for volume load, best to use a decent protocol like MQ, which simply queues any traffic not yet processed, without rejecting it. Web services are inherently unreliable and also non transactionally safe.
But to increase throughput - you can look at increases the instances of each message flow. Try to avoid any waiting in the flow (i.e. any use of inferior synchronous protocols like web services).
Make sure that your maxthreads value is suitable for the number of flow instances that you have. Finally set the acceptCount value higher.
Each incoming request consumes a tomcat thread to process and maxThreads controls the size of the Tomcat thread pool. The accept count parameter controls how many connection requests are accepted and queued when all tomcat threads are in use.
However if you have an http load-balancer in front of the broker - it may use the rejected requests to decide how to distribute load. So queueing more requests inside the EG may reduce the effectiveness of the balancer.
All in all - it's an unreliable mess of a situation that would be avoided simply by using MQ for all forms of integration. But this is "the modern way" - we can't have a proven and robust solution when we can have the industry "standard" rubbish one instead....
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| vinumon |
| Posted: Fri Nov 28, 2014 12:35 am Post subject: |
|
|
Acolyte
Joined: 06 May 2014
Posts: 59
|
| What is meant by acceptCount and what value can I set for high volume brokers..., and currently maxthreads for EG is blank....do i need to set any values...? |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Nov 28, 2014 1:19 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
another check you should do: are you using broker wide http listener or embedded listener ?
currently maxthreads for EG is blank means it is running with default value which is 200 (please confirm this value in documentation).
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| vinumon |
| Posted: Fri Nov 28, 2014 1:57 am Post subject: |
|
|
Acolyte
Joined: 06 May 2014
Posts: 59
|
| We are using embedded listener..... There are almost 2000 users sending requests to the Broker.....so can I set maxThreads at EG level to 2000 and acceptCount 1500...? |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Nov 28, 2014 2:19 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
have you investigated message flows behaviour ? are you shure you have no contention/sync/delay issues in message flows ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| vinumon |
| Posted: Fri Nov 28, 2014 2:23 am Post subject: |
|
|
Acolyte
Joined: 06 May 2014
Posts: 59
|
There is no issue when 500 users are connecting to broker.....but this issue happens only when more than 1000 users connect.....
There are no issues on message flows... |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Nov 28, 2014 2:27 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
No!
Maxthreads are for the in-process requests - set to a number slightly greater than the number of flow instances allowed in that EG.
Do not set huge values, think in hundreds, not thousands, or you will run out of memory in the EG JVM.
The flow instance count will be the number of actual message flows in the EG, plus any additional instance value set. If you don't set it - you don't get it! Each user does not automatically create a new flow instance.
The default accept count is 100, I believe. When I increase values I usually try doubling them.
You set this on the http(s) connector parameter (same place as the you set the explicit port number for the EG).
The option to use the EG embedded listener needs to be set as well.
How are your incoming requests load-balanced?
READ this
http://www-01.ibm.com/support/docview.wss?uid=swg21504029
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Fri Nov 28, 2014 3:06 am; edited 1 time in total |
|
| Back to top |
|
|
| vinumon |
| Posted: Fri Nov 28, 2014 3:00 am Post subject: |
|
|
Acolyte
Joined: 06 May 2014
Posts: 59
|
No: of flow instance u mean is the additional instance set ?
I have set it to 250...as total...
Where can I find the option to use the EG embedded listener needs to be set ? Command to set ?
Current values for maxThreads and acceptcount is blank....which means its taking default values 200 and 100....so shall Iincrease to 400 and 200..? |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Nov 28, 2014 3:07 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Read the product documentation on the points raised. We are not your personal assistants.
Don't go over the top on additonal instances - you may run out of memory.
Make sure the requests are being processed quickly in the flow. If you have too long a delay (even if accepted and queued) - the requests may time out.
It's not a simple - here is the answer matter. You need to use your professional skills to understand the application and tune accordingly.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Fri Nov 28, 2014 3:10 am; edited 1 time in total |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Nov 28, 2014 3:08 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| vinumon wrote: |
No: of flow instance u mean is the additional instance set ?
I have set it to 250...as total...
Where can I find the option to use the EG embedded listener needs to be set ? Command to set ?
Current values for maxThreads and acceptcount is blank....which means its taking default values 200 and 100....so shall Iincrease to 400 and 200..? |
Before tweeking the numbers you should be focusing on the flows.
Is there a legitimate bottle neck? If yes what is it? How can you get rid of it?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vinumon |
| Posted: Fri Nov 28, 2014 5:08 am Post subject: |
|
|
Acolyte
Joined: 06 May 2014
Posts: 59
|
| most of the flows are connected to WAS server, and we used to get timeout from WAS , if the requests are getting more time to get processed...Is this one factor for EG port going to stopped state.... |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Nov 28, 2014 8:10 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
so your main problem is not value of maxthreads, and the "eg port stopped state" is a side effect.
If your message flow (which connects to WAS server somehow .. maybe another web service???) takes too long to process requests and you end with a bunch of new requests trying to get it's work done but waiting as the message flows instances are "busy" waiting for the WAS_server_Connect_Thing.
please note that this "message flows busy waiting for WAS_Server_connect_thing" are killing the ability to scale, which is what you are trying to validate with the load tests your are performing: Can your message flow scale to handle 2000 requests ? ... no it can't because from a certain point you get this "eg port stopped state". Incresing maxthreads can give you more "scalability" ? yes, a little, but you are always bounded to the "message flow busy waiting for WAS_Server_connect_thing"
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| vinumon |
| Posted: Sun Nov 30, 2014 5:33 am Post subject: |
|
|
Acolyte
Joined: 06 May 2014
Posts: 59
|
| @ruimadaleno, you mean to say there is time taken at Broker side to send request to WAS ? |
|
| Back to top |
|
|
|
|
