| Author |
Message
|
| sukhi |
 Posted: Thu Jan 05, 2017 12:00 am Post subject: worloadmanagement issues in production |
 |
|
Novice
Joined: 22 Nov 2016
Posts: 18
|
Hi All,
i have deployed IIB application in production and increased additional instances of message flows to 10, still i am not getting response messages.
please guide me how to handle concurrency of multiple requests.
mobile banking and creditcard app user's are using these flows. |
|
| Back to top |
|
 |
| ruimadaleno |
| Posted: Thu Jan 05, 2017 1:55 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
we need more detail here ... what do you mean with:
" ...still i am not getting response messages..."
do you get timetou ? http 500 internal server error ? other ?
what steps have you followed to solve the problem ? have you done some investigation ? can you show us the results ? do you have some evidences in log files ? what versions are you using ?
(sorry .. no magic crystal ball here to help)
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| sukhi |
| Posted: Thu Jan 05, 2017 2:13 am Post subject: |
|
|
Novice
Joined: 22 Nov 2016
Posts: 18
|
request message send from mobilefirst to IIB , and checked user trace of EG , found that
response message were computed and send through soap reply node.
Coming to mobilefirst logs where it shows request timeout error, and some of places response messages were shown intermittently in the log.
what if we use wlmpolicy for workloadmanagement, i dont know the behaviour of number of notificationthreshold message per second.
what exactly it does, if we keep value as 300, will create 300 thread for each message flow or something different |
|
| Back to top |
|
|
| sukhi |
| Posted: Thu Jan 05, 2017 2:20 am Post subject: |
|
|
Novice
Joined: 22 Nov 2016
Posts: 18
|
iib version is 9.0.0.2 and now increased additional instances as 20.it works
but there are so many users , this issue will occur again.
please help to sort out this scenario |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 05, 2017 5:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sukhi wrote: |
| Coming to mobilefirst logs where it shows request timeout error, and some of places response messages were shown intermittently in the log. |
So if the response messages are in the EG trace, and the matching requests are timing out, what do the network people say?
| sukhi wrote: |
| what if we use wlmpolicy for workloadmanagement, i dont know the behaviour of number of notificationthreshold message per second. |
The behavior is as described in the documentation here; you get a notification when the threshold is breached. I'm unclear how you think this will help or why you think this is a workload problem.
| sukhi wrote: |
| what exactly it does, if we keep value as 300, will create 300 thread for each message flow or something different |
I'm unclear what it is in this context or what value is at 300 but a workload notification doesn't change thread count.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 05, 2017 5:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sukhi wrote: |
| iib version is 9.0.0.2 |
That's old
| sukhi wrote: |
| now increased additional instances as 20.it works |
That's good.
| sukhi wrote: |
| but there are so many users , this issue will occur again. |
Probably.
| sukhi wrote: |
| please help to sort out this scenario |
How? We can't possibly know anything about your volumes. You seem (through blind thrashing about) to have determined that you need more instances. Thus you need to determine what the correct amount of resource you need is to handle current & future user loads, then allocate that.
Alternatively you can just add as much CPU & memory as you can to the server, increase the additional instances to 255, then add more EGs with copies of this flow until the server starts to run short (say 80% of maximum) and then build another server.
All resource problems can be fixed by adding more resource. It's seldom the best or most cost effective way, but it always works.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jan 05, 2017 5:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's usually at least as effective to add an additional EG or additional broker than to add additional instances to a flow. Each instance of a flow is another thread in a process. At some point, you are going to start blocking threads to wait for each other.
If you are delayed getting responses from your flow, then you need to first look at the timing of the requests you are making from your flow. Until you've eliminated that, you can't make any reasonable conclusions or sizing about the real performance of your flow. If all of your instances are waiting for responses from downstream systems, then the more instances you add, the more likely you are to delay more responses out of your flow.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Thu Jan 05, 2017 6:01 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
Is there any web service exposed (soap/rest) and consumed by mobilefirst ? if you evoque the service from soapui (or other similar tool) how much time does the response takes ? is this value acceptable ? if not , what part of the message flow is slowing down responses ? database acess ? other services access ?
Have you done a stress test with soapui/loadmeter or similar tool ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| sukhi |
| Posted: Thu Jan 05, 2017 8:05 pm Post subject: |
|
|
Novice
Joined: 22 Nov 2016
Posts: 18
|
Thanks master's for meaningful answers....
below were the observation from concerned team,
network team: said that sometime acknowledgment are coming and sometimes not from IIB server.
DBA Team - there were database locking issue while select particular record from pl/sql package it was taking 5 to 6 seconds to fetch, hence added indexes as well.
please confirm for below scenario.
i have two message flows for each two applications and assigned additional instances 20 for each, both are running in same execution group.
calculating thread for each message flow basis of additional instances.
20+20+20+20 =80 threads.
if one request came for both applications then what would be the behaviour, is this creating 20 threads or one thread ?, and if it is 20 then when it would be released?
is there any way to release if it is not used through code |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 06, 2017 4:27 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Threads running instances are added as needed - based on how many incoming requests are pending for the current set of instances. "Pending" means "not being serviced by any of the current instances".
So if you have ten messages coming in during a shorter time period than it takes to process using one instance, then you will get more instances until all of the ten messages have been processed - this might not be 10, it might only be one more...
Instances hang around waiting to process more work for, what was it, a minute?
So again, you likely won't have all 20 hanging around, or even doing work all the time. It's based entirely on how fast messages are coming in and how many are waiting to be processed.
This has nothing to do with what the flow is doing - it has only to do with the rate of incoming messages compared to the rate of messages being processed in the same time interval (i.e. five incoming messages in one second and one message being processed in one second, or etc.)
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| sukhi |
| Posted: Fri Jan 06, 2017 5:31 am Post subject: |
|
|
Novice
Joined: 22 Nov 2016
Posts: 18
|
Thanks grand master for brief clarification  |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Jan 06, 2017 7:21 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
as far as i can understand from your post you are unable to find application bottleneck.
have you tried message flows statistics ? check iib http console , what does it tells you the message flows performance ? can you analyze message flow performance using IS03 support pack (not sure if this support pack is "compatible" with version 9?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| sukhi |
| Posted: Fri Jan 06, 2017 8:50 pm Post subject: |
|
|
Novice
Joined: 22 Nov 2016
Posts: 18
|
hi Rui,
Thanks for suggestion.
since this is production , then it will go with change request with approval for installation/access.
this will take time for below accesss |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Mon Jan 09, 2017 1:23 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
Do you need to release/change to analyze http flow statistics ? they can be enable from http console (if disabled) , but has you said, it may need a request for change ... but .. if it's hitting your production systems don't you have available any kind of "emergency change/release" ? how long does it take from your clients complaing about problems to change requests being executed ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
|
|
