ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » worloadmanagement issues in production

Post new topic  Reply to topic
 worloadmanagement issues in production « View previous topic :: View next topic » 
Author Message
sukhi
PostPosted: Thu Jan 05, 2017 12:00 am    Post subject: worloadmanagement issues in production Reply with quote

Novice

Joined: 22 Nov 2016
Posts: 18

Hi All,
i have deployed IIB application in production and increased additional instances of message flows to 10, still i am not getting response messages.
please guide me how to handle concurrency of multiple requests.
mobile banking and creditcard app user's are using these flows.
Back to top
View user's profile Send private message
ruimadaleno
PostPosted: Thu Jan 05, 2017 1:55 am    Post subject: Reply with quote

Master

Joined: 08 May 2014
Posts: 274

we need more detail here ... what do you mean with:

" ...still i am not getting response messages..."

do you get timetou ? http 500 internal server error ? other ?

what steps have you followed to solve the problem ? have you done some investigation ? can you show us the results ? do you have some evidences in log files ? what versions are you using ?

(sorry .. no magic crystal ball here to help)
_________________
Best regards

Rui Madaleno
Back to top
View user's profile Send private message
sukhi
PostPosted: Thu Jan 05, 2017 2:13 am    Post subject: Reply with quote

Novice

Joined: 22 Nov 2016
Posts: 18

request message send from mobilefirst to IIB , and checked user trace of EG , found that
response message were computed and send through soap reply node.

Coming to mobilefirst logs where it shows request timeout error, and some of places response messages were shown intermittently in the log.

what if we use wlmpolicy for workloadmanagement, i dont know the behaviour of number of notificationthreshold message per second.

what exactly it does, if we keep value as 300, will create 300 thread for each message flow or something different
Back to top
View user's profile Send private message
sukhi
PostPosted: Thu Jan 05, 2017 2:20 am    Post subject: Reply with quote

Novice

Joined: 22 Nov 2016
Posts: 18

iib version is 9.0.0.2 and now increased additional instances as 20.it works
but there are so many users , this issue will occur again.
please help to sort out this scenario
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 05, 2017 5:31 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

sukhi wrote:
Coming to mobilefirst logs where it shows request timeout error, and some of places response messages were shown intermittently in the log.


So if the response messages are in the EG trace, and the matching requests are timing out, what do the network people say?

sukhi wrote:
what if we use wlmpolicy for workloadmanagement, i dont know the behaviour of number of notificationthreshold message per second.


The behavior is as described in the documentation here; you get a notification when the threshold is breached. I'm unclear how you think this will help or why you think this is a workload problem.

sukhi wrote:
what exactly it does, if we keep value as 300, will create 300 thread for each message flow or something different


I'm unclear what it is in this context or what value is at 300 but a workload notification doesn't change thread count.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 05, 2017 5:36 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

sukhi wrote:
iib version is 9.0.0.2


That's old

sukhi wrote:
now increased additional instances as 20.it works


That's good.

sukhi wrote:
but there are so many users , this issue will occur again.


Probably.

sukhi wrote:
please help to sort out this scenario


How? We can't possibly know anything about your volumes. You seem (through blind thrashing about) to have determined that you need more instances. Thus you need to determine what the correct amount of resource you need is to handle current & future user loads, then allocate that.

Alternatively you can just add as much CPU & memory as you can to the server, increase the additional instances to 255, then add more EGs with copies of this flow until the server starts to run short (say 80% of maximum) and then build another server.

All resource problems can be fixed by adding more resource. It's seldom the best or most cost effective way, but it always works.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Jan 05, 2017 5:47 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

It's usually at least as effective to add an additional EG or additional broker than to add additional instances to a flow. Each instance of a flow is another thread in a process. At some point, you are going to start blocking threads to wait for each other.

If you are delayed getting responses from your flow, then you need to first look at the timing of the requests you are making from your flow. Until you've eliminated that, you can't make any reasonable conclusions or sizing about the real performance of your flow. If all of your instances are waiting for responses from downstream systems, then the more instances you add, the more likely you are to delay more responses out of your flow.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
ruimadaleno
PostPosted: Thu Jan 05, 2017 6:01 am    Post subject: Reply with quote

Master

Joined: 08 May 2014
Posts: 274

Is there any web service exposed (soap/rest) and consumed by mobilefirst ? if you evoque the service from soapui (or other similar tool) how much time does the response takes ? is this value acceptable ? if not , what part of the message flow is slowing down responses ? database acess ? other services access ?

Have you done a stress test with soapui/loadmeter or similar tool ?
_________________
Best regards

Rui Madaleno
Back to top
View user's profile Send private message
sukhi
PostPosted: Thu Jan 05, 2017 8:05 pm    Post subject: Reply with quote

Novice

Joined: 22 Nov 2016
Posts: 18

Thanks master's for meaningful answers....

below were the observation from concerned team,

network team: said that sometime acknowledgment are coming and sometimes not from IIB server.

DBA Team - there were database locking issue while select particular record from pl/sql package it was taking 5 to 6 seconds to fetch, hence added indexes as well.

please confirm for below scenario.

i have two message flows for each two applications and assigned additional instances 20 for each, both are running in same execution group.

calculating thread for each message flow basis of additional instances.
20+20+20+20 =80 threads.

if one request came for both applications then what would be the behaviour, is this creating 20 threads or one thread ?, and if it is 20 then when it would be released?

is there any way to release if it is not used through code
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri Jan 06, 2017 4:27 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Threads running instances are added as needed - based on how many incoming requests are pending for the current set of instances. "Pending" means "not being serviced by any of the current instances".

So if you have ten messages coming in during a shorter time period than it takes to process using one instance, then you will get more instances until all of the ten messages have been processed - this might not be 10, it might only be one more...

Instances hang around waiting to process more work for, what was it, a minute?

So again, you likely won't have all 20 hanging around, or even doing work all the time. It's based entirely on how fast messages are coming in and how many are waiting to be processed.

This has nothing to do with what the flow is doing - it has only to do with the rate of incoming messages compared to the rate of messages being processed in the same time interval (i.e. five incoming messages in one second and one message being processed in one second, or etc.)
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
sukhi
PostPosted: Fri Jan 06, 2017 5:31 am    Post subject: Reply with quote

Novice

Joined: 22 Nov 2016
Posts: 18

Thanks grand master for brief clarification
Back to top
View user's profile Send private message
ruimadaleno
PostPosted: Fri Jan 06, 2017 7:21 am    Post subject: Reply with quote

Master

Joined: 08 May 2014
Posts: 274

as far as i can understand from your post you are unable to find application bottleneck.

have you tried message flows statistics ? check iib http console , what does it tells you the message flows performance ? can you analyze message flow performance using IS03 support pack (not sure if this support pack is "compatible" with version 9?
_________________
Best regards

Rui Madaleno
Back to top
View user's profile Send private message
sukhi
PostPosted: Fri Jan 06, 2017 8:50 pm    Post subject: Reply with quote

Novice

Joined: 22 Nov 2016
Posts: 18

hi Rui,
Thanks for suggestion.
since this is production , then it will go with change request with approval for installation/access.
this will take time for below accesss
Back to top
View user's profile Send private message
ruimadaleno
PostPosted: Mon Jan 09, 2017 1:23 am    Post subject: Reply with quote

Master

Joined: 08 May 2014
Posts: 274

Do you need to release/change to analyze http flow statistics ? they can be enable from http console (if disabled) , but has you said, it may need a request for change ... but .. if it's hitting your production systems don't you have available any kind of "emergency change/release" ? how long does it take from your clients complaing about problems to change requests being executed ?
_________________
Best regards

Rui Madaleno
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » worloadmanagement issues in production
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.