| Author |
Message
|
| rahuldhanpal |
 Posted: Thu Aug 05, 2010 10:14 am Post subject: Broker taking longer to respond |
 |
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
hi Guys,
we have a strange problem here... whenever a particular message flow is being stopped it is not being stopped and the broker does not respond.
we have had a similar issues couple of times but no clue why is this happening .. i am suspecting that there may be some problems with the msg flow ( like so many internal looping techniques used in the flow which might take longer to respond, this was before )
due to this mqsilist on the EG also does not respond and report properties on the EG does not respond too ..
i tried looking into some other threads and documentation and followed the same but no results...
Aug 5 13:54:39 MTDAPPLX01 WebSphere Broker v6107[26621]: (BRKDEV.StaffServices)[25]BIP2155I: About to 'stop ' the Message flow - 'LCGMS_Mediation_V_1'. : BRKDEV.d0ac9a3e-2301-0000-0080-e1427dfd762c: /build/S610_P/src/DataFlowEngine/ImbDataFlowDirector.cpp: 2044: ImbDataFlowDirector::stopResource: :
Aug 5 14:01:45 MTDAPPLX01 WebSphere Broker v6107[25725]: (BRKDEV)[1]BIP2066E: Broker 'BRKDEV' (UUID '59f64467-2001-0000-0080-baab0166e2e2') was unable to retrieve an internal configuration response message for execution group 'StaffServices' within the 420 second configuration timeout. : BRKDEV.agent: /build/S610_P/src/AdminAgent/ImbAdminAgent.cpp: 2782: ImbAdminAgent::receiveEGResponse: : .
Any ideas highly appreciated...
Thanks
RD |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Aug 05, 2010 10:20 am Post subject: Re: Broker taking longer to respond |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rahuldhanpal wrote: |
| i am suspecting that there may be some problems with the msg flow ( like so many internal looping techniques used in the flow which might take longer to respond, this was before ) |
I suspect you might be right; the broker won't stop until the flows have completed processing.
Does the broker stop eventually?
Do the flows use any database calls or stored procs that could be taking a lot of time to return?
Do the flows use any Java which might have sleep() methods in them?
Do the flows have any MQGet nodes with long (or unlimited) wait periods that are not getting messages?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Thu Aug 05, 2010 10:34 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
thanks for the response...
The Broker does not stop but it does not respond to any of the operations(start,stop, deploy of flows or even mqsilist etc)
And there are no DB calls or Stored procs
And seems like there is java but no sleep method in it
and no mqget nodes...
Actually I am on the broker administration side so i dont have much info on the flows and this is in our dev environment where multiple persons try to execute their operations ( as they have 'F' access on the Broker)
i am trying to analyze which flows(as there are almost 25 flows in this EG and 4 other EG's on the same broker) are trying to get this thing goin crazy ..
thinkin to increase the JVM Heap size on this EG assuming it might solve the problem.. ( Any pointers here ) |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 05, 2010 10:44 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rahuldhanpal wrote: |
| thinkin to increase the JVM Heap size on this EG assuming it might solve the problem.. ( Any pointers here ) |
That's a big assumption - if the broker's run out of memory I'd expect it to crash rather than hang.
Admiting some bias here I'd say the Java is a weak link here. What are these nodes doing? Do they call any external libraries? Or make DB calls themselves rather than via broker?
If the broker doesn't stop at all you're looking for a condition that makes it hang IMHO. To me the first suspect is an external action that never returns (database, external proc, missing response message).
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Thu Aug 05, 2010 10:51 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
Yes i see messages being held up in the SYSTEM.BROKER.CONFIG.REPLY queue of the cmgr.. and sometimes on most of the SYSTEM.BROKER queues.. so trying to figure out what is holding these messages rather than getting exchanged to process the requests....
will update the thread with the flow internals soon(trying to get hold of my developer).. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 05, 2010 10:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rahuldhanpal wrote: |
| so trying to figure out what is holding these messages rather than getting exchanged to process the requests.... |
Broker itself. It won't process changes while a message is in flight. The question is why broker can't get a gap in message flow processing to handle these messages (hence my questions about flows).
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Thu Aug 05, 2010 11:04 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
| with the continous messages logged in the log after killing the eg process(did not find any other option) i see so many starts and stops for a particular msg flow operations being executed from the toolkit (think by the developers,seems like they want everything real quick, this might also be one of the reasons for all this hanging) as they dont see immediate response for their performed operation and redo it immediately to see the result .. i m helpless in this case coz they never listen to admins( which is in most of the cases i guess).. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Aug 05, 2010 11:13 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| rahuldhanpal wrote: |
| Yes i see messages being held up in the SYSTEM.BROKER.CONFIG.REPLY queue of the cmgr.. and sometimes on most of the SYSTEM.BROKER queues.. so trying to figure out what is holding these messages rather than getting exchanged to process the requests.... |
This would point to a lot of deployments. A deployment will stop/restart the flow.
Have also a quick look at the depth of SYSTEM.BROKER.AGG* queues. Usually a depth there points to a temporary "overload" of the broker and it builds way faster than it gets worked off. (depends as well on the level of your broker...).
Hope it helps 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Thu Aug 05, 2010 11:21 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
Currently the queues SYSTEM.BROKER.AGG* dont have any messages
i will continue to monitor the broker and broker qmgr for similar kind of issues and update the thread once i find something ( As i know i will see the same problem again)..
thanks
RD |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 05, 2010 12:36 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
There would only be messages on S.B.AGG queues if the Aggregation nodes were used.
Just like there should only be messages on S.B.EDA queues if Collector is used.
Developers running amuck with deployments and start/stop commands can be reigned in fairly tightly with some basic administration security. It is very usual in anything other than development and SIT environments that developers have no access to do anything other than view the environment.
it is fairly common in development environments for developers to do all of their testing on their own desktop against a local broker, or to have a specific and dedicated EG of their own on the development environment broker, and no control or access outside of that.
It's also fairly common in those environments for the broker admins to deny responsibility for the performance of those development systems. |
|
| Back to top |
|
|
| bloomy |
| Posted: Thu Aug 05, 2010 12:46 pm Post subject: |
|
|
Acolyte
Joined: 11 Feb 2009
Posts: 61
Location: Bloomington, IL USA
|
I have had similar issues all the time and the thing I do to reciver such situations is to reload the execution group and in many situations the relaod will also not respond then i kill the DFE process.
The underlying cause i assume is that some inflight transactions which are yet to be processed when you issue the stop command will never finish to process and they go in loop.
The broker does not know what to do except to wait until the messages get process and they will never, in such situations when i recycle the broker I have seen that the old process stays like that and the broker spawns a new DFE process for the same EG which is kind of odd. |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Thu Aug 05, 2010 12:52 pm Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
i completely agree with you jeff.. the things i have mentioned in my earlier posts are just for an FYI and they are part of the package so we have to live with it. i just made a mention about it and nothing personal.
we are also following the general security measures to restrict and grant access accordingly.
i strongly say yes for the last statement coz we never deploy stuff in our dev, we come into scene whenever the developers have any issues with their deployments etc. (as this is the case in our scenario)
but we(admins) have control over all other environments.
Thanks
RD |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Thu Aug 05, 2010 12:54 pm Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
| bloomy wrote: |
I have had similar issues all the time and the thing I do to reciver such situations is to reload the execution group and in many situations the relaod will also not respond then i kill the DFE process.
The underlying cause i assume is that some inflight transactions which are yet to be processed when you issue the stop command will never finish to process and they go in loop.
The broker does not know what to do except to wait until the messages get process and they will never, in such situations when i recycle the broker I have seen that the old process stays like that and the broker spawns a new DFE process for the same EG which is kind of odd. |
bingo.. its the same case  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 05, 2010 4:36 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| rahuldhanpal wrote: |
i strongly say yes for the last statement coz we never deploy stuff in our dev, we come into scene whenever the developers have any issues with their deployments etc. (as this is the case in our scenario)
but we(admins) have control over all other environments. |
I am suggesting that you could perhaps be a bit more stringent in your "you broke it, you fix it" attitude towards these development environments, and a little more stringent in protecting one developer from the excesses of another...
I realize this is an idealistic approach, and that there are a lot of pressures internally for "assistance". And I would never officially suggest that you plead off because of an "important production issue".
But... you might want to stock up on  |
|
| Back to top |
|
|
| rahuldhanpal |
| Posted: Fri Aug 06, 2010 8:01 am Post subject: |
|
|
Voyager
Joined: 24 Jan 2009
Posts: 84
Location: Kenosha WI
|
Yes i agree with you jeff, but its a bit different in our case ..
maybe i will implement to restrict the access to developers to per EG rather than having Full access on the development broker which might reduce some overhead. |
|
| Back to top |
|
|
|
|
