| Author |
Message
|
| rcrippen |
 Posted: Mon May 16, 2005 7:35 am Post subject: Delay in execution groups starting |
 |
|
Apprentice
Joined: 01 Aug 2002
Posts: 45
Location: Rochester, NY
|
I'm putting a Veritas clustering solution in place on two Windows 2003 servers running WMQ V5.3.9,WBIMB V5.0.4 and DB2 V8.1.2.
When I start up the Broker service, it takes an long amount of time for all three of my Execution groups to start up in the Windows Task manager. Once they all do start up, it takes a little while (about 5 minutes) to see the execution group processes start utilizing memory and connect to their input queues.
Are there any parameters (WBIMB or DB2) that can be modified to have the execution groups to become active faster?
thanks,
Rob Crippen |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon May 16, 2005 7:59 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Is the database fully up and running before you start your Broker? Is the queue manager?
Do you have a lot of flows deployed?
You may not be able to fix this - but increasing the EG JVM heap and memory allocation may help.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| rcrippen |
| Posted: Mon May 16, 2005 8:05 am Post subject: |
|
|
Apprentice
Joined: 01 Aug 2002
Posts: 45
Location: Rochester, NY
|
Jeff,
When you say increase the EG JVM heap size, do you mean in the mqsichangeconfigmgr command? If so, how do you distinguish between EG for this command?
Rob |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 16, 2005 4:00 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Rob, for what its worth, we see the same thing on our Windows 2000 Brokers with Veritas. After a failover, it takes almost 5 minutes before the broker is 100% up. Kinda sucks, as the QM is up way sooner, and starts participating in the MQ round robining cluster algorithim, thus accepting messages before it is really ready to deal with them.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rcrippen |
| Posted: Tue May 17, 2005 12:51 pm Post subject: |
|
|
Apprentice
Joined: 01 Aug 2002
Posts: 45
Location: Rochester, NY
|
Peter,
Thanks for the response. I guess I must not be doing something basically wrong if you are seeing the same results. That's positive!
Are your transactions/messages request-reply or just asynchronous datagrams? I'm processing synchonous request/reply messages and my client is concerned about transactions not processing before the end-user times out during a failover situation.
Another question for you....how are you configuring your broker databases in the Veritas environment? Separate database server(s) or locally to each clustered server? I have defined a separate local database for each of my Active/Active Message Brokers and have the data stored on the SAN, the location associated with each Veritas service group.
Do you see any blatant issues with this scenario?
Any feedback would be appreciated.
Thanks,
Rob |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue May 17, 2005 1:42 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Quote: |
Are your transactions/messages request-reply or just asynchronous datagrams? I'm processing synchonous request/reply messages and my client is concerned about transactions not processing before the end-user times out during a failover situation.
|
Tell me about it.....same problem here. At least for me, with MQ clustered Brokers, only half of my time sensitive transactions time out during a fail over, as the gateway QM sprays messages to both live QMs, unaware that the Broker is taking its time coming up. The other half are processed by the Broker still running.
| Quote: |
Another question for you....how are you configuring your broker databases in the Veritas environment? Separate database server(s) or locally to each clustered server?
|
Same here. DBs are on the same server as the broker.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rcrippen |
| Posted: Fri May 20, 2005 7:28 am Post subject: |
|
|
Apprentice
Joined: 01 Aug 2002
Posts: 45
Location: Rochester, NY
|
Peter,
Did you ever open a PMR on the delay starting the Broker? If not, I'm considering doing that.
Rob |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri May 20, 2005 7:44 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
No, we never did. If you do, please let me know what they say. If they say something like not enough people complained, I'll open one to.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rcrippen |
| Posted: Fri May 20, 2005 10:46 am Post subject: |
|
|
Apprentice
Joined: 01 Aug 2002
Posts: 45
Location: Rochester, NY
|
Here's another symptom that I see when starting up the Broker's on the Windows boxes:
The DB2 process db2syscs.exe consumes a large quantity of memory when the bipbroker.exe initializes.
I've limited the amount of memory that each database uses in the DB2 Control Center's "configuration advisor" utility for each database.
Is there some way to reduce the amount of memory used by the Broker service when connecting to the broker DB. (Another DB2 parameter?)
Rob |
|
| Back to top |
|
|
| rcrippen |
| Posted: Thu May 26, 2005 11:57 am Post subject: |
|
|
Apprentice
Joined: 01 Aug 2002
Posts: 45
Location: Rochester, NY
|
Peter,
Here is the PMR number I opened with IBM for the delay in startup of the EG's on windows. I have asked Randy to go ahead and send it to development since you are interested in the answer also.
Rob
**********************************************************
Hi Rob,
Unfortunately I have not found any hints or tips regarding speeding up the broker start sequence. The general consensus seems to be that the broker is intended to be a long-running application, so they probably didn't worry too much about the startup time under the assumption that it shouldn't impact you that much. If you want, I can go ahead and send this to the development team to see if they have any suggestions, but I would set your expectations low.
Let me know what you want to do.
Randy Duncan
IBM Innovation Center - Dallas
ISV & Developer Relations Technical Support
To open new problems, call 1-800-426-9990
or e-mail to pwisvts@us.ibm.com
"Rob Crippen" <rcrippen@leveragingtechnology.com>
"Rob Crippen" <rcrippen@leveragingtechnology.com>
05/26/2005 12:23 PM
To
mqpwdts/Dallas/IBM@IBMUS
cc
Subject
RE: pmr 90968,756,000: wbimb startup parms
Randy,
All of my Broker servers have 4G of Ram, dedicated to running the Broker, WMQ and DB2. One thing I do see is that the db2syscs.exe process grabs a large quantity of memory when the bipservice.exe process initializes. Looking at the performance tab of the Task Manager I don’t see the memory close to the max during startup of the broker.
If you do find something, let me know.
Thanks,
Rob
--------------------------------------------------------------------------------
From:mqpwdts [mailto:mqpwdts@us.ibm.com]
Sent: Wednesday, May 25, 2005 5:07 PM
To: rcrippen@leveragingtechnology.com
Subject: pmr 90968,756,000: wbimb startup parms
Hi Rob,
Thank you for your question regarding whether there are any startup parameters in WBIMBwhich allow the execution groups to start up more quickly. I don't know of any off hand, but let me dig around a bit to see if I can find anything. I would recommend taking a look at the memory on your machines and what else is running on them. Obviously if there is a significant amount of swapping going on, that will slow things down considerably.
Warm regards,
Randy Duncan
IBMInnovationCenter- Dallas
ISV & Developer Relations Technical Support
To open new problems, call 1-800-426-9990
or e-mail to pwisvts@us.ibm.com |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu May 26, 2005 6:32 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Quote: |
The general consensus seems to be that the broker is intended to be a long-running application, so they probably didn't worry too much about the startup time under the assumption that it shouldn't impact you that much.
|
good lord.....  They obviously did not consider Brokers in an HA environment at all.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PGoodhart |
| Posted: Fri May 27, 2005 3:10 am Post subject: |
|
|
Master
Joined: 17 Jun 2004
Posts: 278
Location: Harrisburg PA
|
This isn't limited to the windows broker either. I see the same thing on AIX. The major factor seems to be the complexity/# of the flows in the execution group. The more complex the longer it takes to start. Oh and the alphabet. Start A then B then C ... If you have 30-40 execution groups it takes a while.
Another think is that flows/execution groups with remote databases take even longer.
_________________
Patrick Goodhart
MQ Admin/Web Developer/Consultant
WebSphere Application Server Admin |
|
| Back to top |
|
|
| Ian |
| Posted: Fri May 27, 2005 4:29 am Post subject: |
|
|
Disciple
Joined: 22 Nov 2002
Posts: 152
Location: London, UK
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri May 27, 2005 4:41 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| Ian wrote: |
| Running WMQI / WBIMB brokers under HA has been covered |
Ten minutes downtime waiting for the broker to start is not HA.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Ian |
| Posted: Fri May 27, 2005 5:47 am Post subject: |
|
|
Disciple
Joined: 22 Nov 2002
Posts: 152
Location: London, UK
|
Hi Jeff,
Prehaps you would like to re-read this thread again ?
Nowhere do I say or suggest that five to ten minutes of downtime waiting for the broker to start should be considered HA.
What I correctly said in reply to Peter's comment, was that running WMQI / WBIMB brokers under HA has been covered (and referenced the relevant SupportPacs).
To that effect it is reasonable to assume that a broker running in an HA environment should start quicker than what we are seeing described here.
So, the question should now be why is this configuration not delivering the required results.
Is there a problem with the broker ?
Is there a problem with the configuration ?
Is there a problem with one of the other components ?
Hopefully, Rob will get the answers to those questions via the PMR he has opened with IBM.
_________________
Regards, Ian |
|
| Back to top |
|
|
|
|
