| Author |
Message
|
| ajomillar |
 Posted: Thu Jul 14, 2005 9:47 am Post subject: Extreme queue depth - high volume messaging |
 |
|
Centurion
Joined: 22 Aug 2003
Posts: 121
Location: Milwaukee, WI
|
Once every so often, the business decides to use the integration layer to send everthing-but-the-kitchen-sink from a legacy front office system. This is done to populate a operational data store (Oracle) database. To get there, we employ several custom adapters that put and get messages from a queue manager. As I found out last weekend, high queue depths (422,000
+ messages on one queue; approx size = 9K/msg) on MQ v5.2 (csd 04) spells near disaster...we got busy semaphore errors in the MQ error logs and broken connections. In short, it interfered with existing message flows and impacted payroll processing...Ah, life in IT is beautiful with all its imperfections 
I'd like to build a separate queue manager to handle high queue depths in those cases where we have to do this again (fortunately, these are planned events). We are constrainted by the sequencing of messages (driven by the source system) and have no interest (or funding) to modify the adapters. Also, there are many other offices that use the same message flow to transmit their daily information. Any insights would be greatly appreciated...
My solution: create a new queue manager with all the same MQ objects and point the adapters to the new queue manager for the duration of this high volume message flow. We'll have this QM for special purposes only. Once this job is done, then we reconfig the adapters to point back to the "normal" production QM. This is, at least, the cheapest solution. |
|
| Back to top |
|
 |
| malammik |
| Posted: Thu Jul 14, 2005 9:54 am Post subject: |
|
|
Partisan
Joined: 27 Jan 2005
Posts: 397
Location: Philadelphia, PA
|
During the cutover, make sure all connections are gone from the old qm or at least the connections that are suppose to be gone. With this volume the last thing you want to happen is for some app to keep putting messages that are not going to be consumed any time soon. Also, from experience, it is a wise idea to let the consuming applications finish up with all the messages before switching them over.
HTH.
_________________
Mikhail Malamud
http://www.netflexity.com
http://groups.google.com/group/qflex |
|
| Back to top |
|
|
| wschutz |
| Posted: Thu Jul 14, 2005 9:55 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
If at all possible, use a current" version of MQ (aka V5.3) with the latest CSD.
_________________
-wayne |
|
| Back to top |
|
|
| ajomillar |
| Posted: Thu Jul 14, 2005 9:57 am Post subject: |
|
|
Centurion
Joined: 22 Aug 2003
Posts: 121
Location: Milwaukee, WI
|
| certainly the cutover and cutback is critical...not a problem. Has anyone experienced degraded MQ performance or broken connections on high queue depths (400,000+ msgs)? I've also proposed momentarily stopping the "put" adapter to slow the accumulation onto the input queue and allow the consuming app to catch up. |
|
| Back to top |
|
|
| malammik |
| Posted: Thu Jul 14, 2005 10:21 am Post subject: |
|
|
Partisan
Joined: 27 Jan 2005
Posts: 397
Location: Philadelphia, PA
|
My previous client was running about 200K(75% on temp dyn queues) messages per hour on 4way p660 AIX and about 100K on Z/OS. Aside from occasional disk failures, mq worked smoothly running 5.3. CPU hovered at about 50-70%. Mostly listener. Linear logs.
_________________
Mikhail Malamud
http://www.netflexity.com
http://groups.google.com/group/qflex |
|
| Back to top |
|
|
| ajomillar |
| Posted: Thu Jul 14, 2005 10:42 am Post subject: |
|
|
Centurion
Joined: 22 Aug 2003
Posts: 121
Location: Milwaukee, WI
|
| The existing queue manager is running on Windows 2000 server. Since we are standardizing on AIX, we have production and test QMs for Integrator Broker, Workflow, and more new implementations on that platform. I think moving onto AIX (running MQ v5.3) will help with performance. |
|
| Back to top |
|
|
| wschutz |
| Posted: Thu Jul 14, 2005 10:58 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
You might want to look at supportpac MP6I WebSphere MQ for AIX V53 - Performance Evaluations. It shows a dramatic improvement for persistent messaging between v5.2 and v5.3.
_________________
-wayne |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 14, 2005 4:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
We run 5.3 CSD06(solaris sender) and put up to 400,000 msgs onto the queue (5.3 CSD08 AIX) in about 4 hours (due to sending application).
As sequence is important there is an MDB with only 2 instances picking it up (IBM WAS 5.0x)
It drains nicely over 18 to 24 hours. The messages undergo some intense DB processing (so timing here is due to receiving application)...
Enjoy  |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jul 14, 2005 4:17 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Upgrade yourself to the current version of MQ, make sure you got the diskspace, enable the large files option at the O/S level (if UNIX - not needed on Windows), and you should easily be able to queue that many messages.
9K * 400,000 = about 4 GIG of data.
As a test, I put 200,000 messages that were each 100K in length on a Windows 2000 box, MQ 5.3.0.8 with no problem. The q file was 20GB.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 14, 2005 4:19 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| And make sure you have the log space as well if the messages are persistent. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jul 14, 2005 4:40 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| fjb_saper wrote: |
| And make sure you have the log space as well if the messages are persistent. |
If the messages are Persistent AND the logs are Linear, yes, otherwise this is not an issue, unless for some crazy reason they are trying to put all these persistent message in a single unit of work, in which case your circular logs will have to be big enough. In my test, that was 20GB of persistent messages, even thought the circular logs were only about 4 GB (as big as 5.3 will allow)
(See other posts for discusions on whether to use Linear or Circular logging )
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
