MQSeries.net :: View topic - High(100%) CPU usage with Triggered Application

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Java / JMS » High(100%) CPU usage with Triggered Application

Goto page 1, 2 Next

High(100%) CPU usage with Triggered Application

« View previous topic :: View next topic »

Author

Message

mdfaraz

Posted: Fri Dec 03, 2010 8:37 am Post subject: High(100%) CPU usage with Triggered Application

Novice

Joined: 31 Mar 2010
Posts: 19
Location: Houston, TX

Hi,
Few months ago, I had developed a .NET application that wud run on an app server and poll for messages on an MQ Queue on an remote MQ server. this app would run in an infinite loop looking for messages and when queue depth > 0, it would read and do some business specific processing on that message and once processed continue to the next message/ wait for more. There were 20-25 queues which wud use the same logic to connect to the QMGR/Queue and then do their deal. With MQ server being remote, I had to use a code like this to connect

InputQueueMgr = New MQQueueManager(QMgrName, "ServerConnectionChannel", "192.x.x.x(1415)")
InputQueue = InputQueueMgr.AccessQueue(QLName, MQC.MQOO_FAIL_IF_QUIESCING Or MQC.MQOO_INPUT_SHARED Or MQC.MQOO_BROWSE Or MQC.MQOO_INQUIRE)

The application generally ran fine except that due to network issues it would crash with QMGR not found and wud exit and someone in support wud have to restart it.
To overcome the manual restart, we decided to go triggered with apps and MQ server on the same box. I made the necessary changes to read queue manager name/queue and then connecting using the exact same code as above(specifying the server connection channel and hostip(part)) - I didn't feel I had to change it since its still a client connecting to a server. I have seen people using a different overload of the above function to connect to QMGR when app is triggered(QMGR is local to app).
InputQueueMgr = New MQQueueManager(QMgrName)

but when we installed this to production (with process definition for all triggered queues specifying the app to start). these apps, when triggered(using trigger type FIRST) were using 3-4% (each) of the CPU and adding those on 25 queues took the CPU usage to 100% bringing the server to its knees.
In the process manager it showed these exes taking up all of CPUs. I found it really alarming as the old code was same in all aspects and running 25 instances together wud not even take 5% of the CPU.

Can anyone shed light as to why such a small app cud be taking high CPU? On code review I did find that I missed the cleanup code of queue.close() and qmgr.disconnect() - But I am sure that's not the case.

Any thoughts for you gurus out there...
Help appreciated

Vitor

Posted: Fri Dec 03, 2010 9:17 am Post subject: Re: High(100%) CPU usage with Triggered Application

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mdfaraz wrote:

this app would run in an infinite loop looking for messages and when queue depth > 0, it would read and do some business specific processing on that message and once processed continue to the next message/ wait for more.

2 points: you should always have a method to cleanly close and disconnect your app; checking for messages by looking for queue depth is misleading and unnecessary. Because uncommitted messages count towards queue depth, the queue can be non-zero and still have no messages to read, It's unnecessary because an MQGet call will only return if there's a valid message to give your application.

While the get call is in progress, your application will sleep. Rather than spin round and round in an infinate loop polling the queue and chewing CPU.

mdfaraz wrote:

InputQueue = InputQueueMgr.AccessQueue(QLName, MQC.MQOO_FAIL_IF_QUIESCING Or MQC.MQOO_INPUT_SHARED Or MQC.MQOO_BROWSE Or MQC.MQOO_INQUIRE)

At least you had fail if quiescing on the open and (I hope) on the connect & get in your infinite looped application. This will make it unnecessary for the WMQ admin on your site to kill you....

mdfaraz wrote:

I didn't feel I had to change it since its still a client connecting to a server. I have seen people using a different overload of the above function to connect to QMGR when app is triggered(QMGR is local to app).
InputQueueMgr = New MQQueueManager(QMgrName)

True. A client can still use the network to connect to a queue manager hosted on the same machine. The other overload is for a binding connection only available if the queue manager is hosted locally. You can look it up.

mdfaraz wrote:

Can anyone shed light as to why such a small app cud be taking high CPU? On code review I did find that I missed the cleanup code of queue.close() and qmgr.disconnect() - But I am sure that's not the case.

Your application design chews CPU as I indicated above. Depending on how you're obtaining queue depth (by asking the queue manager or examining the queue) that could not be helping either.

I would suspect CPU usage is lower when the app is remote because the network is introducing some delays and keeping the CPU your application is using down. Or the app server was doing something clever.

As a side point, any application using a client connection shouldn't crash and need to be manually restarted just because the connection is lost. You should have put retry logic into the application (possibly with a CCDT) or used the v7 auto-reconnect facility.
_________________
Honesty is the best policy.
Insanity is the best defence.

mdfaraz

Posted: Fri Dec 03, 2010 1:48 pm Post subject: Polling Queue depth continously not chugging CPU much

Novice

Joined: 31 Mar 2010
Posts: 19
Location: Houston, TX

@Vitor: Thanks for your reply. However I see some different behaviour. I ran the polling app(which used to run on a different app server) on the MQ server. And I saw the CPU use going upto 30-35%. amqrmppa.exe *32 is taking about 6% of CPU(16 core 2.2 GHz each, 32GB RAM) which I believe is a lot of comsumption.
When I ran this very same app on the app server, reading from the CPU. The CPU on both these boxes would not even go to 5-6% each.(and the app server has other apps on it as well).
This polling app looked for queuedepth> 0 infinitely and when true, went about processing messages(Browse, process, delete)
The triggered app, when started, looks at queuedepth(which is ofcourse > 0),saves the numeric value in an integer(say n), and loops n times, does a READ(now under syncpoint-wonder if it matters but this is what was changed for a UOW), PROCESS, and then COMMIT. After looping these n times exits without looking at queue depth again
And this app is taking 4% * (25, 1 for each queue) = 100% of the CPU.

So where is the problem in InputQueue.depth is called just once and all others are .GET()??????

Vitor

Posted: Fri Dec 03, 2010 3:32 pm Post subject: Re: Polling Queue depth continously not chugging CPU much

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mdfaraz wrote:

amqrmppa.exe *32 is taking about 6% of CPU(16 core 2.2 GHz each, 32GB RAM) which I believe is a lot of comsumption.

That's what you get for not closing & disconnecting. I suspect also you're starting a new connection and/or handle for each read to get that level of usage. Again, it's a design issue.

mdfaraz wrote:

When I ran this very same app on the app server, reading from the CPU. The CPU on both these boxes would not even go to 5-6% each.(and the app server has other apps on it as well).

Does the app server have connection pooling? That would explain this.

mdfaraz wrote:

This polling app looked for queuedepth> 0 infinitely and when true, went about processing messages(Browse, process, delete)

This is a poor design.

mdfaraz wrote:

The triggered app, when started, looks at queuedepth(which is ofcourse > 0),saves the numeric value in an integer(say n), and loops n times

This is a poor design. Queue depth could easily change during the lifetime of this application instance.

mdfaraz wrote:

, does a READ(now under syncpoint-wonder if it matters but this is what was changed for a UOW),

It does matter. This is what you should be doing rather than a browse & delete.

mdfaraz wrote:

After looping these n times exits without looking at queue depth again

So what happens if more messages have turned up while the program was running? Or one of the messages on the queue was rolled back?

You need to seriously think about this code.

mdfaraz wrote:

So where is the problem in InputQueue.depth is called just once and all others are .GET()??????

You're thrashing the queue manager. Probably without the benefit of connection pooling. Redesign your code to a more best-practice pattern and then see if you still have a problem. Pay particular attention to how you're dealing with connections, and ensure you reuse them. Connection to a queue manager is a CPU expensive operation. You'll also get savings by not enquiring on queue depth continuously.
_________________
Honesty is the best policy.
Insanity is the best defence.

mdfaraz

Posted: Sat Dec 04, 2010 2:46 pm Post subject:

Novice

Joined: 31 Mar 2010
Posts: 19
Location: Houston, TX

Hi Vitor,
Thanks again for your leads. Few comments..

mdfaraz wrote:
amqrmppa.exe *32 is taking about 6% of CPU(16 core 2.2 GHz each, 32GB RAM) which I believe is a lot of comsumption.

Vitor wrote:
That's what you get for not closing & disconnecting. I suspect also you're starting a new connection and/or handle for each read to get that level of usage. Again, it's a design issue

I changed the code to close/disconnect all MQ objects when exiting, still I see the same behaviour.

Running the polling app which enquires QueueDepth continously, on the MQ box also does NOT increase CPU beyond 30%.

I am working on getting rid of this queue depth enquiry for good and using your suggestion to do a get and handle 2033. Lets see if it works.!!

Vitor

Posted: Sat Dec 04, 2010 2:53 pm Post subject:

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mdfaraz wrote:

I changed the code to close/disconnect all MQ objects when exiting, still I see the same behaviour.

Interesting. One question I should have asked is what version of WMQ are you using? Are you up to date with maintenance? You shouldn't be seeing that level of channel activity.

mdfaraz wrote:

I am working on getting rid of this queue depth enquiry for good and using your suggestion to do a get and handle 2033. Lets see if it works.!!

It does, and better than checking queue depth. There has been much discussion in here of why this is, as many developers find it counter intuitive.
_________________
Honesty is the best policy.
Insanity is the best defence.

fjb_saper

Posted: Sat Dec 04, 2010 11:19 pm Post subject:

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

mdfaraz wrote:

I am working on getting rid of this queue depth enquiry for good and using your suggestion to do a get and handle 2033. Lets see if it works.!!

Don't just use a get and handle the 2033 reason code. It will be far less CPU intensive to do a get with wait (interval) and handle the return and reason codes and handle any termination signal to the program.

Have fun

_________________
MQ & Broker admin

zpat

Posted: Sat Dec 04, 2010 11:29 pm Post subject:

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

Incidentally your original design was a common misconception with MQ.

Applications should not look at queue depth, nor "poll" at intervals.

They should issue a MQGET with MQGMO_WAIT and allow the queue manager to return control when a message arrives. With MQ 7 there is also a call back option that can be used.

That way you do not continually bother the queue manager with MQGET requests on an empty queue, and you do not have the latency of a wait interval between the MQGET attempts.

When a message arrives the application can process it immediately with no latency and should continue to get from the queue until it is empty, or simply allow the MQGET to resume waiting for messages using MQGMO_WAIT.

mdfaraz

Posted: Sun Dec 05, 2010 8:35 pm Post subject: Performance Problems still

Novice

Joined: 31 Mar 2010
Posts: 19
Location: Houston, TX

Hi Vitor and others,
I am using MQ version 7 and assume most updated with service packs and all the good stuff.
Thanks a lot for your words of wisdom. I got this issue the moment I installed into production(We had not done any performance testing in Dev/QA due to other reasons, but not in my wildest dream did I imagine that I wud max out on CPU of a 16 core/32GB RAM server). So, I was working on a war footing this weekend. I took your advice and the design of my app now looks like this
1. Connect to QMGR, Open queue as using code I mentioned before
2. In a while loop, do a Get on the queue with MQG_Wait and MQGMO_SYNCPOINT.
3. Process this message(Few programs take this message and put to two different queues)
4. Any error while doing 2, causes the program to exit out of while
5. Commit on QMGR
6. Clean up code - Disconnect QMGRs, close Qs, etc

On the MQ server, there are 40+ queues, All of them are triggered (so there are 40+ trigger monitors run by 40+ services(pointing to triggered apps by 1 process definition for each. All apps use a single Server connection channel to connect.
In a live environment, messages come on all these queues keep triggering their respective app which for the most part follow the above algorithm(steps 1-6)
The message traffic is 2-3 million messages on half of the queues and 6-7 million on the other half of 40 queues.

Question 1: Is this traffic too much to handler for MQ?
Question 2: Will following these best practices give me better performance(I can do with 20-30% of CPU but 70+ is unacceptable)?

As I said earlier I am desperately down on CPU with my earlier app. the changes I made this weekend have to go in but I have no environment to test!!!

fjb_saper

Posted: Sun Dec 05, 2010 9:45 pm Post subject: Re: Performance Problems still

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

mdfaraz wrote:

As you have no environment to test:

Make sure you check the message flag saying how many times it was delivered and move to backout queue when appropriate (bothresh reached) and commit. (Poison msg handling).
Make sure you roll back the message in case of problems
Make sure you commit every message on the happy path.
Don't forget the FAIL_IF_QUIESCING option
Make sure you have a user condition to exit the while loop if needed

This way if your traffic is too high for one instance of the app you can scale the number of instances with little to no problems.

Have fun

_________________
MQ & Broker admin

Vitor

Posted: Mon Dec 06, 2010 5:20 am Post subject:

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

I'm assuming you do step 2 until the wait comes back with a 2033, thereby processing all available messages. The trick is not to do a close and disconnect until the queue is empty or the application is signalled to stop (echoing the 5th comment of my most worthy associate).

Indeed, each of his points is good advice.

You also want to consider (as you're using v7) leveraging the auto-reconnect faciltities.

Other issues:

Don't have all the applications on the same svrconn channel. If there's a problem and you need to do any work on the channel to fix a given application you'll impact all the others.

Running one trigger monitor for each application is pointless and a waste of CPU. If you think about what the trigger mechanism does, you'll realise you only need one trigger monitor for any number of applications.

mdfaraz wrote:

I have no environment to test!!!

Ah, the joy of making changes directly into production. You might want to speak to your management about this.
_________________
Honesty is the best policy.
Insanity is the best defence.

mdfaraz

Posted: Mon Dec 06, 2010 7:22 am Post subject:

Novice

Joined: 31 Mar 2010
Posts: 19
Location: Houston, TX

Thanks, Vitor and fjp_saper for your comments

1. I open the connection to the Queue Manager once and also open the Queue only once per program and do subsequent Gets. I close the QMGR/Queue only when done.
2. I have 41 queues and have to have 41 trigger monitors. I have one application per queue and not one per application.
3. Do u think running all apps on single server connection channel have performance considerations?
4. Do u think having 41 trigger monitors cud contribute to high CPU usage? I am thinking on these lines as my polling apps, when run local to MQ server(and enquire queue depth all the time) taking about 30% of the CPU but running 41 triggered apps takes CPU over the roof whene all queues have messages(all apps running) but CPU goes down after messages are processed.

yeah, I wont be making changes to production without approval from my management of course

The most amazing thing to me is that the polling apps looking for queue depth (21 instances - 1 for each queue) are taking 30% CPU and the triggered app doing roughly the same taking more than 100% !!

Vitor

Posted: Mon Dec 06, 2010 7:33 am Post subject:

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mdfaraz wrote:

4. Do u think having 41 trigger monitors cud contribute to high CPU usage?

Well you're running 40 unnecessary applications, plus all the WMQ set up you've put in with them (init queues and so forth). So it's wasting CPU at best.

mdfaraz wrote:

yeah, I wont be making changes to production without approval from my management of course

I was thinking more in terms of asking them to pay for a test environment for you, rather than just ensuring there's more people taking responsibility when the change brings production down (by increasing rather than reducing CPU usage due to an unforseen factor).

mdfaraz wrote:

Well the polling apps are only taking that much because the design is inefficient. The triggered apps have all this infrastructure you've inexplicably decided you need. You've also not demonstrated anywhere in this thread I can see where this additional CPU is being consumed: applications, queue manager, channels, etc. You just seem to be saying, "I run this set up on one server and it uses 30% of the CPU. I run a similar set up on a completely different server and it peaks at 100%. Obviously it's the WMQ".

Identifying exactly what the problem is will go a long way to fixing it.
_________________
Honesty is the best policy.
Insanity is the best defence.

Mr Butcher

Posted: Mon Dec 06, 2010 7:38 am Post subject:

Padawan

Joined: 23 May 2005
Posts: 1716

imho 1 trigger monitor is enough

dont query for q depth. you get triggered, loop till queue is empty, then end. (as already written here before).

some mix it (triggered and polling). do a get wait after being triggered, e.g. with a waittime of 5 seconds. if additional messages arrive a bit later after the one(s) that cause the trigger you still get them in that run without a new trigger / application start. but be careful, if a message arrives every second your application will never end or be active for a long period of time.

Quote:

The most amazing thing to me is that the polling apps looking for queue depth (21 instances - 1 for each queue) are taking 30% CPU and the triggered app doing roughly the same taking more than 100% !

Because your applications are not written trigger-firendly. they get started too often, and perform unnessesary actions like q depth query every time, ending queue processing leaving new arrived messages are on the queue, get triggered again, query q depth, .... and so on.
_________________
Regards, Butcher

mdfaraz

Posted: Tue Dec 07, 2010 7:10 am Post subject: Miracle!

Novice

Joined: 31 Mar 2010
Posts: 19
Location: Houston, TX

Hi Vitor,fjb_saper and Mr. Butcher,
I do not have words to thank you guys enough,
I put in all the guidelines you all stated in these posts(and moved the code directly to production(my management trust me so much )!!

and it works like a charm. The CPU usage is now in the normal 8-10% range with triggered apps running on all queues with full traffic and I just cudn't believe how following best practices affected the performance by 2000%!
We are still monitoring for a couple of days before we hand off to system support and it's really amazing as it looked pretty bleak on Friday!!

One thing about triggering that I couldn't get hold of.I had the functionality to stop processing. Its just a flag in a table that the program looks while reading messages from queue(for efficiency, every 1000th message) and if its set to 0, the app exits out. Right now I have the same logic, but the moment the app exits, the trigger monitor is starting it back.
I thought with triggerinterval property set to a value(300 seconds in my case), MQ wont trigger this app if a message arrives within 300 seconds of the last time the app was triggered. http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzal.doc/fg13910_.htm
Whats the way to make it work if I dont want the app to get retriggered until certain interval?
Thanks again for all your time and wise words

Display posts from previous:

Goto page 1, 2 Next

Page 1 of 2

MQSeries.net Forum Index » IBM MQ Java / JMS » High(100%) CPU usage with Triggered Application

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP