| Author |
Message
|
| guest468 |
 Posted: Mon Jan 28, 2008 12:32 pm Post subject: Application performance while processing a large queue |
 |
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
Hi
We have a Linux server hosting a QMGR which is solely used by WAS 5. The qmgr has only 4 application local queues. Once in a while we get a flood of incoming messages onto some of these queues. The message size is say 50kb-200kb and at a depth of 30k or more the JVM stops processing messages(kinda hangs and needs a restart). The only MQ error I see is transaction log full which occurs at other times as well. this is WMQ5.3 and the number of log files and size are at the max.
Not sure what is causing this.
Thanks |
|
| Back to top |
|
 |
| bower5932 |
| Posted: Mon Jan 28, 2008 12:53 pm Post subject: Re: Application performance while processing a large queue |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| guest468 wrote: |
| this is WMQ5.3 |
I'd suggest moving forward to v6. There have been quite a few fixes in the JMS area. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Jan 28, 2008 12:55 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
As well as overall performance improvements.
Likely, however, this has absolutely nothing to do with MQ at all, and has to do with tuning or performance capacity of the WAS v5 server (which should be upgraded as well... ) or the WAS application.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| guest468 |
| Posted: Mon Jan 28, 2008 12:57 pm Post subject: |
|
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
Actually this is a long running stable application and would be long before we can upgrade to V6. Though we have plan in place.
Would appreciate if there is any temporary tuning to resolve the problem.
Thanks. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Jan 28, 2008 1:13 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
We don't know if there's any tuning you can do.
We don't know what tuning you've done, or what monitoring you have of the situation, to show where you might be out of resources or etc.
There probably IS some tuning you can do. But it's gonna take some dedicated analysis to figure out what.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| guest468 |
| Posted: Mon Jan 28, 2008 1:14 pm Post subject: |
|
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
Thanks Jeff, I will inform the WAS team.
Actually on a different subject I see a lot of transactions roll back errors (AMQ7469). I think it has nothing to do with JVM hang. But is there anyway this can be fixed without an upgrade? (log is set to max available).
 |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Jan 28, 2008 1:18 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Again...
We don't know.
It could be that you're using up log with MQ transactions from channels.
It could be that your App is making ridiculously large transactions.
It could be a combination of both.
You can tune MQ channel batch size.
You may or may not be able to do the same with your App.
Doing tuning on this may result in a decrease in throughput performance.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Jan 29, 2008 5:07 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| guest468 wrote: |
Actually on a different subject I see a lot of transactions roll back errors (AMQ7469). I think it has nothing to do with JVM hang. But is there anyway this can be fixed without an upgrade? (log is set to max available).
|
Please post relevant stanza from qm.ini and a few lines from ls -l /var/mqm/log/qmgrname/active
And as others have said, MQ 6.0 has some log capacity improvements, and on most platforms uses 64-bit addressing to increase capacity / performance in other ways. |
|
| Back to top |
|
|
| guest468 |
| Posted: Wed Jan 30, 2008 2:22 pm Post subject: |
|
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
mvic,
Here are the relevant contents from qm.ini
Log:
LogPrimaryFiles=61
LogSecondaryFiles=2
LogFilePages=16384
LogType=CIRCULAR
LogBufferPages=0
LogPath=/var/mqm/log/QMASP01/
LogWriteIntegrity=TripleWrite
And the log file listings are...
$ ls -ltr
total 4002332
-rw-rw---- 1 mqm mqm 67117056 Jan 29 02:53 S0000000.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 03:59 S0000001.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 05:09 S0000002.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 06:14 S0000003.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 07:20 S0000004.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 07:50 S0000005.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 08:08 S0000006.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 08:21 S0000007.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 09:01 S0000008.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 09:38 S0000009.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 10:11 S0000010.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 10:42 S0000011.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 11:12 S0000012.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 11:43 S0000013.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 12:10 S0000014.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 12:37 S0000015.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 13:04 S0000016.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 13:33 S0000017.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 14:02 S0000018.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 14:32 S0000019.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 14:58 S0000020.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 15:28 S0000021.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 15:59 S0000022.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 16:33 S0000023.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 17:02 S0000024.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 17:37 S0000025.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 18:12 S0000026.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 18:55 S0000027.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 19:37 S0000028.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 20:31 S0000029.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 21:22 S0000030.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 22:25 S0000031.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 29 23:35 S0000032.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 00:53 S0000033.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 02:07 S0000034.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 03:12 S0000035.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 04:28 S0000036.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 05:36 S0000037.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 06:48 S0000038.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 07:52 S0000039.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 08:36 S0000040.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 09:12 S0000041.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 09:48 S0000042.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 10:16 S0000043.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 10:45 S0000044.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 11:10 S0000045.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 11:36 S0000046.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 12:04 S0000047.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 12:34 S0000048.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 13:05 S0000049.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 13:34 S0000050.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 14:03 S0000051.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 14:22 S0000052.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 14:46 S0000053.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 15:03 S0000054.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 15:23 S0000055.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 15:49 S0000056.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 16:17 S0000057.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 16:44 S0000058.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 17:08 S0000059.LOG
-rw-rw---- 1 mqm mqm 67117056 Jan 30 17:20 S0000060.LOG |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 30, 2008 2:25 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
So if the WAS app hangs while it has a large transaction open against the logs, and then the channels try to move a large volume of persistent messages out of the queue manager...
You may get a full log message, and transactions rolled back either in the WAS app or in the qmgr from the channel transactions.
The solution is to either increase the total log space to accomodate your largest transactions, or reduce the size of your transactions.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jan 30, 2008 2:46 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| I reckon you have about 3 Gb of space in that log, before the qmgr starts to take action to roll back the oldest unfinished transactions. It seems highly likely that for the whole duration of time in which your 30,000 50-200KB messages were flowing into the queue manager, an application somewhere - it could be any application - had a transaction open without calling MQCMIT. However the only ill effect should be to that very app (its transaction gets rolled back preemptively, and it must retry it), not to any other app. I'm not sure why any app on the system would "hang" - perhaps this is a piece of behaviour in the app code rather than in the MQI itself. Hope this helps. |
|
| Back to top |
|
|
|
|
