| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Log files filling up quickly |
« View previous topic :: View next topic » |
| Author |
Message
|
| MABeatty1978 |
 Posted: Thu Sep 11, 2014 5:45 am Post subject: Log files filling up quickly |
 |
|
Acolyte
Joined: 17 Jul 2014
Posts: 54
|
I'm looking for some advice as to how to track down an issue with a queue manager in which is log files are filling up and advancing quickly.
We have a network of several thousand qmgrs. Typically it takes between 10 and 30 days for a log file to fill up and a new one to be created. However in one of our qmgrs, it will start rolling logs over about every 4 hours. It seems to do this for a few days, then stops. This particular qmgr is essentially the same and handles the same volume of traffic as all of the others, so I'm at somewhat of a loss as to what is happening or how to determine what's happening.
Thanks for the help. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Sep 11, 2014 5:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You've identified the queue manager, presumably based on monitoring tools. That's great!
The next step is to identify the applications using the queue manager.
Presumably you mean the transaction logs, not the AMQERR logs?
It sounds like there's an application that is using large, long, units of work, rather than using smaller units of work.
It's also slightly possible that there's a channel with an unusual batch size. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Sep 11, 2014 6:37 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
I don't think the size or length of the Units of Work is going to make much difference. 1000 UOWs with 1 transaction each, or 1 UOW with 1000 transactions is gonna take pretty much the same amount of log space.
A UOW of work that takes 1 second to complete is going to take about the same amount of log space as a UOW that takes days to complete. OK, yes, if, IF, there is other work going on that very long UOW will span multiple logs, but its that OTHER work that stretchs the log, not the long UOW by itself.
I think you occasionally have a lot of volume that uses the logs. High volume of transactions that use the MQ logs is what drives the logs to expand. Either you have lots of big persistent message occasionaly, or lots and lots of little persistent messages occasionally, and/or lots of MQPUTs or MQGETS that use syncpoint and then stop.
You need to identify the app(s) that are doing this occasionally. An MQ trace would certainly show it. Or an MQ Application Activity trace set to even Low would capture it, and then you can switch to medium if you need persistence details.
http://www.ibm.com/developerworks/websphere/library/techarticles/1306_bushby/1306_bushby.html
As mqjeff suggested, a QM to QM channel going nuts trying to resend the same batch of messages over and over might be a culprit. The AMQERR01.LOG would show evidence of that.
Thosands of QMs? Wow.....None of them are candidates to change to MQ Client mode?...never mind, that is a whole 'nother conversation.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| MABeatty1978 |
| Posted: Thu Sep 11, 2014 6:39 am Post subject: |
|
|
Acolyte
Joined: 17 Jul 2014
Posts: 54
|
| That was helpful, thank you. I can see looking through my monitoring logs that during the time frame the log files were filling up, the cluster sender channels were struggling being in "BINDING" and "RETRYING" states. The cluster xmit queue is also showing over a 100 messages backed up, the oldest being several days old. Are these symptoms synonymous with your "application that is using large, long, units of work, rather than using smaller units of work." diagnosis? |
|
| Back to top |
|
|
| MABeatty1978 |
| Posted: Thu Sep 11, 2014 7:27 am Post subject: |
|
|
Acolyte
Joined: 17 Jul 2014
Posts: 54
|
| PeterPotkay wrote: |
Thosands of QMs? Wow.....None of them are candidates to change to MQ Client mode?...never mind, that is a whole 'nother conversation. |
There is 1 qmgr in each of our 3,000+ retail locations across the world. Each QM handles the point of sale messaging of the several client point of sale devices attached to them.
So no, none of them are candidates to change to MQ Client mode. No need for a whole 'nother conversation. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 11, 2014 7:44 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9399
Location: US: west coast, almost. Otherwise, enroute.
|
This (thousands of qmgrs) is a common config for large chain-store retailers.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Sep 11, 2014 9:13 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| MABeatty1978 wrote: |
| PeterPotkay wrote: |
Thosands of QMs? Wow.....None of them are candidates to change to MQ Client mode?...never mind, that is a whole 'nother conversation. |
There is 1 qmgr in each of our 3,000+ retail locations across the world. Each QM handles the point of sale messaging of the several client point of sale devices attached to them.
So no, none of them are candidates to change to MQ Client mode. No need for a whole 'nother conversation. |
I didn't say it had to be a long conversation! 
Thanks for the background info, makes sense.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| dware |
| Posted: Thu Sep 11, 2014 11:51 pm Post subject: |
|
|
Novice
Joined: 18 Nov 2013
Posts: 13
|
| MABeatty1978 wrote: |
| That was helpful, thank you. I can see looking through my monitoring logs that during the time frame the log files were filling up, the cluster sender channels were struggling being in "BINDING" and "RETRYING" states. The cluster xmit queue is also showing over a 100 messages backed up, the oldest being several days old. Are these symptoms synonymous with your "application that is using large, long, units of work, rather than using smaller units of work." diagnosis? |
It's possible that this could be related, so worth investigating. When messages are queued up on a cluster transmission queue waiting to be sent to another queue manager that's not currently available via a running channel, the queue manager will try to re-route suitable messages via an alternative channel (either to the same queue manager or another which hosts the same queues). It does this every time the channel retries its connection and fails. If there is no alternative route then this process can result in messages repeatedly cycling through the log.
So if your retry interval on the channel is very short, and the target queue managers are not available for a long period of time you could see increased log use.
There are a number of ways to reduce this:
- Increase the retry interval, possibly just the long timer, that's the one that kicks in after a number of failed short retries.
- Manually stop the channel if you know the target is not coming back for a long time.
- Bind the messages to a single route (e.g. MQOO_BIND_ON_OPEN), that excludes messages from being re-routed, although that obviously has more of an impact if you do have alternative targets for HA/WLB.
There's a bit more of a description of this 'reallocation' process towards the bottom of this page https://www-304.ibm.com/support/docview.wss?uid=swg21127527 |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
