| Author |
Message
|
| Decky |
 Posted: Wed Jun 13, 2007 2:20 am Post subject: WMQ logs filling up when running message flow |
 |
|
Novice
Joined: 16 May 2005
Posts: 16
Location: London UK
|
Was wondering if anyone could help. I think this is more an MQ question than WBI but see what you think.
I have a message flow that takes an 8-10MB single message (all queue, channel, xmit, qm defs are all tweaked for this). The message consists of a load of data records which are of a fixed width all together in one string of bytes. I basically parse these with a message set and send each data record out one at a time so we we end up with 1 message propagating 1000's of new ones. Co-ordinated transaction is set to 'no' so all messages are sent immediately and don't build up on the xmit queue. An in house adapter receives these messages and builds a file at the other end. The file isn't named until the last message is recieved (I set a flag in the message flow) and so by this theory I know we're getting all the messages at the other end.
The problem is that the file is created 3 times. Now the backout threshold on the input queue is 3 which would explain that, but why is it being rolled back? There are no wbi exceptions. So after mulling this over for a while I checked the mq logs and lo and behold MQ is rolling back the message and complaining that the logs are getting full. In the environent we're using our log size is about 256MB (circular) - in production it will be 1GB (linear). We're coming to the conclusion that we should increase this but are wondering if there's anything else we should/could do to avoid rebuilding the queue manger in order to change the log size?
What's baffling me is why do all the output messages get sent before it's rolled back?
Other things to note, the flow sometimes will take in messages of around 60-70MB but these are in a different format and processed by a different nodes and message defs. The flow has its own channels and xmit queue, all components relating to this flow have a maxmsglen of 100MB. We're using WMQ v5.3 CSD 10 and wbi v5 |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Jun 13, 2007 2:26 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Are all the new messages persistent? If so, they'll be logged as they're sent, along with all the other log work the queue manager is doing at the time, including that associated with the other flows.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Decky |
| Posted: Wed Jun 13, 2007 2:38 am Post subject: |
|
|
Novice
Joined: 16 May 2005
Posts: 16
Location: London UK
|
| yeah, they are persistent. There is some data stripped off from the original message but we're still only talking about 20MB (probably less) of actual data then. Is that enough to consistantly fill the log? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jun 13, 2007 2:45 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Decky wrote: |
| yeah, they are persistent. There is some data stripped off from the original message but we're still only talking about 20MB (probably less) of actual data then. Is that enough to consistantly fill the log? |
Clearly, if the log keeps filling! 
I suspect it's not these messages per se filling the log, but that the log is relatively full from other sources and these are tipping it over the edge.
Depending on your design you could switch to non-persistent (and therefore unlogged) messages to avoid the problem.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Decky |
| Posted: Wed Jun 13, 2007 2:51 am Post subject: |
|
|
Novice
Joined: 16 May 2005
Posts: 16
Location: London UK
|
| Thanks Vitor, switching to non persistent is not an option. I guess we'll just have to increase the log size. It's just not something we've come across before but then again we're really only beginning to use larger messages. The thing is this is in a test environment where there isn't that much going on and I've seen it happen in dev with this flow where there's even less. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jun 13, 2007 3:03 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
There could be a problem with the flow where the commit is being done too infrequently, whereas in flows where you don't see this the logged unit of work is much smaller.
Realy depends a lot on how you've got things set up - a lot of variables.
Message to file can cause problems. This is how they make money out of PM4Data!
(Other file transfer products over MQ are available, no recommendation or endorsement is implict, etc, etc) 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Decky |
| Posted: Wed Jun 13, 2007 6:30 am Post subject: |
|
|
Novice
Joined: 16 May 2005
Posts: 16
Location: London UK
|
I checked the commit count on the flow and it's set to 1 so I assume each message gets commited after it is propagated. We're going to try and increase our log size in our dev environment. Anyone got any suggestions as to how best to allocate 1GB in terms of primary and secondary log files, how big should they be?
Thanks in advance, |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jun 13, 2007 7:08 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Disk space is cheap.
Set the logs as big as they can be.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 13, 2007 2:37 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And don't just look at log size. Remember that the the actual log space is made up of two things:
- Log file size
- Number of logfiles (primary + secondary)
- limited by the available disk space
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
