| Author |
Message
|
| vivica12 |
 Posted: Wed Oct 24, 2007 11:17 am Post subject: rcdmqimg causing delay of Q response |
 |
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
When a rcdmqimg runs, can it cause a 'hold' on a queue so that applications can't put/get from that queue? Does it do it to the whole qmgr while this is running, or only to the queue it is recording at that time?
We have seen some issues with apps that coincide with when a rcdmqimg runs. We have this set as a monitor for when our file system grows to 60% to force a rcdmqimg.
If the rcdmqimg does a hold on the queue to archive it, would it be helpful if the log file size was less when this ran? Is there any way to make this better?
thanks for any guidance, again I've been reading tons about this, but can't seem to figure out what is the best archiving strategy, and ours is not working.
_________________
Vivica - signing off |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Oct 24, 2007 1:36 pm Post subject: Re: rcdmqimg causing delay of Q response |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vivica12 wrote: |
| If the rcdmqimg does a hold on the queue to archive it, would it be helpful if the log file size was less when this ran? Is there any way to make this better? |
Strictly speaking, it doesn't archive the queue, but takes an image of the MQ objects so I'm uncertain what impact log size would have.
As to the taking of a lock, that's a question for a passing IBMer. It's not a behaviour I've observed using rcdmqimg.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 24, 2007 8:33 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If you have a lot of persistent messages on the queue it can be a problem, even though it is very shortlived...
I think you main problem might be lack of adequate logspace (i.e. logfilesize not optimized...  Or not archived regularly
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mvic |
| Posted: Thu Oct 25, 2007 3:25 am Post subject: Re: rcdmqimg causing delay of Q response |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vivica12 wrote: |
| If the rcdmqimg does a hold on the queue to archive it, would it be helpful if the log file size was less when this ran? Is there any way to make this better? |
What are your numbers, right now, and/or at the time of running rcdmqimg, for:
- Log file no of primaries
- Log file no of secondaries
- Log file size (get an ls -l of one of the S000xxx files in /var/mqm/log/QMNAME/active)
- Queue file size (get an ls -l of one of the directory /var/mqm/qmgrs/QMNAME/queues/QNAME) |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 5:23 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
My total log space is 10GB available
My MQ settings are
LogPrimaryFiles=3
LogSecondaryFiles=2
LogFilePages=1024
Meaning 4MB file size limit
We have some queues that process 500,000 requests a day, where many of those messages are persistant -- so I know that impacts my log space, and my rcdmqimg.
Here is a log file listing
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:00 S0349799.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:00 S0349800.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:00 S0349801.LOG
etc...
there are currently 83 logs in the directory with the required logs being
checkmqlogs: MQHUBP1: Needed for Media Recovery: S0349799.LOG
checkmqlogs: MQHUBP1: Needed for Qmgr Restart: S0349876.LOG, where the first log is the media recovery log.
Here is a listing of one of our large queues, which is the one that we get application complaints that they cannot put/get from this during the time that a rcdmqimg runs (which does include our own mv and gz process to get rid of unecessary files).
ls -la IAF!SOARTM!LOG!INPUT
total 57952
drwxrwx--- 2 mqm mqm 256 Feb 10 2005 .
drwxrwx--- 491 mqm mqm 36864 Oct 05 05:53 ..
-rwxrwx--- 1 mqm mqm 29802496 Oct 25 08:17 q
The fun thing about this queue is that the q size says it's 28.26 MB, but the queue itself is has CURDEPTH(0).
One question is when a rcdmqimg runs, does it have to have enough space to cover the actual size of the Q-file, or the actual size of the message physically on the queue -- which can obviously be very different. With the Q above, does the rcdmqimg take an image of the Q-File or the messages in the Q?
I feel as if I'm going in circles, but not really making progress. All I know is that our app owners/mgrs have asked us to STOP the archival process so that it doesn't cause app impact since the Q's become 'inaccessable' at this point.
Any thoughts are much appreciated.
_________________
Vivica - signing off |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 25, 2007 5:35 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
In general, your log is very very small for your situation.
You need enough active log space to cover all of your available transactions.
I would consider it more likely that you're seeing slowdowns because an app is sending lots of persistent messages, or there's a lot of persistent messages moving across a channel and the qmgr is having to do a lot of I/O with the logs - including allocating new files.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 5:42 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
So what do you do when you don't know your 'available transactions', and what type of data would get me that information?
All I hear from app owners is that they cannot define message sizes, or transactions -- atleast when bringin an app in. How could I use current data to size better when building my v6 qmgrs?
_________________
Vivica - signing off |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 25, 2007 5:56 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
As an MQ administrator, I usually try and plan for the worst, assuming that the developers will find a way to exceed that.
If nothing else, if you can do a qmgr restart - it seems you have the disk space available and the processes in place to handle increasing your LP to at least double, and probably to 16 or 32. (these are not meaningful numbers, just pretty ones).
It's a small change, it only requires a qmgr restart... and it might eliminate your slowdown (for now, at least).
You can then make estimates based on enqueue/dequeue rates for app queues and xmit queues. Assume a reasonable percentage of persistent messages, based on your understanding of what the apps are doing. Then assume a message size of 1Mb, and figure out what your log space will look like.
Or just max out LP and LS, and recreate the qmgr with as large an LF as you can stand, or is allowed on your platform. That's the gist of that "familiar quote" that PeterPotkay mentions - he's a smart guy in these areas.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 6:39 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
So today and tomorrow are our PEAK traffic in MQ for the entire year.
So if I look at the total size of ALL logs written for today -- let's say that ends up being 5GB, which in my system will be 1280 actual physical logs.
If I take that number, 5GB -- being our largest volume of required log space, without a rcdmqimg. So if I say I want 200MB for each file, that equates to a total of 26 physical files. So then I could set 16 Primary, 10 Secondary....
Would this type of 'math' make sense if I'm rebuilding my QMGR to improve our logging? Do these numbers look troublesome?
_________________
Vivica - signing off |
|
| Back to top |
|
|
| mvic |
| Posted: Thu Oct 25, 2007 7:13 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vivica12 wrote: |
| -rwxrwx--- 1 mqm mqm 29802496 Oct 25 08:17 q |
OK the real number that matters is the size at the time rcdmqimg runs. Let's say it's 29 Mb (and these are real data, not empty space) at that time. rcdmqimg will read 29 Mb from the Q file and write 29 Mb or so into the log files. That's the best part of 4 log files, and this process takes a finite time. And while it runs your apps won't be able to access the queue. I'd expect this process to take a few seconds at normal load.
If the queue file is 100s of Mb it'll take a long time. Actually the best time to run rcdmqimg is when the q depth is zero. Then it should take a very short time indeed.
| Quote: |
| The fun thing about this queue is that the q size says it's 28.26 MB, but the queue itself is has CURDEPTH(0). |
MQ doesn't immediately release space from its data files. That's a perfectly normal observation. However I thought MQ did compact the queue before recording the image. but I might be wrong. If you think MQ is recording unreasonable amounts of data when the queue is demonstrated to be empty, and this is giving your system performance problems, this would be a valid query for IBM Support I think. IBM Support would want you to substantiate this with documented evidence of course.
| Quote: |
| One question is when a rcdmqimg runs, does it have to have enough space to cover the actual size of the Q-file, or the actual size of the message physically on the queue -- which can obviously be very different. With the Q above, does the rcdmqimg take an image of the Q-File or the messages in the Q? |
I thought it was somewhere close to just "the messages in the Q" - ie. MQ would compact the queue before taking the image. But I am not sure.
| Quote: |
| our app owners/mgrs have asked us to STOP the archival process so that it doesn't cause app impact since the Q's become 'inaccessable' at this point. |
This will cause the set of files required for media recovery to grow without limit. rcdmqimg keeps this set moving forwards. The answer (at least in theory) is to run rcdmqimg at a time of day when the queue is empty. Sometime just AFTER a batch run, where apps empty the queues, might be an example. |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 7:32 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
|
| Back to top |
|
|
| mvic |
| Posted: Thu Oct 25, 2007 7:59 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vivica12 wrote: |
| Does anyone have any examples of what they set for Primary/Secondary and Logfilesize? I want to reset ours, but feel confident that I'm going in the right direction and not cuasing more performance issues. |
No examples to share. Anyway your "real" numbers for data throughputs are the crucial factor - other people's will be different.
Having big logs will not slow your system down. In fact it might speed things up for you slightly, since less activity will be required, and less waiting, for new log files to be created - eg. the 4 or so new ones that you need during rcdmqimg of that queue (assuming it really is 29 Mb at the time rcdmqimg runs of course).
The document you linked mentions that compaction occurs at rcdmqimg time ... and I'm not sure the mentions of IPPROCS and OPPROCS are related to these rules on when compaction is done. It seems unlikely to me that apps should need to be coordinated with the rcdmqimg activity in this way. But check with IBM Support for a more authoritative answer.
(By the way, I just noticed that that document's "Modified date" is more than 4 years ago... which I think is before the release of v6, even though the document mentions v6!) |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 11:50 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
I'm asking something again, since I wasn't sure about the answer.
I do not know my data throughput -- at this time. I hope to soon with some monitoring tools.
With that, I am looking at the fact that over the course of a full day I utilized 5GB of log space, 1280 actual files of 4MB each. So knowing that's my needed log space in a day, during peak time in the year...does it make sense to use that number as a basis for recreating a new QMGR.
If I use the 5GB as my required log space, what is the best way to allocate primary vs secondary logs, and how big they should be. I can choose any random number from 1MB files to 1GB files...but how do I determine how to seperate these. I can pick arbitrary numbers of 100MB file size, which equates to 52 files needed...but again, is there any way to choose them properly, so as not to hinder rcdmqimg or archiving?
_________________
Vivica - signing off |
|
| Back to top |
|
|
| mvic |
| Posted: Thu Oct 25, 2007 1:29 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vivica12 wrote: |
| over the course of a full day I utilized 5GB of log space, 1280 actual files of 4MB each. So knowing that's my needed log space in a day, during peak time in the year...does it make sense to use that number as a basis for recreating a new QMGR. |
OK that's how much this log filesystem grows by each day... but the crucial factor in calculating required log file size and numbers, is the maximum persistent data throughput that happens while transactions are open.
Eg. if you have transactions that last 1 minute, and you have 100 Mb throughput in that 1 minute, then that 100 Mb is the number to pay attention to. You must ensure 100 Mb plus "margin for error" of log space is available in your primary+secondary logs, or else the queue manager will have to roll back the longest-running transactions.
Let's say you want a margin for error of 10 times the "normal" througput. So, 10 x 100 Mb = 1 Gb active log space is required. 50 x 20 Mb log files would do it, or 20 x 50 Mb log files. It doesn't much matter I think.
| Quote: |
| is there any way to choose them properly, so as not to hinder rcdmqimg or archiving? |
Large log file sizes should provide least disturbance to the system (and slowest growth in log file numbers). But this does not change the fact that if rcdmqimg is going to copy (say) 100 Mb from Q file to log files this is going to take many seconds to achieve. In which time, the queue is pretty much inaccessible to apps. So the biggest improvement to be made on that item is in choosing a time to run rcdmqimg that is when both (a) queue is empty and (b) apps are not trying to use the queue. |
|
| Back to top |
|
|
|
|
