| Author |
Message
|
| hopsala |
 Posted: Fri Aug 19, 2005 10:42 am Post subject: WMQ5.3 WinXP / Queue Files - Research |
 |
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
First if all, see following topic in progress - Please do not post in this thread without reading that one first.
http://www.mqseries.net/phpBB2/viewtopic.php?t=23835
I have made a few interesting experiments concerning queue file size, and wanted to share my newly found knowledge, posted ala chess game moves. In the best scientific measure, all experiments were repeated at least three times if not more, though admittadly on the same machine.
This thread is only for the internally minded, people with heart-problems and pregnant women should avoid this topic.
* Specs: WinXP, MQ5.3 CSD07.
* QFS=3KB means Queue File Size is 3KB...
* All msgs were commited at MQPUT unless stated otherwise.
* Linear logs.
For those who haven't the patience of going through the entire thing (though I assure you, interest is guarenteed!) skip to the end, there's a summary there.
Btw, some of this behavior is documented, but usually in parts, and I personally always like to see things with my own two eyes...
Last edited by hopsala on Mon Aug 22, 2005 1:01 pm; edited 9 times in total |
|
| Back to top |
|
 |
| hopsala |
| Posted: Fri Aug 19, 2005 11:08 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Experiment 1: About non persistent messages
01. define new queue | QFS=1.58KB
02. MQPUT 30KB | QFS=1.58KB
03. MQPUT 30KB | QFS=1.58KB
04. MQPUT 30KB | QFS=32KB (!)
05. Waited 10 minutes | QFS=32KB
06. MQPUT 30KB | QFS=64KB
07. MQPUT 30KB | QFS=96KB
08. MQPUT 15KB | QFS=112KB
09. MQPUT 15KB | QFS=128KB
10. MQPUT 7KB | QFS=128KB
11. MQPUT 7KB | QFS=144KB
(* All messages put are non-persistent...)
This shows that queue files do not contain all non persistent messages in queue (see steps 01-04). During the experiment one can actually see amqzlaa0 allocate more memory.
My take on this is that memory buffer size is around 64KB, thus in steps 02,03 the msgs still reside in the memory buffer and message 04 fills the buffer, thus flushing it to the queue file.
Also interesting is that the smallest allocation unit for queue files is 16KB (see 09-11)
Last edited by hopsala on Fri Aug 19, 2005 11:31 am; edited 1 time in total |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 11:17 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Experiment 2: About Persistent messages
01. define new queue | QFS=1.58KB
02. MQPUT 7KB | QFS=16KB (!)
03. MQPUT 7KB | QFS=32KB
04. MQPUT 7KB | QFS=32KB
05. MQPUT 7KB | QFS=32KB
06. MQPUT 7KB | QFS=48KB
07. Wait 10 min | QFS=48KB
(* All messages put are persistent...)
This shows that queue files contain all persistent messages in queue (steps 02-06).
Again, smallest allocation unit for queue files is 16KB.
Last edited by hopsala on Fri Aug 19, 2005 11:32 am; edited 1 time in total |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 11:22 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Experiment 3: About queue file size decreased, and
01. define new queue | QFS=1.58KB
02. MQPUT 7KB * 5 times | QFS=48KB (see experiment 2)
03. MQGET | QFS=48KB
04. MQGET | QFS=48KB
05. MQGET | QFS=48KB
06. MQGET | QFS=48KB
07. endmqm | QFS=48KB (!)
08. strmqm | QFS=9.5KB (!!)
(* All messages put are persistent.)
This shows that queue file size is only decreased upon starting a QM (steps 02-08) .
Note that here the allocation unit is flexible, simply according to msg size (7KB msg + 1.58KB bootstrap + 1KB MQMD = ~9.5KB) |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 11:31 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Experiment 4: About queue file size when MQGETting messages
01. define new queue | QFS=1.58KB
02. MQPUT 7KB * 5 times | QFS=48KB (see experiment 2)
03. MQGET * 4 times | QFS=48KB (see experiment 3)
04. MQPUT 7KB | QFS=48KB
05. MQPUT 7KB *5 | QFS=48KB
(* All messages are persistent...)
This shows that although queue file size is not decreased upon MQGET (see experiment 3), space within the file is cleared. |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 11:47 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Experiment 5: Non-persistent msgs revisited - a few putting applications
01. define two new queues | QFSA=1.58KB | QFSB=1.58KB
02. MQPUT A 30KB | QFSA=1.58KB | QFSB=1.58KB
03. MQPUT A 30KB | QFSA=1.58KB | QFSB=1.58KB
04. MQPUT B 30KB | QFSA=1.58KB | QFSB=1.58KB
05. MQPUT B 30KB | QFSA=1.58KB | QFSB=1.58KB
06. MQPUT A 30KB | QFSA=32KB| QFSB=1.58KB (!)
(* All messages put are non-persistent...)
This shows that there is a seperate queue buffer of 64KB for each queue . |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 12:16 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Experiment 6: Persistent messages revisited - SYNCPOINT
01. define new queue | QFS=1.58KB
02. MQPUT 7KB | QFS=16KB (!)
03. MQPUT 7KB | QFS=32KB
04. MQCMIT | QFS=32KB
(* All messages put are persistent and put under SYNCPOINT.)
This shows that queue files contain all persistent messages non-commited in queue (steps 02-06).
Naturally, a variant in which step 04 is MQBACK does not decrease queue file size until restart - see experiment 3.
Last edited by hopsala on Fri Aug 19, 2005 12:44 pm; edited 2 times in total |
|
| Back to top |
|
|
| bower5932 |
| Posted: Fri Aug 19, 2005 12:20 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 12:28 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
SUMMARY
* Queue files do not contain all NON PERSISTENT messages in queue (Exp. 1)
* Queue files contain all PERSISTENT messages in queue - commited and uncommited. (Exp. 2 + 6)
* Queue file size is only decreased upon starting a QM (Exp. 3)
* Although queue file size is not decreased upon MQGET space within the file is cleared. (Exp. 4)
* There is a seperate queue buffer of 64KB for each queue (Exp. 5)
* The smallest queue file allocation unit is 16KB.
IMPORTANT NOTES
* This discussion only applies to Windows (and possibly Unix), in z/OS in a whole other ball game...
* These experiments do not take checkpoints into consideration, which naturally causes slightly different behavior.
REPERCUSSIONS
* Every persistent message put, either commited or uncommited, costs IO. Mind that committed msgs are also written to log - two IO operations.
* Non persistent messages, if smaller than 64KB and do not reside too long in the queue, are only transferred through memory - no IO.
I must say I am rather surprised by this behavior; non-persistent messages behave logically - only written to queue file when buffer is full, but why are persistent messages written to queue file??
To me this seems like a waste of IO, since these messages are already written to log. What SHOULD happen is that only if the buffer is full (btw, is 64KB not really small??) or if the message stays very long in the queue, only then the msgs are written to queue file.
This is even stranger when you think about uncommited messages, every uncommited messages costs IO !
Any ideas?
Last edited by hopsala on Fri Aug 19, 2005 1:01 pm; edited 3 times in total |
|
| Back to top |
|
|
| EddieA |
| Posted: Fri Aug 19, 2005 12:34 pm Post subject: |
|
|
Jedi
Joined: 28 Jun 2001
Posts: 2453
Location: Los Angeles
|
| Quote: |
| but why are persistent messages written to queue file |
Because if you lose the logs when the system crashes, you then lose the messages.
The only way to ensure you never lose a message is to have it "hardened" in 2 independent places. OK, not 100% as you "could" lose both together, but it certainly lessens your chances.
Cheers,
_________________
Eddie Atherton
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 12:46 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Sorry, but that's really no reason. You do not back things up twice - it's over redundancy at its worst. Moreover, today discs do not "crash", it simply doesn't happen anymore - there are RAIDs, remote copies, shadow images - the need for software oriented backup is no more.
The only case in which this is needed is poor hardware backup, usually due to bureaucracy or a bad hardware support team.
Also, the worst of it is uncommited persistent msgs - you cannot "lose" such messages, as i'm sure you'll agree. Among other things, It defeats the whole purpose of better performance using SYNCPOINT.
I find no logical reason to put uncommited msgs to anywhere but memory buffers. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Fri Aug 19, 2005 7:24 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3253
Location: London, ON Canada
|
| hopsala wrote: |
| * Queue file size is only decreased upon starting a QM (Exp. 3) |
For Unix & Windows, if you can successfully issue a CLEARQ command against that particular queue then magic happens to the queue file size.
Regards,
Roger Lacroix
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Aug 20, 2005 9:15 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| RogerLacroix wrote: |
| hopsala wrote: |
| * Queue file size is only decreased upon starting a QM (Exp. 3) |
For Unix & Windows, if you can successfully issue a CLEARQ command against that particular queue then magic happens to the queue file size.
Regards,
Roger Lacroix |
But, the clear q command needs to have at least one message that it can delete to cause the file to shrink.
Load up a queue with 200,000 Persistent messages. Each 100K in length. Queue File is 20GB. Issue Clear Q command. Q file drops to 1K.
Load up a queue with 200,000 Persistent messages. Each 100K in length. Queue File is 20GB. Destructivly MQGET all the messages till the queue is empty. Queue file is still 20GB. Issuing Clear Q against the empty q does nothing, still 20 GB.
Add 1 message to that empty queue with a 20GB file size, and then issue the clear queue command against that one little message, and it shrinks the queue file down to 1K!
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Aug 20, 2005 4:02 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
As well according to the manuals:
The file space is kept so that it can be quickly reused without allocation. However if the file space is needed the qmgr will adjust the queue's file space.
Don't know how to effectively model that in a scenario. You would need to restrict your file space so as to hit near max and perhaps move messages from one queue to the other knowing that keeping the filespace on both queues would bust the max....
Enjoy  |
|
| Back to top |
|
|
| hopsala |
| Posted: Sun Aug 21, 2005 1:03 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Good, some of this is new, solid knowledge for me 
However, you did not comment about my main point - why put *uncommited* msgs to queue files? any ideas? |
|
| Back to top |
|
|
|
|
