| Author |
Message
|
| thimerion |
 Posted: Tue May 27, 2008 7:17 am Post subject: Has increasing active logs impact on performance |
 |
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Hi,
On the system I use, all messages are persistant.
Now a system wants to send through 20MB messages,
so I have to increase (double) the size or amount of active logs.
I was wondering what impact this change has on performance of Mq ...
Regards,
Tim |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue May 27, 2008 7:28 pm Post subject: Re: Has increasing active logs impact on performance |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| thimerion wrote: |
Hi,
On the system I use, all messages are persistant.
Now a system wants to send through 20MB messages,
so I have to increase (double) the size or amount of active logs.
I was wondering what impact this change has on performance of Mq ...
Regards,
Tim |
Huge messages have a tendency to slow MQ Down. (Size of retry on CRC mismatch etc...)
The right way to speed this up is to fragment the message and recreate it as logical at the receiving end... You can use segmentation or grouping or whatever makes sense to you.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue May 27, 2008 10:59 pm Post subject: Re: Has increasing active logs impact on performance |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| fjb_saper wrote: |
Huge messages have a tendency to slow MQ Down. (Size of retry on CRC mismatch etc...)
The right way to speed this up is to fragment the message and recreate it as logical at the receiving end... You can use segmentation or grouping or whatever makes sense to you.
Enjoy |
Surely fragmented (aka segmented) messages will slow down MQ even further with the extra overhead of handling more mesages.
20MB is not a huge message. Just increase the size &/or number of logs to cope with the expected workload. Many gigabytes of logs is not unusual.
_________________
Glenn |
|
| Back to top |
|
|
| thimerion |
| Posted: Tue May 27, 2008 11:53 pm Post subject: size |
|
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
I was thinking of 25 logfiles of 4Mb = 100Mb of active logs, this is not unusual according to your comments so thank you for your answers.
crtmqm -c "<Description> Queue Manager" -ll -lf 1024 -lp 15 -ls 10 –q <Qm_Name> |
|
| Back to top |
|
|
| exerk |
| Posted: Tue May 27, 2008 11:57 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Or allow for potential future growth of message size and make the logs the maximum size - I prefer the idea of managing a small number of large logs, to a large number of small logs - especially when using linear logging.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Last edited by exerk on Tue May 27, 2008 11:59 pm; edited 1 time in total |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Tue May 27, 2008 11:58 pm Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
Note: its important to monitor the disk space usage too...
Regards
Gayathri
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed May 28, 2008 3:20 am Post subject: Re: Has increasing active logs impact on performance |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| gbaddeley wrote: |
| fjb_saper wrote: |
Huge messages have a tendency to slow MQ Down. (Size of retry on CRC mismatch etc...)
The right way to speed this up is to fragment the message and recreate it as logical at the receiving end... You can use segmentation or grouping or whatever makes sense to you.
Enjoy |
Surely fragmented (aka segmented) messages will slow down MQ even further with the extra overhead of handling more mesages.
20MB is not a huge message. Just increase the size &/or number of logs to cope with the expected workload. Many gigabytes of logs is not unusual. |
Agreed there is more overhead. But MQ was not just built for perfect connections. On a real bad connection a 20MB message might never make it across whereas a segmented msg might... Remember size of retry block on a bad crc? Or if the channel goes down in the middle of the transmission ??.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed May 28, 2008 6:23 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| exerk wrote: |
| Or allow for potential future growth of message size and make the logs the maximum size - I prefer the idea of managing a small number of large logs, to a large number of small logs - especially when using linear logging. |
If you are using Backup Queue Managers, they rely on the Linear Logs being shipped over from the primary QM to the backup QM. If each linear log is HUGE, then they will be shipped over less frequently because it will take longer for the latest one to fill. Really big Linear Logs would mean your backup QM will be farther behind more often than if you used smaller Linear Logs.
Outside of that, I wonder what has more overhead: A QM cutting new Linear Logs more often because they are smaller, or cutting a HUGE new Linear Log file every so often? I dunno; never seen anything written on this topic.
With Circular logs its less important, I guess, since your entire ring of primary logs is created at QM build time, so make them as big as possible and as many as your disk will allow (within reason).
Geberal rule of thumb is just make a lot of logs and make them big. Disk space is very cheap. Your time worrying about this is not cheap. Outages caused because you made your logs tiny are not cheap.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| exerk |
| Posted: Wed May 28, 2008 6:28 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| PeterPotkay wrote: |
| If you are using Backup Queue Managers, they rely on the Linear Logs being shipped over from the primary QM to the backup QM. If each linear log is HUGE, then they will be shipped over less frequently because it will take longer for the latest one to fill. Really big Linear Logs would mean your backup QM will be farther behind more often than if you used smaller Linear Logs. |
Fair point, in which case I'd go for a happy medium - or will when I can convince the powers that be that BQM's are a good idea!
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
|
|
