| Author |
Message
|
| JeffM. |
 Posted: Fri Dec 22, 2006 5:49 am Post subject: Batch Size set to 1 |
 |
|
Newbie
Joined: 26 Jul 2006
Posts: 9
Location: Albany, NY
|
I've read and read and have found reference in the IBM documentation regarding how and why to set batch size, but I found reference to NOT setting it to 1.
My problem: An outside agency which is completely unwilling to follow best practices is trying to enforce a sort of logical order of sending messages without implementing groups by setting their batch size to one. They believe this will solve the problem. I don't think that is the case, but even if for the moment I drink the coolaid and beleve it would work I've read that it is ill advised to set the batch size to 1 due to potential data integrity issues.
I cannot find anything to support this beyond the warning, though. Can anyone tell me what those issues are? Or point me to where I can find the information... again I have the documentation that says, "don't do it" but can't find any other supporting information.
Thanks and Happy Holidays! |
|
| Back to top |
|
 |
| wschutz |
| Posted: Fri Dec 22, 2006 6:07 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
Aside from having a HORRIBLE effect on performance, there is no downside to setting BATCHSZ to 1.
However, I don't see how that can help with "assured sequential delivery"....
_________________
-wayne |
|
| Back to top |
|
|
| JeffM. |
| Posted: Fri Dec 22, 2006 6:47 am Post subject: |
|
|
Newbie
Joined: 26 Jul 2006
Posts: 9
Location: Albany, NY
|
I did get the whole performance issue, but wonder why it states in the documentation [Note the bold section below]:
... excerpt from MQ v6 intercommunication documentation....
Some considerations when choosing the number for batch size:
If the number is too large, the amount of queue space taken up on both ends of the link becomes excessive. Messages take up queue space when they are not committed, and cannot be removed from queues until they are committed.
If there is likely to be a steady flow of messages, you can improve the performance of a channel by increasing the batch size. However, this has the negative effect of increasing restart times, and very large batches may also affect performance.
If message flow characteristics indicate that messages arrive intermittently, a batch size of 1 with a relatively large disconnect time interval may provide a better performance.
The number may be in the range 1 through 9999. However, for data integrity reasons, channels connecting to any of the current platforms, as described in this book, should specify a batch size greater than 1. (A value of 1 is for use with Version 1 products, apart from MQSeries(R) for MVS/ESA(TM).)
Even though nonpersistent messages on a fast channel do not wait for a syncpoint, they do contribute to the batch-size count. |
|
| Back to top |
|
|
| killer |
| Posted: Fri Dec 22, 2006 8:17 am Post subject: |
|
|
Apprentice
Joined: 06 Jul 2006
Posts: 35
|
First of all try to get clear ,y we set batchsz .
So it's depend upon you,what's the requirement is?
For your information batchsz indicates ,
how many messages will be get commited within a unit of work.
Say suppose batcsz is 50.
so between two synpoint ,I mean in single unit of work 50(max value) messages should either get commited or rolled back.
So if you ,set it to 1,
Between two syncpoint ,single message will get commmited.
For more refernce go and look into Intercommunication pdf
 |
|
| Back to top |
|
|
| tleichen |
| Posted: Fri Dec 22, 2006 8:28 am Post subject: Re: Batch Size set to 1 |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
| JeffM. wrote: |
| ...An outside agency which is completely unwilling to follow best practices is trying to enforce a sort of logical order of sending messages without implementing groups by setting their batch size to one. They believe this will solve the problem. .... |
Sounds familiar... Well, you've already identified the the vendor's advice in the matter. But you know what they say about advice... Wise men don't need it and fools won't heed it! Good luck. 
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| JeffM. |
| Posted: Fri Dec 22, 2006 9:34 am Post subject: Re: Batch Size set to 1 |
|
|
Newbie
Joined: 26 Jul 2006
Posts: 9
Location: Albany, NY
|
| tleichen wrote: |
| Sounds familiar... Well, you've already identified the the vendor's advice in the matter. But you know what they say about advice... Wise men don't need it and fools won't heed it! Good luck. |
I get what batch size is all about... The only problem is that I need to find a real answer to support the little blurb in the documentation so I can counter what they are doing - piling up one bad practice on top of another... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 22, 2006 1:50 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| JeffM. wrote: |
I did get the whole performance issue, but wonder why it states in the documentation [Note the bold section below]:
... excerpt from MQ v6 intercommunication documentation....
Some considerations when choosing the number for batch size:
If the number is too large, the amount of queue space taken up on both ends of the link becomes excessive. Messages take up queue space when they are not committed, and cannot be removed from queues until they are committed.
If there is likely to be a steady flow of messages, you can improve the performance of a channel by increasing the batch size. However, this has the negative effect of increasing restart times, and very large batches may also affect performance.
If message flow characteristics indicate that messages arrive intermittently, a batch size of 1 with a relatively large disconnect time interval may provide a better performance.
The number may be in the range 1 through 9999. However, for data integrity reasons, channels connecting to any of the current platforms, as described in this book, should specify a batch size greater than 1. (A value of 1 is for use with Version 1 products, apart from MQSeries(R) for MVS/ESA(TM).)
Even though nonpersistent messages on a fast channel do not wait for a syncpoint, they do contribute to the batch-size count. |
Well it looked like pretty clear to me...
And if you are still using a Version 1 product ...??
Now let's be serious: The only reason for setting a value of 1 is to:
- Ensure a smooth flow for very big messages and minimize the log usage and backout size.
- If you ABSOLUTELY can't getting it working any other way.
You do remember that the batch size only kicks in if there are enough messages to have it kick in. Any lesser batch size will still be transmitted immediately when the xmitq has no more messages in a gettable state.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| JeffM. |
| Posted: Tue Dec 26, 2006 5:40 am Post subject: |
|
|
Newbie
Joined: 26 Jul 2006
Posts: 9
Location: Albany, NY
|
| fjb_saper wrote: |
Well it looked like pretty clear to me...
And if you are still using a Version 1 product ...??
Now let's be serious: The only reason for setting a value of 1 is to:
- Ensure a smooth flow for very big messages and minimize the log usage and backout size.
- If you ABSOLUTELY can't getting it working any other way.
You do remember that the batch size only kicks in if there are enough messages to have it kick in. Any lesser batch size will still be transmitted immediately when the xmitq has no more messages in a gettable state.
|
Yes what is says is clear, "You shouldn't do it" and I get how batch size works - All you have to do is read the documentation to get it. You make good points in your list of reasons of why TO do it. But my question was why NOT to do it. There has to be a reason or reasons. The documentation doesn't give a clear reason. Oh, it says there can be "data integrity" issues, but that isn't a reason that is a result of something bad happening. Also, I don't just read something and assume it is true unless there is supporting information. That is the "fall in line" mentality.
I'm in need of concrete reasons of what bad things can happen if it is set to 1. Maybe none, or maybe at least none I need to be concerned with. However, in order to combat this outside agency I was looking for more supporting documentation than just one line that says, "becuase there can be problems don't do it". I certainly can't make an argument based on that.
I have read just about all the IBM MQ documentation available but haven't been able to solve this one. I just thought that someone here might have some more information on it that goes a bit farther...
I do appreciate all the responses, though! |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Dec 26, 2006 7:14 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Peformance is the big reason not to set the Batch Size to 1.
As Wayne said.
You "get" the whole performance issue, so I won't go into it any more than that.
I suspect that the "data integrity" point in the documentation you've quoted has to do with message ordering. But I'd have to go back and read the parts of the Intercommunications manual that talk about message ordering to figure out why - and I'm sure you've read those sections more recently.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| deragonken |
| Posted: Tue Dec 26, 2006 9:35 am Post subject: |
|
|
Newbie
Joined: 26 Dec 2006
Posts: 1
Location: Division of Criminal Justice, Albany NY
|
Since I'm the person that made the suggestion to set the batch size to 1, I thought I would explain my reasoning.
The problem is our two agencies are replacing legacy LU6.2 conversions that are between an IBM and Unisys mainframe. These LU6.2 conversations operate using a single transaction commit which provides a certain guarentee of order. When these applications were originally developed (20 years ago), a logic dependency on this order was relied upon, and if something got out of order the applications fail.
We are replacing this LU6.2 connection with MQSeries sender/receiver channels where neither queue manager resides on the application mainframes. Our side uses a gateway to send and receive messages between MQSeries and the mainframe and we are confident that the order is preserved for messages leaving our  mainframe applications and getting delivered to our local queue manager.
Where order becomes a little questionable is in the delivery to the remote queue manger over the MQ channels. IBM does not guarentee order (at least according the the documentation) in this case, although it is my understanding that order will only get lost in some very specific scenerios. Our configuration is a straight forward TCP/IP transport, a single channel, and a single transmit queue. Since setting batch size to one single threads the messages that traverse across this channel, I thought that it should provide an additional assurance that the order will get preserved.
I do not object to the use of MQSeries groups, although in the case of these applications the delineation of a group is not straight forward. In one case, I believe a new group is defined once every 24 hours which could be a real problem.
I welcome any suggestions as to how we might solve our problem and whether there is a real concern about loosing message order when traversing MQ channels.
Thanks |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Dec 26, 2006 9:58 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The best suggestion of all is to make each message completely independant of every other message. That way order doesn't matter.
I think you should sit down and re-examine each message flow and re-examine the need for that particular message flow to be dependent on the order of the messages.
A single message should represent a single logical unit of work. If the unit of work is big, it may make sense to send several messages that represent the single unit of work - and message grouping is one way to do this. Then the order in the group represents the order that each piece of the unit of work should be processed in.
I can't really conceive of a good reason to have a new group of messages created every twenty-four hours. What's different between the message at 23:59:59.xxxx hours and the next message? The fact that it's been twenty-four hours? That's not a logical group, that's a timestamp.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Dec 26, 2006 2:49 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Furthermore if your group is a big batch I would suggest some additional measures:
Have a place in the message to specify batch identification (in the payload). Log messages in a DB. So if for what ever reason message xyz in batch uvw did not process right and you need to resend it, you can.
But seriously the best thing is to go away from a batch mode and work more real time. One atomic transaction = 1 message...= 1 unit of work....
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
