| Author |
Message
|
| atheek |
 Posted: Sun Sep 14, 2008 2:24 am Post subject: MSGBATCHSZ |
 |
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
This is in relation to this issue which I raised a couple of months back:
http://www.mqseries.net/phpBB2/viewtopic.php?p=210333&highlight=#210333
We still face this issue, when messages go out of sequence after an app server bounce. It works perfectly normal otherwise.
We use a singleton MDB( max_bean_in_free_pool and initial_bean_in_free_poool set to 1) to pick messages off the queue,but still we notice that the redelivery message order is not the same as initial delivery order. After, some investigation I found that the MSGBATCHSZ attribute of the queue connection factory is set to the default value of 10. I am wondering if this is causing the issue and whether we need to set this to 1 to ensure the mdb or the ejb container only picks one message at a time.
As per IBM doc this field sets maximum number of messages to be taken from a queue in one packet when using asynchronous message delivery
Can anyone one shed light on what this means. We have only 1 mdb instance in the pool and we have this value as 10.
Thanks,
Atheek |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Sun Sep 14, 2008 10:36 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You have discovered another point of why you NEED to KILL that message affinity. Prepare a cost / benefit analysis for it, and management should support you...
Now as to the MDB problem... You should look into the MDB setup. There is something as to wait time before reprocessing a failed (maybe poison) message. This could contribute to your problem. As well if you don't have a DLQ and no backout queue has been defined and max bothresh is set to 1 and max reprocess rate on the MDB is set to 1(or 0)... what happens? (Never done that so I have no idea)...
Ideally in your case you would need to force the MDB to stop in case of failure of processing a message. Read up in your appserver manual on how to ensure this.
Again I understand your plight but this design is really insane. Using the MDB to just store the messages in a DB and process in order from the DB after that makes much more sense. Way more scalable...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| atheek |
| Posted: Sun Sep 14, 2008 2:25 pm Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| fjb_saper wrote: |
.
Ideally in your case you would need to force the MDB to stop in case of failure of processing a message.. |
Yes, that's want we are doing...we never faced a issue in normal scenario as messages from the sender always arrives to us in sequence, even though it comes via channel..May be there is not much in the network as
both the sender and receiver lie on n the same firewall zone within the network...
We get the error, when there is an app server bounce, in which case the inflight message doesn't rollbacks to the front of the queue, but at some position behind. In all scenarios I have observed, the out of sequence occurs within the first 10 messages, ie why I was wondering whether MSGBATCHSZ has any role in it. The app server vendor says its an MQ issue as there is only 1 instance of MDB running and when the transaction rollbacks , its upto the resource manager (MQ) to ensure that messages are rolled back in sequence.
Can anyone elaborate on what that parameter indicates? I would like to know what is meant by "packet" in that definition.
Thanks,
Atheek |
|
| Back to top |
|
|
| atheek |
| Posted: Sun Sep 14, 2008 3:49 pm Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| fjb_saper wrote: |
Now as to the MDB problem... You should look into the MDB setup. There is something as to wait time before reprocessing a failed (maybe poison) message. This could contribute to your problem. |
This should be a setting on the jms provider..right ? In this case MQ? Does there any parameter in the jndi binndings settings for this?
The app servers jms destinations and connection factories provide a redelivery delay parameter. Is there a similar setting for MQ JMS? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Sep 14, 2008 6:30 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| atheek wrote: |
| fjb_saper wrote: |
Now as to the MDB problem... You should look into the MDB setup. There is something as to wait time before reprocessing a failed (maybe poison) message. This could contribute to your problem. |
This should be a setting on the jms provider..right ? In this case MQ? Does there any parameter in the jndi binndings settings for this?
The app servers jms destinations and connection factories provide a redelivery delay parameter. Is there a similar setting for MQ JMS? |
No the msg batch size has to do with channel transfer qmgr to qmgr.
My guess is it has to do with the MDB setting on wait between reprocess....
Anyways there are ways to force a stop of the MDB. You will have to read that up on the app server doc. However what I see is that a wrong design is forcing you into one gap stop after the other and costing you a lot of hours and money...
Build a case with your management and budget it. I bet you with getting rid of the affinity problem (see my DB solution above) you could even be done earlier in processing... Don't know if that would give you any extra points with your case...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| atheek |
| Posted: Sun Sep 14, 2008 11:14 pm Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| fjb_saper wrote: |
No the msg batch size has to do with channel transfer qmgr to qmgr.
|
Do you mean to say the msg batch size on the queue connection factory is used for channel transfer? The doc says otherwise as the max number of messages delivered in one packet in asynchronous delivery.
We won;'t be able to convince the management, since the apps works fine 99.99% of the time. We get this issue only once a month when the servers are rebooted as part of scheduled maintanance. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Sep 15, 2008 2:17 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Once a month is once too many. How much time do you spend EVERY month fixing this??? This time is money too. Budget it!!
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Sep 15, 2008 2:32 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| atheek wrote: |
| fjb_saper wrote: |
No the msg batch size has to do with channel transfer qmgr to qmgr.
|
Do you mean to say the msg batch size on the queue connection factory is used for channel transfer? The doc says otherwise as the max number of messages delivered in one packet in asynchronous delivery.
|
OK, looking at asynchronous delivery, we are talking MDB. So this is max # of messages delivered in a single UOW if you request multiple messages in the same UOW. The smart thing however would be to set a different UOW for each message delivered to an MDB. Scale the number of MDB instances... Oh wait, you can't do that => you have message affinity....
Sorry
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| atheek |
| Posted: Mon Sep 15, 2008 2:55 am Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| fjb_saper wrote: |
OK, looking at asynchronous delivery, we are talking MDB. So this is max # of messages delivered in a single UOW if you request multiple messages in the same UOW. |
How can I know i am requesting multiple messages? I am using container managed transaction and my understanding was that there is only 1 message per UOW and the UOW gets committed when the onMessage method completes succesfully. As per this each message has its own UOW , but I can't process them concurrently using multiple mdb instances because of the limitation of the design. So I am processing them serially. Is this correct? or am I specifying 10 messages in a UOW coz of the MSGBATCHSZ parameter? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Sep 15, 2008 3:04 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| atheek wrote: |
| fjb_saper wrote: |
OK, looking at asynchronous delivery, we are talking MDB. So this is max # of messages delivered in a single UOW if you request multiple messages in the same UOW. |
How can I know i am requesting multiple messages? I am using container managed transaction and my understanding was that there is only 1 message per UOW and the UOW gets committed when the onMessage method completes succesfully. As per this each message has its own UOW , but I can't process them concurrently using multiple mdb instances because of the limitation of the design. So I am processing them serially. Is this correct? or am I specifying 10 messages in a UOW coz of the MSGBATCHSZ parameter? |
No this is a limiting factor. The actual behavior is set on the MDB (MDB definition and deployment parameters). You can try setting it to 1 to see if that has any positive effect for you.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| atheek |
| Posted: Mon Sep 15, 2008 3:15 am Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| fjb_saper wrote: |
| The actual behavior is set on the MDB (MDB definition and deployment parameters). |
I couldn't see any parameters in the deployment descriptors . ejb-jar has the basic stuff and some security identifiers and sets transaction type to container and trans-attribute to required.
| fjb_saper wrote: |
| You can try setting it to 1 to see if that has any positive effect for you. |
Yep.Thats what I am going to do. I know its bit difficult to re create the scenario as we get this issue only on some instances of server restart
Will try and let you know the outcome. Thanks for your help
-Atheek |
|
| Back to top |
|
|
| atheek |
| Posted: Mon Sep 15, 2008 4:01 pm Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| fjb_saper wrote: |
I bet you with getting rid of the affinity problem (see my DB solution above) you could even be done earlier in processing...
|
Hi FJ, I was wondering how your db solution removes the message affinity. The late processing approach just shifts the bottleneck from the mdb to the database reader process. The reader should still be single threaded...right?
The application is actually an integration application which brokers messages from a source sytem to various downstream systems. Some of the downstreams were designed in such a way that they depend upon their upstream to ensure sequential delivery of message. As long as we have these downstreams, we do need to have this single threaded bottleneck some where in the application..
Thanks,
Atheek |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Sep 15, 2008 7:54 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Well there are ways to remove message affinity.
Assuming that your operations are create/insert, modify/update, delete.
Set your message to hold a date-timestamp. Set the DB to only flag.
Make sure that each message carries the full object and not just the delta.
Ignore messages with the same object key and a timestamp previous to the current one.
On delete, only mark the record for deletion. The actual delete can happen in batch with a delay.
On modify with no record do a create.
It's as easy as that.
As to my remarks before: Putting the msg to a DB and processing from the DB: - Your message affinity has been moved from the queue to the DB.
- You can multithread your puts to the DB. The messages will be available for process sooner.
- you can select from the DB by criteria. You don't have to run through all messages to get the right ones from a specific group....
- It will be easier to handle poison msgs because in the meantime the DB handles volume much better than the qmgr which is designed as a pass through...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| atheek |
| Posted: Fri Sep 19, 2008 5:32 pm Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
Hi FJ, This is from the weblogic jms documentation:
| Quote: |
Asynchronous Message Pipeline
If messages are produced faster than asynchronous message listeners (consumers) can consume them, a JMS server will push multiple unconsumed messages in a batch to another session with available asynchronous message listeners.
These in-flight messages are sometimes referred to as the message pipeline, or in some JMS vendors as the message backlog.
The pipeline or backlog size is the number of messages that have accumulated on an asynchronous consumer, but which have not been passed to a message listener. |
| Quote: |
| Configure the message pipeline for asynchronous consumers — for ordered redelivery to occur for all asynchronous consumers, explicitly set the Messages Maximum = 1 on the connection factory used by the receiving application |
I think the MSGBATCHSZ parameter is equivalent of the MessagesMaximum provided by weblogic.This shouldn't have any relation to the UOW. This is just a setting to increase the concurrency of processing messages in the queue.
-Atheek |
|
| Back to top |
|
|
| atheek |
| Posted: Mon Dec 01, 2008 1:36 pm Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| Looks like setting MSGBATCHSZ to 1 fixes the issue. We had many restarts during the last two and a half months, and never got the out of sequence problem. |
|
| Back to top |
|
|
|
|
