| Author |
Message
|
| thimerion |
 Posted: Wed May 09, 2007 1:31 am Post subject: Sequence problems with MQ series |
 |
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Hi,
I have a problem with my messages getting out of sync, I am looking for possible causes and solutions.
I have following setup:
A java program using the following commands:
queue.put(putMessage, pmo);
qMgr.commit();
The setup is as follows:
message is put on a remote def queue QUEUE1, transmission queue TRANS1, to another queue manager QM2, again on a remote def queue QUEUE2, transmission queue TRANS2, to another queue manager QM3, there it is put on a local queue.
I see in the log that the qMgr.commit(); of the first message is issued 6 seconds later than the one for the second message. Still the message is out of sync when it reaches the destination.
Any help possible?
Also, is there any logging / stats in mq how I can see in the logs when the message was received?
Great thanks,
Tim |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed May 09, 2007 1:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
On a general point, message sequence is not guaranteed by MQ. Designing applications to process in this way is called "message affinity" and is considered a bad idea. Do a search in the forum, it's been discussed many times and there might be some useful tips to aid your situation.
If I had to guess, I'd say your messages are being batched up on the middle queue manager and then sent LIFO. If this guess is correct you might fix it by changing the channel parameters, but you'd pay a heavy price in terms of resources. This is one of the reasons message affinity is a bad thing.
If you're using v6 MQ you can use the new accounting and statistics functions to get the information you need.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| thimerion |
| Posted: Wed May 09, 2007 2:07 am Post subject: sequence very important |
|
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Hi,
I really need the messages to be kept in sequence. they are delivered 6 secs later to the queue, so I wonder how the otherone could go first.
I use mq version 5.3, any way of seeing in there the time when a message passed a queue.
Tim |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 09, 2007 2:19 am Post subject: Re: sequence very important |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| thimerion wrote: |
| I really need the messages to be kept in sequence. they are delivered 6 secs later to the queue, so I wonder how the otherone could go first. |
Because they're going through an intermediate queue manager, the most likely reason is they're being held there to form a batch & then delivered to the final target.
Think long and hard if there are any alternatives to sequences of messages. It's a long, hard road you're walking down and this will not be your last problem.
| thimerion wrote: |
I use mq version 5.3, any way of seeing in there the time when a message passed a queue.
|
Not easy out of the box.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| thimerion |
| Posted: Wed May 09, 2007 2:25 am Post subject: I agree |
|
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Hi,
I agree on your comments of message sequence, it's not a good idea ... But that is not our descision, we have to deliver the messages to the customer in sequence. However we will talk to him about it.
As a short term fix to reduce the chance to go out of sequence I understand,
- We can reduce the number of remote def queues ... sending directly to QM2 over TCP?
- We can change the config to go FIFO ... how do I do that?
Tim |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Wed May 09, 2007 2:31 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
are you setting message priority by any chance?
the default delivery sequence for queues is priority.
when you increase the interval, do messages still arrive out of sequence?
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| thimerion |
| Posted: Wed May 09, 2007 2:41 am Post subject: to Michael |
|
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Hi Michael,
We are not setting priority, so it's all out as default priority.
What interval do you mean? The channels are set up to be triggered.
Tim |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 09, 2007 2:48 am Post subject: Re: I agree |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| thimerion wrote: |
- We can reduce the number of remote def queues ... sending directly to QM2 over TCP? |
No, send directly to QM3. I'm assuming you're already using TCP for your channels so that won't be a change.
| thimerion wrote: |
- We can change the config to go FIFO ... how do I do that?
|
In your application, or you can change the default setting on the queue. If I'm right about what's happening it won't help because it simply makes the priority of the message irrelevent. It doesn't change delivery sequence, which MQ never guarantees.
(If you send 3 high priority persistent messages followed by 3 low priority persistent messages, it's likely that the high priority ones will be recieved first but not certain. Depends on things like commit points, batch sizes, channel priorities, etc, etc. All that is guaranteed is that all 6 messages will be delivered)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| thimerion |
| Posted: Wed May 09, 2007 2:51 am Post subject: QM3 |
|
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Thanks Vitor,
Sending to QM3 that will not be possible due to firewall restrictions, machine 1 cannot send to QM3 but has to go over the machine running QM2. QM1 we can skip however if we use TCP instead of bindings mode.
Tim |
|
| Back to top |
|
|
| thimerion |
| Posted: Wed May 09, 2007 4:40 am Post subject: read receipt |
|
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Hi,
Has Mq the possibility to send a sort of read receipt, so that we can wait for that receipt, before sending out the next message?
Tim |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Wed May 09, 2007 4:43 am Post subject: Re: read receipt |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
| thimerion wrote: |
Hi,
Has Mq the possibility to send a sort of read receipt, so that we can wait for that receipt, before sending out the next message?
Tim |
look up COA (confirmation of arrival) and COD (confirmation of delivery)
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed May 09, 2007 4:43 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You mean a Confirm On Arrival or Confirm on Delivery Report Message?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed May 09, 2007 4:44 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
But let's be clear.
The *best* choice is to put a sequence number in the data. Then the receiver will be able to store and retrieve messages that arrive OUT of sequence, and know which messages may or may not be "missing".
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 09, 2007 4:53 am Post subject: Re: read receipt |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| thimerion wrote: |
Has Mq the possibility to send a sort of read receipt, so that we can wait for that receipt, before sending out the next message?
|
Using these facilities (as described by my colleague) will single thread your message flow, slowing it down and preventing any possibility of scaling.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| thimerion |
| Posted: Wed May 09, 2007 4:53 am Post subject: out of sequence |
|
|
Acolyte
Joined: 08 May 2007
Posts: 67
Location: Belgium
|
Hi,
I know it is better to have the program to detect the out of sequence messages, but I don't control that part of the flow.
The only thing that I have under my control is to receive the message, and I need to send it out in a different format, but in sequence as I received it.
I am looking for the best case scenario here, without that I have to change the source or the target program.
Tim |
|
| Back to top |
|
|
|
|
