| Author |
Message
|
| Sam Uppu |
 Posted: Thu Mar 05, 2009 1:00 pm Post subject: Messages on SYSTEM.CLUSTER.XMIT.QUEUE |
 |
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
Hi,
We are using MQ version 6 on Solaris.
We have 3 QMgrs, QM1, QM2 and QM3 in a cluster called CLUS1.
There is a cluster queue called CLUSTER.QUEUE is defined on both QM2 and QM3. Now the source application is connected to QMgr, QM1 and placing the messages onto CLUSTER.QUEUE. The messages will round robin between CLUSTER.QUEUE queues on QM2 & QM3.
This is in our performance test. When the testers apply heavy load like 72,000 messages per 30 minute span, I see some messages still sitting on the SYSTEM.CLUSTER.TRANSMIT.QUEUE even after the test is over(there is nothing coming in). The cluster sender/receiver channels are up and running and the queue depths are zero on all QMgrs, QM1, QM2 and QM3.
I see this is odd.
I see both cluster sender and receiver channels are running on both ends and why the messages are still sitting on this SYSTEM.CLUSTER.XMIT.QUEUE?. If we run one more small test, the messages are reaching the destination but the messages what were there earlier on SYSTEM.CLUSTER.TRANSMIT.QUEUE are still there. Not sure how to diagnose this. Anybody faced such kind of issue before.
Thanks for your thoughts. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Mar 05, 2009 1:29 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Did a total of 72,000 messages arrive on the CLUSTER.QUEUEs?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Mar 05, 2009 1:38 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| bruce2359 wrote: |
| Did a total of 72,000 messages arrive on the CLUSTER.QUEUEs? |
.
Some are missing. Thatswhy I started to look where the reming are. The missing count is around 4,500 and the messages on SYSTEM.CLUSTER.XMIT.QUEUE is around 2000.
We tried one more small test with around 1000 messages for a period of 5 minutes. The 1000 messages reached the destination queues but still I see the ~2000 messages on SYSTEM.CLUSTER.XMIT.QUEUE.
Thanks for showing some light on this. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Mar 05, 2009 1:54 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Browse the messages remaining on the S.C.T.Q. to see where they were supposed to go.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Mar 05, 2009 2:32 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| bruce2359 wrote: |
| Browse the messages remaining on the S.C.T.Q. to see where they were supposed to go. |
I browsed the S.C.T.Q and all those messages are from the last month. not sure why they are still there even the messages are now delivered to the destination.
I was try to clear those messages but getting object is in use. May be need to stop the channels to clear the queue.
Still I dont understand the concept of queueing. All the messages which are destined for QM2 and QM3 from QM1 will go via S.C.T.Q of QM1. Queue is a first in first out and how the messages are delivered when there are older messages sitting on SCTQ?.
Can anybody explore a littlebit on this.
Thanks. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Mar 05, 2009 2:43 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Queue is a first in first out and how the messages are delivered when there are older messages sitting on SCTQ?. |
The SCTQ is special. Clustering software manages this queue, and understands that messages may be delayed, and it will send messages even though older msgs are still in the SCTQ.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Mar 05, 2009 3:07 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| bruce2359 wrote: |
| Quote: |
| Queue is a first in first out and how the messages are delivered when there are older messages sitting on SCTQ?. |
The SCTQ is special. Clustering software manages this queue, and understands that messages may be delayed, and it will send messages even though older msgs are still in the SCTQ. |
Thanks for the info. Why a message stays on SCTQ?.
My assumption is only when the channels are down, the msg will stay on the XMIT queue. In my case the channels are up and running.
Thanks. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Mar 05, 2009 3:12 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Sam Uppu wrote: |
Thanks for the info. Why a message stays on SCTQ?.
My assumption is only when the channels are down, the msg will stay on the XMIT queue. In my case the channels are up and running.
Thanks. |
Yes but are they the right messages for the right destination?
Have you recreated the destination qmgr since?
What channel name is in the correlId ? Is this still the channel to the destination or has the channel name changed?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Mar 05, 2009 3:12 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I repeat: Browse the messages remaining on the S.C.T.Q. to see where they were supposed to go.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Mar 05, 2009 8:36 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| fjb_saper wrote: |
| Sam Uppu wrote: |
Thanks for the info. Why a message stays on SCTQ?.
My assumption is only when the channels are down, the msg will stay on the XMIT queue. In my case the channels are up and running.
Thanks. |
Yes but are they the right messages for the right destination?
Have you recreated the destination qmgr since?
What channel name is in the correlId ? Is this still the channel to the destination or has the channel name changed?
|
| Quote: |
| Yes but are they the right messages for the right destination? |
You are right. They are destined for a different QMgr which is in the same cluster for a different queue. Right now this Queue manager is down and now the messages just stayed on the SCTQ of the source QMgr, QM1.
| Quote: |
| Have you recreated the destination qmgr since? |
No
| Quote: |
| What channel name is in the correlId ? Is this still the channel to the destination or has the channel name changed? |
I see the CorrelId of the message on SCTQ is the channel name pointing to QMgr where the message is destined to. As the destination QMgr is down, the messages are stacking up in the SCTQ of the source QMgr.
Question:
Normally CorrelId is something unique for each message with which the application can retrieve the message based on CorrelId if the app wants to.
How come the CorrelId field of the message has channel name?. Is this case applicable only to messages in SCTQ?.
Thanks for sharing good info.. . |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Mar 05, 2009 8:38 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| bruce2359 wrote: |
| I repeat: Browse the messages remaining on the S.C.T.Q. to see where they were supposed to go. |
I did it and found out why the messages are on this queue. I have learnt many new things from this discussion.
Thanks a lot. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Mar 05, 2009 8:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Sam Uppu wrote: |
Question:
Normally CorrelId is something unique for each message with which the application can retrieve the message based on CorrelId if the app wants to.
How come the CorrelId field of the message has channel name?. Is this case applicable only to messages in SCTQ?.
Thanks for sharing good info... |
Yes this case is only applicable to messages on the SCTQ.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| zhanghz |
| Posted: Fri Mar 06, 2009 1:31 am Post subject: |
|
|
Disciple
Joined: 17 Jun 2008
Posts: 186
|
| You should have expected messages not delivered if the target qmgr is down and the same cluster queue is not defined in other cluster qmgrs... |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Fri Mar 06, 2009 6:56 am Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| fjb_saper wrote: |
| Sam Uppu wrote: |
Question:
Normally CorrelId is something unique for each message with which the application can retrieve the message based on CorrelId if the app wants to.
How come the CorrelId field of the message has channel name?. Is this case applicable only to messages in SCTQ?.
Thanks for sharing good info... |
Yes this case is only applicable to messages on the SCTQ. |
Thank You Sir! |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Fri Mar 06, 2009 7:09 am Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| zhanghz wrote: |
| You should have expected messages not delivered if the target qmgr is down and the same cluster queue is not defined in other cluster qmgrs... |
We didnot define the same cluster queue on the QMgr which is down. The destined QMgr and queue is for a different application which we tested the performance last month. The source app used to send some kind heart beat messages for every minute or so. As it is our performance environment, the destination QMgrs were shutdown when they are done with their performance testing. But the source app was still pushing the heart beat messages. As the destination QMgr was done, the messages were sitting on the SCTQ. I didn't check there are messages on SCTQ of the source QMgr.
Later we used the same source QMgr(QM1) to send messages for a different application QMgr(different queue). As the messages missing somewhere, I thought the messages on SCTQ belong to that application which we are testing right now but they are not when we browse the SCTQ.
Now from you guys I came to know there can be messages sitting on SCTQ to process other messages.
Thanks. |
|
| Back to top |
|
|
|
|
