| Author |
Message
|
| BBM |
 Posted: Thu Dec 01, 2011 9:34 am Post subject: Multiple sequence number mismatches |
 |
|
Master
Joined: 10 Nov 2005
Posts: 217
Location: London, UK
|
Hi,
We are seeing lots of sequence number errors on our receiver channels that external clients connect into.
After resetting the channel to the correct sequence number it runs until the disconnect interval then next time it starts up we get another mismatch.
We are getting this on multiple receiver channels.
MQ is version 7.0.1.5
Sometimes the mismatch is quite large - 1 versus 3184 for example. I'm worried that we are losing some sync information somewhere. No one has tampered with the SYSTEM.CHANNEL.SYNCQ or anything like that.
We are running BlockIP2 on the receiver channels.
Thanks
BBM |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Dec 01, 2011 9:39 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Do you see any errors, at all, between when you resolve the channel and when it disconnects the next time?
If everything is absolutely quiet on your side, then your partners are gratuitously resetting the channels on their side. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 01, 2011 9:42 am Post subject: Re: Multiple sequence number mismatches |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| BBM wrote: |
| Sometimes the mismatch is quite large - 1 versus 3184 for example. I'm worried that we are losing some sync information somewhere. |
If the number on one side is 1 then it's almost certain that the channel has been manually reset or has been ALTERed or DELETEd/DEFINEd. I would assume more WMQ-esq issues if the mismatch was 1 between 2 large numbers or the same size as the batch the channel was using.
It is within my experience for clients to have scripts that run automatically to do this "to ensure the objects match the definition in the source control library". 
It is also within my experience for such commands to be run manually by inexperienced admins who don't admit it until properly motivated. Especially when you start asking round and they realise they're causing a problem.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| BBM |
| Posted: Thu Dec 01, 2011 11:55 am Post subject: |
|
|
Master
Joined: 10 Nov 2005
Posts: 217
Location: London, UK
|
Thanks for the replies.
The sequence number is always lower on my (the receiver) end.
I'm getting no other errors apart from the AMQ9526s
I didn't realise a ALTER on the channel will reset the channel - is this the case?
The weird thing is that 5 or 6 channels just started doing it today out of the blue and they all had to repeatedly reset their sequence numbers during the day to get working again. Activity has ramped up on our queue manager significantly this week.
I'd be shocked if they were inadvertently all resetting their sequence numbers manually to these random and sometimes quite high values. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 01, 2011 12:09 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| BBM wrote: |
| I'd be shocked if they were inadvertently all resetting their sequence numbers manually to these random and sometimes quite high values. |
That would be very strange indeed. What's much less strange is a sequence number reverting to 1 (rather than 13, 42, 666 or 1-batch-interval) which is what a channel does when it's reset or changed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| BBM |
| Posted: Thu Dec 01, 2011 2:32 pm Post subject: |
|
|
Master
Joined: 10 Nov 2005
Posts: 217
Location: London, UK
|
Yep well the channels are not reverting to 1 always to some random fairly high number which is worrying.
My current trains of thought are network issues or the latest version of BlockIP2 somehow causing this issue.
I have disabled BlockIP2 and have asked networks to run trace.
If all other avenues have been explored as to why these mismatches are occurring so frequently I guess the only thing left is network issues. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Dec 01, 2011 2:57 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'm aware of only two methods of setting the seqwrap field value to 1, and both have been identified by my learned colleagues:
1) delete and recreate the receiver channel definition; and
2) use the mqsc reset (or equivalent WMQ Explorer gui) at the receiving end of the channel
Both/either will keep the channel from successfully starting.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| BBM |
| Posted: Thu Dec 01, 2011 11:41 pm Post subject: |
|
|
Master
Joined: 10 Nov 2005
Posts: 217
Location: London, UK
|
| Yep agreed that was my understanding and why I was puzzled when Vitor mentioned an ALTER could reset a channel or maybe I misunderstood. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 02, 2011 12:04 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
What will also reset the sequence number (or give you a seq # mismatch) is a change of target in the sender channel...
Suppose you have 2 qmgrs:
QMGRA on machine A
QMGRB on machine B (clone of QMGRA)
Suppose you have an alias on QMGRB to consume msgs destined to QMGRA initially...
Now you have QMGRC sending msgs to QMGRA.
you have the same receiver definition (QMGRC.TO.QMGRA) on QMGRB.
On QMGRC you change the sender channel (conname) to point to QMGRB.
As the sequences numbers on QMGRB do not match the ones on QMGRA you have a sequence number problem...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 02, 2011 2:40 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Is it in any way possible that there are two channels on the other side, both talking to the same receiver/etc. channel on your side? |
|
| Back to top |
|
|
| Koit |
| Posted: Mon Jan 23, 2012 7:10 am Post subject: SEquence counter goes to 1 |
|
|
Novice
Joined: 02 Jan 2006
Posts: 12
|
I just wanted to add that we are experiencing the same problem, i.e. the receivers counter goes to 1 while the sender still has some large number. The receiver is on a Solaris with MQ 7.0.1.3. The senders have been on either Windows or OpenVMS. Mostly the problem occurs when there some activity on the sender side such as a fail over. The Windows installations are on MSCS. The OpenVMS is not a clustered system.
I have seen the problem on two different qmgrs.
We definatly haven't done anything to the channels on either end when these errors occur.
At one time I saw a logging on the receiver side that the channel wasn't defined. After a while I got the sequence error. It seems like there is a bug in MQ that makes it loose the definition of the channel but then finds it again.
I have raised a PMR with IBM for this problem. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jan 23, 2012 9:24 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Getting back to your failover scenario...
Looks like everytime the failover kicks in, you are getting sequence number difference. I would not be surprised about that as the ip address from the sender seems to have changed... This is recognized as a different sender channel by the receiver as I understand it.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Mon Jan 23, 2012 10:39 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
| Quote: |
| Is it in any way possible that there are two channels on the other side, both talking to the same receiver/etc. channel on your side? |
what mqjeff said that was my first thought too. We do have customers that install more than one test environment, while at our end there is only one test environment and only one receiver. depending on, which test environment the client is using we get sequence errors or not ....... sometimes they use the same ip address for all systems, and start either this or that system. or they have a "active" and a "backup" (cold standby) system and switch to that from time to time and then back. or they restored a image backup taken.
imho, these issues cover almost all sequence number problems <> 1 at the remote (sending) end.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Jan 24, 2012 1:16 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Mr Butcher wrote: |
| Quote: |
| Is it in any way possible that there are two channels on the other side, both talking to the same receiver/etc. channel on your side? |
what mqjeff said that was my first thought too. We do have customers that install more than one test environment, while at our end there is only one test environment and only one receiver. depending on, which test environment the client is using we get sequence errors or not ....... sometimes they use the same ip address for all systems, and start either this or that system. or they have a "active" and a "backup" (cold standby) system and switch to that from time to time and then back. or they restored a image backup taken.
imho, these issues cover almost all sequence number problems <> 1 at the remote (sending) end. |
Well, this is interesting. As some of you may know that there is not much difference between the way receiver channels work compared to Server Connection channels (exept that they use completely different channel protocols, of course). They both can run several instances of the channel. The same way a Server Connection Channel can be shared by several clients simultaneously, a Receiver channel can serve several Sender channels. All instances have their own bookkeeping, which means that they maintain their own message sequences as well. Afaik this is one of the cornerstones of Queue Manager Clusters.
So in a case where the first queue manager instance crashes and is not able to shut down the Sender channel properly, some messages in the transmission queue may stay in precommitted state.
Now when the other instance starts the Sender channel, the receiving MCA starts a new instance of the Receiver channel, which may look like the receiver had reset the message sequence counter. But in fact the new instance has started a new sequence.
The messages that are in precommitted state on the sender side are trapped. You cannot resolve the channel because the Sender you are running is talking to another instance of the Receiver channel.
Now why doesn't the Receiver reuse the existing instance? Should it not be able to identify the Queue Manager in the handshake process? Is it because the connection is opened from another ip address?
The channel handshake should communicate the information that the sender queue manager is a multi-instance queue manager. And in that case the receiver should reuse an existing instance even if the connection now comes from another ip address. It seems it doesn't happen now.
Just guessing again, of course.... |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Jan 24, 2012 2:31 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Esa wrote: |
| Is it because the connection is opened from another ip address? |
AFAIK, Yes.
_________________
Glenn |
|
| Back to top |
|
|
|
|
