| Author |
Message
|
| Snggr |
 Posted: Tue Nov 24, 2009 4:27 pm Post subject: Seq Number Error |
 |
|
Newbie
Joined: 24 Nov 2009
Posts: 6
Location: Canberra, Australia
|
Hi
I had a sequence number error on sender and receiver channels between two internal servers after some work was done. None of the work was MQ related. One of the servers, VMWare, had the drive where all the MQ logging and object storage is kept resized. This was done after a copy of the drive was made. The resized drive had the old image reloaded to it. After this the channel sequence numbers were out by 300k. I checked the logs and couldn't see anything indicating lost messages so I reset the channels at both servers and solved the problem. My question is has anyone else seen this kind of behaviour before, or have an explanation for it. I suspect an old drive image, but I couldn't see any gaps in the logs at all indicating missing time.
Cheers |
|
| Back to top |
|
 |
| Mr Butcher |
| Posted: Wed Nov 25, 2009 3:21 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
from my experience. ....
if one of the channel ends has sequence number 1, things have been reinstalled, recreated, re-whatever at that end.
if both sequence numbers are <> 1 and the gap is very big, then one end has been restored from an old image. in most cases the end with the lower sequence nuber.
little gaps (within batsize of the channel) with both ends <> 1 may result form an machine outage, failure or whatever with data loss at one end.
of course there are other situations too.
so from your writing i assume somebody restored an old image.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| Snggr |
| Posted: Wed Nov 25, 2009 4:29 pm Post subject: |
|
|
Newbie
Joined: 24 Nov 2009
Posts: 6
Location: Canberra, Australia
|
That was my suspicion as well. The server administrator thinks this is impossible as he did a fresh copy and used a different copy method to a backup.
The gap was 300k in a message total of around 2.6 million for the channel since last reset. I've been looking further into it as well and I've got some strange dates on object creation and modification as well. For instance the modified date on the object is in 2007 and the creation date is in 2009.
I'm still looking, I was just wondering if anyone has struck anything similar.
Cheers |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Nov 25, 2009 4:32 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| somebody restored an old image. |
This has my vote.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Nov 25, 2009 4:42 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| bruce2359 wrote: |
| Quote: |
| somebody restored an old image. |
This has my vote. |
Keep in mind that the definition of old image fits as well the scenario where the qmgr has been running for ANY length of time between the image being taken and the image being restored... Doesn't matter that it was the last created image... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| student |
| Posted: Thu Nov 26, 2009 4:18 am Post subject: Seq Number Error |
|
|
Apprentice
Joined: 17 Mar 2009
Posts: 31
Location: UK
|
| I have faced sequence mismatch issues normally during the movement of queue Queue manager from one server to another server. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Nov 26, 2009 4:36 am Post subject: Re: Seq Number Error |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| student wrote: |
| I have faced sequence mismatch issues normally during the movement of queue Queue manager from one server to another server. |
Would that be HP-NSS (Tandem as was) by any chance?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Snggr |
| Posted: Sun Nov 29, 2009 7:56 pm Post subject: |
|
|
Newbie
Joined: 24 Nov 2009
Posts: 6
Location: Canberra, Australia
|
I asked the questions a little bit differently and did some extra analysis and the sequence number that was returned after the drive resizing was about what I would have expected it to be judging by traffic volumes on the channel about the time of the last reboot.
The admin admitted that it might have picked up an old version of parts of the disk as the back-up was kind of incremental, he was getting more and more evasive at this point.
It's usually nice to get the full story first up not to have to press for it later.
Thanks for responses. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Nov 30, 2009 6:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Snggr wrote: |
| It's usually nice to get the full story first up not to have to press for it later. |
And so seldom the case!
An evasive admin is virtually a smoking gun. 
More seriously, if the admin can't be sure what the contents of the back-up used were, then you can't be sure why WMQ is acting the way it is. An almost textbook case of "it must be that MQ thing that's broken". 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Nov 30, 2009 6:26 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
If the only issue you faced from an incomplete backup and restore was a sequence-wrap mismatch, consider yourself fortunate (lucky).
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Dec 07, 2009 3:42 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Another possibility - If you change the XMITQ a SNDR channel uses, the seq # for that channel gets reset to 1.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Snggr |
| Posted: Mon Dec 14, 2009 8:57 pm Post subject: |
|
|
Newbie
Joined: 24 Nov 2009
Posts: 6
Location: Canberra, Australia
|
I've never tried changing the XMITQ of a sender channel like that, however, in this case the sequence number changed from 2.6 million to 2.3 million.
A simple reset fixed the problem, I was just trying to understand how it happened for future reference. I'm amazed that just the sequence number was effected, the actual logs and queues, with unprocessed messages, were all perfectly intact.
Sysadmin has instigated a rigorous inspection of all backup and restore procedures including VM migrations.
Cheers. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 15, 2009 6:01 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Sysadmin has instigated a rigorous inspection of all backup and restore procedures including VM migrations. |
Why am I not reassured?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Dec 15, 2009 6:09 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| bruce2359 wrote: |
| Quote: |
| Sysadmin has instigated a rigorous inspection of all backup and restore procedures including VM migrations. |
Why am I not reassured? |
Because there is no such thing as cynicism, but there is pain born of experience...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| nathanw |
| Posted: Tue Dec 15, 2009 6:57 am Post subject: |
|
|
Knight
Joined: 14 Jul 2004
Posts: 550
|
| Snggr wrote: |
That was my suspicion as well. The server administrator thinks this is impossible as he did a fresh copy and used a different copy method to a backup.
|
mmmm yeh right!
_________________
Who is General Failure and why is he reading my hard drive?
Artificial Intelligence stands no chance against Natural Stupidity.
Only the User Trace Speaks The Truth  |
|
| Back to top |
|
|
|
|
