| Author |
Message
|
| brian_r |
 Posted: Tue May 29, 2012 5:03 am Post subject: Channel Keeps going out of Sync |
 |
|
Apprentice
Joined: 28 Jan 2008
Posts: 31
Location: Dublin
|
Hi,
I have a channel that keeps going out of sync no matter how much I reset it.
This first started to occur after the RCVR channels server crashed. After which the following is happening.
Sequence of events is:
1. Messages arrive on xmitq to start channel. Channel is out of sequence as the RCVR is expecting 1(usually).
2. Stop channel. Reset SDR and Reset RCVR. Start Channel.
3. Channel runs for approx 8 hours during the day and messages are transmitted fine. Ends normally at end of business.
4. Batch runs and starts up the channel again. Again the channel goes into retry state with out of sequence RCVR expecting 1 again. Channel has to be reset again.
Looks like for some reason the seq no of the RCVR is going back to 1 when the channel ends normally.
For info its MQ version 7 on both sides and going from mainframe shared queue manager to distributed Solaris queue manager.
Thanks in advance |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue May 29, 2012 5:35 am Post subject: Re: Channel Keeps going out of Sync |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| brian_r wrote: |
| Looks like for some reason the seq no of the RCVR is going back to 1 when the channel ends normally. |
When I see channel sequence numbers going back to 1 my first thought is the channel's being altered. Is it possible that, after the 1st crash, some well meaning admin wrote a script that's running at the end of business to "clean" the channel & prevent further problems?
Illustravtive war story - exactly the problem you describe but sequence numbers were 1 first thing on a Monday morning. Problem traced to a script that deleted & redefined the channel using the version is the source control "to remove unauthorized changes" at the request of the people who audited the change control process. Got very defensive & embarassed when I asked where the change control was for the script & why I wasn't on the approval list....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| brian_r |
| Posted: Tue May 29, 2012 5:51 am Post subject: |
|
|
Apprentice
Joined: 28 Jan 2008
Posts: 31
Location: Dublin
|
Thanks Vitor,
I can confirm there is no clean up script in place. I built the Queue Manager so would know of such things. |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Tue May 29, 2012 6:07 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
what happens if - during step 3, you stop the sender channel and restart it (simulating your "normal end of business" without the "batch run").
is it again out of sync or does everything continues normally without sequence errors?
to me also the batch run (or whatever runs on the receiving end) is suspicious to do something that resets the channel, as vitor already said.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue May 29, 2012 6:14 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| brian_r wrote: |
| I can confirm there is no clean up script in place. I built the Queue Manager so would know of such things. |
I admire your confidence & procedures. In the example I quoted, I built all of the queue managers in question & was on the approval list for all WMQ related changes. Yet still this script got itself onto the schedule without my knowledge.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| SJ |
| Posted: Tue May 29, 2012 6:26 am Post subject: |
|
|
Newbie
Joined: 02 May 2003
Posts: 8
Location: US
|
| We have had the same situation. Tandem-NonStop to Unix. Though I don't think the platforms matter. For some unknown reason the channel sequence number sync information was not getting saved correctly. We ended up deleting and readding both the sender and receiver channels. This fixed the issue. Next time we will try the receiver channel first. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue May 29, 2012 4:09 pm Post subject: Re: Channel Keeps going out of Sync |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| brian_r wrote: |
| 2. Stop channel. Reset SDR and Reset RCVR. Start Channel. |
The minimum requirement for resetting the sequence number on SDR / RCVR channels is:
1. Reset RCVR to the sequence # the SDR is expecting.
OR
2. Reset SDR to any sequence # (eg. 1).
The channels should start running the next time a retry occurs, or if you can't wait that long, stop and then start the SDR channel.
| Quote: |
| 4. Batch runs and starts up the channel again. Again the channel goes into retry state with out of sequence RCVR expecting 1 again. |
The RCVR queue manager could be losing the channel sync message in the SYSTEM.CHANNEL.SYNCQ or the message could be corrupted. Deleting and redefining the channel might fix this.
Another possibility is that a different IP address is being used to contact the RCVR system.
_________________
Glenn |
|
| Back to top |
|
|
| mvic |
| Posted: Tue May 29, 2012 5:00 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| brian_r wrote: |
| I have a channel that keeps going out of sync no matter how much I reset it. |
Reset it once. Then leave it.
| brian_r wrote: |
| I can confirm there is no clean up script in place. I built the Queue Manager so would know of such things. |
If you are sure, then you should consider opening a PMR with IBM, and asking IBM why it is happening.
IBM will likely ask you to trace what is going on with the channel. Now would be a good time to double-check that no "special" processing from home-grown scripts is going on. If it is going on, it will be visible in the trace. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue May 29, 2012 5:23 pm Post subject: Re: Channel Keeps going out of Sync |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| brian_r wrote: |
4. Batch runs and starts up the channel again. Again the channel goes into retry state with out of sequence RCVR expecting 1 again. Channel has to be reset again. |
Exactly and precisely, what does the batch script do? What commands does it issue?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 29, 2012 6:25 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
worst case scenario: you are running the channel with an mcauser that has no or not enough rights to the SYSTEM.CHANNEL.SYNC.QUEUE.... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| brian_r |
| Posted: Wed May 30, 2012 1:28 am Post subject: |
|
|
Apprentice
Joined: 28 Jan 2008
Posts: 31
Location: Dublin
|
Thanks to all replies!!
To answer a few posts.
Resetting actions: I have tried setting the sequence number in the RCVR manually and resetting the SDR to 1, but neither resolve the issue.
The Batch simply issues an MQCONN, puts msgs on a remote queue, which go to a xmitq queue which triggers the starting of a channel.
Regarding the SYNCQ I cant see any issues with that. Also there are other SDR/RCVR channels working fine that are using this SYNCQ. The mcauser has permissions to this queue.
At this stage I have raised a PMR, will update with the outcome.
In the interim Ive recreated the RCVR channel and reset the SDR which has solved the problem. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed May 30, 2012 5:22 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
It's been my experience that channels get out of sync because someone in the mqm group deletes/redefines/alters the RCVR end of the channel.
Wen your channel next gets out of sync, note the channel change/alter dates.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| brian_r |
| Posted: Wed May 30, 2012 5:34 am Post subject: |
|
|
Apprentice
Joined: 28 Jan 2008
Posts: 31
Location: Dublin
|
| bruce2359 wrote: |
It's been my experience that channels get out of sync because someone in the mqm group deletes/redefines/alters the RCVR end of the channel.
Wen your channel next gets out of sync, note the channel change/alter dates. |
From before channel was replaced:
ALTDATE(2012-03-21) |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed May 30, 2012 5:54 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Channels have two ends.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| cplittle |
| Posted: Thu May 31, 2012 6:49 pm Post subject: |
|
|
Newbie
Joined: 05 Apr 2011
Posts: 2
|
|
| Back to top |
|
|
|
|
