| Author |
Message
|
| Sunntu |
 Posted: Mon Aug 05, 2002 4:40 am Post subject: Disappearing messages (between S390 & AIX) |
 |
|
Novice
Joined: 04 Aug 2002
Posts: 11
|
Hi,
We have a cobol application running on S390 placing messages on remote queues residing on AIX platform.
I have set persistence at message level.
In the normal case:
that is, when the sender channel and the remote listener are running everything goes fine. I have defined a process to trigger the channel when a message comes into the XMITQ. And this happens as desired.
But, if I stop the channel and restart it (we are tesing this specific case), then everything goes haywire. When the channel is in stopped state, even if I place persistent messages, they disappear from the queue once I start the channel (they do not reach the destination). The messages which I place from now on remain in the XMITQ and the only way to resolve the problem was to stop the channel again, empty the XMITQ and then restart the channel again.
Any idea what may be wrong ? The missing messages are not found in the DLQs on OS390 nor on AIX.
Any help is appreciated.
Thanks in advance.
Sundari |
|
| Back to top |
|
 |
| mrlinux |
| Posted: Mon Aug 05, 2002 4:46 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
A couple of questions
1) How did you set the persistence ???
2) Did the application return any error codes during the put ???
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| Sunntu |
| Posted: Mon Aug 05, 2002 4:57 am Post subject: Disappearing messages |
|
|
Novice
Joined: 04 Aug 2002
Posts: 11
|
| mrlinux wrote: |
A couple of questions
1) How did you set the persistence ???
I set by setting the message descriptor MQMD to MQPER-PERSISTENT in my program.
2) Did the application return any error codes during the put ??? |
No error code. Everything went successfully. |
|
| Back to top |
|
|
| mrlinux |
| Posted: Mon Aug 05, 2002 5:04 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
How are you stopping the channel ????
try your test again,
and do the following
runmqsc
dis chs(CHANNEL_NAME) all
and post the results.
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| Sunntu |
| Posted: Mon Aug 05, 2002 5:17 am Post subject: Disappearing messages |
|
|
Novice
Joined: 04 Aug 2002
Posts: 11
|
Oh..no I am on OS/390. I cannot do a dis(...).
I stop the channel in stop mode Quiesce. And the status changes to 'STOP'. My stop command finishes successfully. |
|
| Back to top |
|
|
| mrlinux |
| Posted: Mon Aug 05, 2002 5:24 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
actually you can.
from the AIX box (Assume you have channels from AIX to OS390 ) and it is the default qmgr.
runmqsc -x -w 40 MAINFRAME_QMGR_NAME
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| poki |
| Posted: Mon Aug 05, 2002 3:28 pm Post subject: |
|
|
Newbie
Joined: 05 Aug 2002
Posts: 9
Location: US
|
Sunnutu,
The trigger application which u have defined on OS - 390 would have been picking up the message.
If its CICS then put your triggered transaction in CEDF mode and then send your message from AIX. You will come to know wheather its getting picked up by appl.
Happy debugging |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Mon Aug 05, 2002 7:43 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3253
Location: London, ON Canada
|
Yup, I have seen this before.
Check the persistance setting for your XMITQ. I bet it is set to "DEFPSIST(NO)". Change it to "DEFPSIST(YES)" and your problems will go away.
The rules for XMITQ(s) are as follows:
- If a message has persistance set to NONE then it stays as NONE regardless of the XMITQ persistance setting.
- If a message has persistance set to YES then it stays as YES ONLY if the XMITQ also has persistance set to YES. The MCA has a right to discard ANY non-persistance messages on restart!!!!!
Weird but true.
later
Roger... |
|
| Back to top |
|
|
| Sunntu |
| Posted: Tue Aug 06, 2002 3:32 am Post subject: Disappearing messages |
|
|
Novice
Joined: 04 Aug 2002
Posts: 11
|
Jeff,
When I tried giving the runmqsc command from the AIX side, I got the message "Object not found". It may have to do with access rights to access queue managers between the two platforms.
Poki,
No, I am using MQ without CICS
Roger,
I set the persistence to YES and still the result was the same. If by any chance my application program places messages when the channel is stopped then I am doomed. When I restart the channel something vague happens. Some messages disappear and some stay on the XMITQ. And until I clear the XMITQ and restart the channel after that nothing works.
I have two questions:
1. Is it advisable to keep the disc interval 0 so that the channel runs always. (I have already seen the posts and I want to know if there are any more pros/cons in using the same)
2. Next question, is it possible for an application program to check the channel status ?
I am not giving up as yet. I am looking at what else may be wrong.
Thanks all,
Sunntu |
|
| Back to top |
|
|
| mrlinux |
| Posted: Tue Aug 06, 2002 4:48 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
Do you have the queue managers configured to use a dead letter queue,
on AIX
runmqsc
dis qmgr DEADQ
on OS390
1 Display Manager
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| jc_squire |
| Posted: Tue Aug 06, 2002 7:31 pm Post subject: |
|
|
Centurion
Joined: 14 Apr 2002
Posts: 105
Location: New Zealand
|
Anything in the error log files?
Regards
_________________
J C Squire
IBM Certified Specialist - MQSeries |
|
| Back to top |
|
|
| Sunntu |
| Posted: Tue Aug 06, 2002 8:57 pm Post subject: Disappearing messages |
|
|
Novice
Joined: 04 Aug 2002
Posts: 11
|
Hi,
This is what was logged in the error log on the AIX side.
08/07/02 05:23:10
AMQ9213: A communications error for TCP/IP occurred.
EXPLANATION:
An unexpected error occurred in communications.
ACTION:
The return code from the TCP/IP (select) [TIMEOUT] call was 11 (X'B'). Record
these values and tell the systems administrator.
On the OS390 side, this is what I get:
CSQX209E ! CSQXRCTL Connection unexpectedly terminated,
channel RPM.CHL,
connection nnn.nn.n.nn,
TRPTYPE=TCP
CSQX599E ! CSQXRCTL Channel RPM.CHL ended abnormally
Note:I have replaced our IP and Channel name etc. 'cos of my company policies.
I am not sure if this of any help, on the AIX side, its a qmgr cluster. QM1 is the main QMGR and it distributes messages to queues residing on QM2 and QM3.
Rgds,
Sundari |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Tue Aug 06, 2002 9:02 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3253
Location: London, ON Canada
|
(Update - I was typing this posting as you posted your message, so this may change things). 
No offence Sunntu, but I smell something fishy here. I have worked for several financial institutions over the years doing regression testing and if a persistent message is put onto a XMITQ, it will never be dropped (short of the queue manager being fried).
- Is this channel / XMITQ dedicated for your testing?
- When you start the channel, is the XMITQ emptied? (in a timely fashion)
- How are you putting the message onto the XMITQ? e.g. remote queue definition or opening the remote directly with both queue and queue manager name or are you attempting to put the message directly onto the XMITQ?
- Do you have a dead letter queue defined?
- If so, are there any messages in the queue
- If not, WHY NOT?
- How are you verifying that the message(s) on the XMITQ are persistent?
- Do any of the messages have an expiry value set?
Also, "Object not found" message is NOT related to access rights because you would have received a "not authorized" message.
You should provide us with the queue defintions for both AIX and OS/390. Because the actions you are taking and the things you are saying, don't add up.
Yes an application can get the channel status but again WHY? It should be up to your MQ Admin to have a monitoring tool in-place to make sure that the appropriate people are alerted when a channel goes down.
later
Roger... |
|
| Back to top |
|
|
| Sunntu |
| Posted: Tue Aug 06, 2002 9:38 pm Post subject: |
|
|
Novice
Joined: 04 Aug 2002
Posts: 11
|
Roger,
No offence taken. Thanks for anything I am getting here. This whole things really perturbs me. Here are my replies:
- Is this channel / XMITQ dedicated for your testing?
Yes
- When you start the channel, is the XMITQ emptied? (in a timely fashion)
I didnt get u'r question, when I start the channel, the messages take sometime to leave the XMITQ. If the messages were persistent, then none of the messages reach the remote queues. If the messages were non-persistent then just two messages reach. (The first of the two in the XMITQ). One each in qmgrs QM2 and QM3.
Whenever I do this test, something goes wrong. Even after I start the channel, the same things repeats.
- How are you putting the message onto the XMITQ? e.g. remote queue definition or opening the remote directly with both queue and queue manager name or are you attempting to put the message directly onto the XMITQ?
I use remote queue definition
- Do you have a dead letter queue defined? Yes
- If so, are there any messages in the queue
Some are there. But they have been there even before I started my testing.
- If not, WHY NOT?
I have no idea.
- How are you verifying that the message(s) on the XMITQ are persistent?
I am not verifying in the XMITQ. I just assume they are persistent since I set the message persistence at the programming level.
- Do any of the messages have an expiry value set?
I am explicitly setting it in my app to MQEI_UNLIMITED.
Also, "Object not found" message is NOT related to access rights because you would have received a "not authorized" message.
Sorry, what I get is not object not found, what i get is this:
AMQ8146: MQSeries queue manager not available.
The channel status is my last resort if I am unable to solve this problem. There is still time for that though.
-- Sunntu |
|
| Back to top |
|
|
| jc_squire |
| Posted: Wed Aug 07, 2002 1:24 pm Post subject: |
|
|
Centurion
Joined: 14 Apr 2002
Posts: 105
Location: New Zealand
|
Hi,
Ok, lets see if I got this right. Your OS390 box is not in the cluster, your cluster consists of QM1, QM2 and QM3 and QM1 acts as a "gateway" to the cluster for messages from the OS390 server. Messages appear to be going missing between OS390 and QM1. Your error logs indicate that you have a tcp/ip problem.
We have had a similar problem before. I think the messages that you are missing might reappear on your xmitq if you backout the chl. Your chls/messages could be indoubt i.e. chls start and run, comms is dirupted but because the tcp/ip timeout is set to 300 seconds (every 5 minutes the queue manager checks if the chl is still alive) by default the chls still appear to be running, OS390 qmgr sends messages and waits for QM1 to commit which it can't because there is no connection. This leaves chl/msgs indoubt but this will not neccessarily appear in your error logs - a big pain in the butt !!!!!!! Not sure how you do it on OS390 but on other platforms it is "resolve chl(......) action(backout)". You might also have to reset sequence numbers on both ends of the chl "reset chl(......) seqnumber(....)". If you want you can also reduce your tcp/ip timeout which is twice that of your HBINT value on the chl definition - this will force the queue manager to check the status of the connection more frequently.
Can you discribe your test plan and what steps you are taking? How many chls are "bound" to the xmitq? What version of MQ are you running?
Suggest you try the following:
- Get disable Q's on QM2 & QM3
- Resolve chl from OS390 to QM1
- At this stage it is not confirmed where messages are going missing so you might have to resolve the chls from QM1 to QM2 & QM3
- Reset sequence nrs on chls
- Stop your sender chl on OS390 (this will also stop the triggering).
- Ensure OS390 xmitq and DLQ on all qmgrs are clear.
- Once sender is stopped put a X nr of messages to the RQ.
- Check your xmitq to confirm that X nr of messages are on the queue.
- XMITQ attributes should be GET(DISABLED) and DEFPSIST(YES).
- Browse the queue and check the MD for persistent attribute.
- Check the DLQ
- Stop the clusrcvr chls on QM2 & QM3
- Ensure the SYSTEM.CLUSTER.TRANSMIT.QUEUE on QM1 is empty
- Start the sdr chl on OS390
- Check chl status on OS390 and QM1
- Check xmitq qdepth on OS390
- Check DLQ on OS390 and QM1
- Check error logs on OS390 and QM1
- Check xmitq and SYSTEM.CLUSTER.TRANSMIT.QUEUE and verify that X nr of messages have left xmitq and are on SYSTEM.CLUSTER.TRANSMIT.QUEUE
- Start clusrcvr chls on QM2 and QM3
- Confirm SYSTEM.CLUSTER.TRANSMIT.QUEUE is cleared and that X nr of messages are on QM2 and QM3.
- Check errors logs and DLQ on QM1, QM2 & QM3
Try to get your network guys to monitor the network while this is happening. Look forward to hear the results and hope this helps .
Regards
_________________
J C Squire
IBM Certified Specialist - MQSeries |
|
| Back to top |
|
|
|
|
