| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| MQ Server 8.0.0.1 command server timeout? |
View previous topic :: View next topic |
| Author |
Message
|
| LouML |
 Posted: Thu Feb 05, 2015 3:48 am Post subject: MQ Server 8.0.0.1 command server timeout? |
 |
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
We seem to have an issue where the command server may be timing out.
We’re based in the Northeast. All of our MQ servers are local except one in the UK.
We’ve recently upgraded our MQ environment from Solaris zones running MQ Server 7.5.0.2 (US) and 7.0.1.? (UK) to Linux VM’s running MQ Server 8.0.0.1 in both the US & UK. We’re also running ITM 6.2.3 on all MQ servers.
Since the upgrade, we have an issue where it seems that the command server may be timing out to the server in the UK only. This leads me to believe we may have a networking issue to the new server in the UK. However, I’m being told to open a PMR with IBM. Before I do so, I want to explore all possibilities.
We are noticing the following:
1 – We have a perl script running locally that does a channel status on all remote queue managers. It uses MO72 mqsc as a client and logs the data on an administration server. This works fine most of the time. A few times a day, we get messages on the QMUK.DEAD.LETTER.QUEUE queue. The dead letter header shows the reason as MQRC_UNKNOWN_OBJECT_NAME. The data shows that the Unknown Object is actually a temporary queue used by the MO72 mqsc call (MQMON.nobody.54CD7AA9203F15D4). You can see my runmqsc command
| Code: |
00000 44 4C 48 20 01 00 00 00--25 08 00 00 4D 51 4D 4F |DLH ....%...MQMO|
00010 4E 2E 6E 6F 62 6F 64 79--2E 35 34 43 44 37 41 41 |N.nobody.54CD7AA|
00020 39 32 30 33 46 31 35 44--34 20 20 20 20 20 20 20 |9203F15D4 |
00030 20 20 20 20 20 20 20 20--20 20 20 20 51 4D 55 4B | QMUK|
00040 20 20 20 20 20 20 20 20--20 20 20 20 20 20 20 20 | |
00050 20 20 20 20 20 20 20 20--20 20 20 20 20 20 20 20 | |
00060 20 20 20 20 20 20 20 20--20 20 20 20 22 02 00 00 | "...|
00070 33 03 00 00 4D 51 41 44--4D 49 4E 20 06 00 00 00 |3...MQADMIN ....|
00080 6D 71 73 63 20 20 20 20--20 20 20 20 20 20 20 20 |mqsc |
00090 20 20 20 20 20 20 20 20--20 20 20 20 32 30 31 35 | 2015|
000A0 30 32 30 35 30 39 30 37--31 31 31 35 01 00 00 00 |020509071115....|
000B0 24 00 00 00 01 00 00 00--26 00 00 00 01 00 00 00 |$.......&.......|
000C0 01 00 00 00 00 00 00 00--00 00 00 00 02 00 00 00 |................|
000D0 03 00 00 00 10 00 00 00--F9 03 00 00 01 00 00 00 |........ù.......|
000E0 04 00 00 00 40 00 00 00--C6 0B 00 00 33 03 00 00 |....@...Æ..3...|
000F0 29 00 00 00 64 69 73 20--63 68 73 28 2A 29 20 61 |)...dis chs(*) a|
00100 6C 6C 20 77 68 65 72 65--20 28 63 68 6C 74 79 70 |ll where (chltyp|
00110 65 20 6E 65 20 73 76 72--63 6F 6E 6E 29 00 00 00 |e ne svrconn)...|
|
2 - There are also messages on the MQ explorer temporary queue (AMQ.MQEXPLORER.1967055720).
| Code: |
00000 02 00 00 00 24 00 00 00--01 00 00 00 A1 00 00 00 |....$.......¡...|
00010 01 00 00 00 01 00 00 00--00 00 00 00 00 00 00 00 |................|
00020 10 00 00 00 04 00 00 00--44 00 00 00 DF 07 00 00 |........D...ß...|
00030 00 00 00 00 30 00 00 00--51 4D 55 4B 20 20 20 20 |....0...QMUK |
00040 20 20 20 20 20 20 20 20--20 20 20 20 20 20 20 20 | |
00050 20 20 20 20 20 20 20 20--20 20 20 20 20 20 20 20 | |
00060 20 20 20 20 20 20 20 20--03 00 00 00 10 00 00 00 | ........|
00070 7D 04 00 00 02 00 00 00--03 00 00 00 10 00 00 00 |}...............|
00080 CE 04 00 00 4A 00 00 00--03 00 00 00 10 00 00 00 |Î...J...........|
00090 D1 04 00 00 02 00 00 00--03 00 00 00 10 00 00 00 |Ñ...............|
000A0 D0 04 00 00 02 00 00 00--04 00 00 00 24 00 00 00 |Ð...........$...|
000B0 44 08 00 00 00 00 00 00--0D 00 00 00 49 6E 73 74 |D..........Inst|
000C0 61 6C 6C 61 74 69 6F 6E--31 00 00 00 04 00 00 00 |allation1.......|
000D0 1C 00 00 00 45 08 00 00--00 00 00 00 08 00 00 00 | ...E...........|
000E0 2F 6F 70 74 2F 6D 71 6D--04 00 00 00 14 00 00 00 |/opt/mqm........|
000F0 43 08 00 00 00 00 00 00--00 00 00 00 03 00 00 00 |C...............|
00100 10 00 00 00 81 05 00 00--00 00 00 00 03 00 00 00 |................|
00110 10 00 00 00 2D 05 00 00--00 00 00 00 04 00 00 00 |....-...........|
00120 38 00 00 00 02 0C 00 00--00 00 00 00 23 00 00 00 |8.... ......#...|
00130 2F 76 61 72 2F 6D 71 6D--2F 6C 6F 67 2F 51 4D 55 |/var/mqm/log/QMU|
00140 4B 52 45 53 54 47 57 31--21 30 31 2F 61 63 74 69 |K/active/.......|
00150 76 65 2F 00 04 00 00 00--14 00 00 00 FF 0B 00 00 |............ÿ..|
00160 00 00 00 00 00 00 00 00--04 00 00 00 14 00 00 00 |................|
00170 00 0C 00 00 00 00 00 00--00 00 00 00 04 00 00 00 |...............|
00180 14 00 00 00 01 0C 00 00--00 00 00 00 00 00 00 00 |...............|
00190 04 00 00 00 20 00 00 00--67 0C 00 00 00 00 00 00 |.... ...g......|
001A0 0C 00 00 00 32 30 31 35--2D 30 32 2D 30 31 20 20 |...2015-02-01 |
001B0 04 00 00 00 1C 00 00 00--68 0C 00 00 00 00 00 00 |.... ...h......|
001C0 08 00 00 00 30 31 2E 30--30 2E 32 35 |....01.00.25 |
|
3 – I’m not sure if this is related to #1 & #2 but since it only happens on the UK server, I suspect it might be. I see the following in the MQ error log on the UK server (yet I never see the queue full):
| Code: |
-------------------------------------------------------------------------------
01/29/2015 12:55:02 PM - Process(8841.603) User(mqm) Program(amqzlaa0)
Host(MY.COMPANY.com) Installation(Installation1)
VRMF(8.0.0.1) QMgr(QMUK)
AMQ7305: Trigger message could not be put on an initiation queue.
EXPLANATION:
The attempt to put a trigger message on queue SYSTEM.CHANNEL.INITQ on queue manager QMUK failed with reason code 2053. The message will be put on the dead-letter queue.
ACTION:
Ensure that the initiation queue is available, and operational.
----- amqktrga.c : 442 --------------------------------------------------------
01/29/2015 12:55:02 PM - Process(8841.603) User(mqm) Program(amqzlaa0)
Host(MY.COMPANY.com) Installation(Installation1)
VRMF(8.0.0.1) QMgr(QMUK)
AMQ7305: Trigger message could not be put on an initiation queue.
EXPLANATION:
The attempt to put a trigger message on queue SYSTEM.CHANNEL.INITQ on queue manager QMUK failed with reason code 2053. The message will be put on the dead-letter queue.
ACTION:
Ensure that the initiation queue is available, and operational.
----- amqktrga.c : 442 --------------------------------------------------------
|
4 – We also seem to get this issue with the ITM agents timing out to the UK server.
Some random notes:
– The new Linux server in the UK is on the same segment as the old Solaris server. We did not have the issue before.
– We do not have any issue with the new local Linux servers.
– As of now, we do not seem to have the issue with the UK Dev server (although we just started ITM to that server yesterday).
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude
Last edited by LouML on Thu Feb 05, 2015 6:28 am; edited 1 time in total |
|
| Back to top |
|
 |
| PaulClarke |
| Posted: Thu Feb 05, 2015 3:56 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
To get 2053 (MQRC_Q_FULL) on a trigger queue (SYSTEM.CHANNEL.INITQ) is rather concerning. Do you really have lots of messages on this queue or is there something wrong with your configuration. ie. your maxdepth is not set to a silly value like 5. Also check that you are not using TRIGTYPE(FIRST) for your transmission queues.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| LouML |
| Posted: Thu Feb 05, 2015 4:15 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| PaulClarke wrote: |
To get 2053 (MQRC_Q_FULL) on a trigger queue (SYSTEM.CHANNEL.INITQ) is rather concerning. Do you really have lots of messages on this queue or is there something wrong with your configuration. ie. your maxdepth is not set to a silly value like 5. Also check that you are not using TRIGTYPE(FIRST) for your transmission queues.
Cheers,
Paul. |
MAXDEPTH was initially set to 1000. I increased it to 5000 just to see if that helped, but it didn't. Again, I never actually see the queue full, or even have any messages.
We're using TRIGTYPE(FIRST) for all but on of the transmission queues (EVERY). These values have been set since before I joined this firm. I've seen this debated endlessly here. SO, EVERY is better than FIRST?
Actually, this issue happened only 5 times in one day and has not happened in almost a week, so maybe this is unrelated to the command server issue.
My main concern is the command server timeout.
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu Feb 05, 2015 4:30 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Channels are not designed to be used with TRIGTYPE(EVERY) use TRIGTYPE(FIRST) or, for very particular uses, TRIGTYPE(DEPTH).
Yes the command server issue is a strange one. I vaguely remember someone talking about something similar a week or two ago but I can't remember the outcome. Something about the command server sending more responses after it sent the last one or some such. There's just a possibility that it's related.
If this happens every time then it ought to be possible to capture it in an MQ trace and see what is deleting the reply queue. Of course it should only be the closing of the queue but you never know.
To be honest I don't fully understand what you are doing. I don't understand how you can issue an MQSC command, from MO71, via Perl. Unless you are starting MO71 for each command which seems an odd thing to do. Why not just use RUNMQSC or MQSCX ? I don't see why a GUI would be involved.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| LouML |
| Posted: Thu Feb 05, 2015 4:35 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| PaulClarke wrote: |
Channels are not designed to be used with TRIGTYPE(EVERY) use TRIGTYPE(FIRST) or, for very particular uses, TRIGTYPE(DEPTH).
Yes the command server issue is a strange one. I vaguely remember someone talking about something similar a week or two ago but I can't remember the outcome. Something about the command server sending more responses after it sent the last one or some such. There's just a possibility that it's related.
If this happens every time then it ought to be possible to capture it in an MQ trace and see what is deleting the reply queue. Of course it should only be the closing of the queue but you never know.
To be honest I don't fully understand what you are doing. I don't understand how you can issue an MQSC command, from MO71, via Perl. Unless you are starting MO71 for each command which seems an odd thing to do. Why not just use RUNMQSC or MQSCX ? I don't see why a GUI would be involved.
Cheers,
Paul. |
That's what happens when you try to post from memory. I meant MO72.
I run the following in a few different scripts to gather various pieces of information throughout the day.
/apps/mqm/etc/SupportPacs/mo72/LinuxIntel64/mqsc -m $qm -l -c $svrconn -h '$server($port)' -p format=no -p wait=120 -C $command`
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu Feb 05, 2015 5:18 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Ah ok....that makes a lot more sense. I guess I was also thrown a little since your reply queue is MQMON.*.
Well, as I said before this would seem to indicate that the queue is being closed before the command server sends the last response. An MQ trace seems like the way forward.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 05, 2015 5:28 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
A couple of notes.
I'm reasonably sure that the MQ v8 client includes a client-capable version of runmqsc. So it's worth double-checking that against MO72. Certainly, I'm sure that L2 will ask you to do the same.
Secondly, I remember a recentish thread (in the last two months) about the command server leaving messages on the DLQ, and I believe there wasn APAR. Some poking around might find something. Although it's possible that was on the vienna listserv, too. |
|
| Back to top |
|
|
| LouML |
| Posted: Thu Feb 05, 2015 5:50 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| mqjeff wrote: |
A couple of notes.
I'm reasonably sure that the MQ v8 client includes a client-capable version of runmqsc. So it's worth double-checking that against MO72. Certainly, I'm sure that L2 will ask you to do the same.
Secondly, I remember a recentish thread (in the last two months) about the command server leaving messages on the DLQ, and I believe there wasn APAR. Some poking around might find something. Although it's possible that was on the vienna listserv, too. |
Since this doesn't happen all the time it'd be hard to reproduce even with MO72 let alone runmqsc client mode.
We run these scripts from cron. I suppose I could copy a script and change it to use the runmqsc client mode. Then they would both run at approximately the same time. If it were some kind of network issue I should see it happen to both.
However, two facts remain:
1 - We're having timeout issues with other products (MQExplorer and ITM) as well.
2 - This only happens on a server that is a great distance away and not on the local servers.
I will look around for the thread.
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu Feb 05, 2015 6:03 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Well, I suppose it's always possible that MO72 just isn't waiting long enough for the reply.
I think, by default, it will wait 5 seconds. If all your channels are down and there's a lot of them etc then I guess that could be the problem. You could try increasing it to say 30 seconds and see whether the problem goes away. I think you have to say something like WANT=30 in the command parms. Or you can set an environment variable.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| LouML |
| Posted: Thu Feb 05, 2015 6:05 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| PaulClarke wrote: |
Ah ok....that makes a lot more sense. I guess I was also thrown a little since your reply queue is MQMON.*.
Well, as I said before this would seem to indicate that the queue is being closed before the command server sends the last response. An MQ trace seems like the way forward.
Cheers,
Paul. |
The queue closing before the command server reply makes sense for the messages going to dead letter queue.
However, the other issue is the messages on the AMQ.MQEXPLORER.* queue. That queue remains and has messages, instead of having them processed and going away.
And it supposedly happens with ITM and the temporary queue KMQ.* although I haven't yet captured a queue with messages.
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| LouML |
| Posted: Thu Feb 05, 2015 6:07 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| PaulClarke wrote: |
Well, I suppose it's always possible that MO72 just isn't waiting long enough for the reply.
I think, by default, it will wait 5 seconds. If all your channels are down and there's a lot of them etc then I guess that could be the problem. You could try increasing it to say 30 seconds and see whether the problem goes away. I think you have to say something like WANT=30 in the command parms. Or you can set an environment variable.
Cheers,
Paul. |
My initial WAIT time was 60 seconds. I doubled it to 120.
/apps/mqm/etc/SupportPacs/mo72/LinuxIntel64/mqsc -m $qm -l -c $svrconn -h '$server($port)' -p format=no -p wait=120 -C $command`
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 05, 2015 6:08 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
are you able to get a status on the svrconn instance when you see a timeout?
If it's retrying, that says it's a network issue. |
|
| Back to top |
|
|
| LouML |
| Posted: Thu Feb 05, 2015 6:10 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| mqjeff wrote: |
are you able to get a status on the svrconn instance when you see a timeout?
If it's retrying, that says it's a network issue. |
I haven't yet. I generally don't see the issue until it's too late. I need to discuss this with the ITM guy to see if he has any historical data that may shed some light.
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu Feb 05, 2015 6:11 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Oh yes, sorry.. I didn't notice you had specified a timeout.
Well, I guess what you could do it look at the times involved. If you know when MO72 started and when it ended and then look at the put time of the DLQ message you should be able to work out which of the two events happend first.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 05, 2015 6:21 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PaulClarke wrote: |
Oh yes, sorry.. I didn't notice you had specified a timeout.
Well, I guess what you could do it look at the times involved. If you know when MO72 started and when it ended and then look at the put time of the DLQ message you should be able to work out which of the two events happend first. |
That's making the assumption that the times are correctly synchronized between the server issuing the command and the server in the UK processing it... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
