| Author |
Message
|
| Navya |
 Posted: Thu Feb 03, 2011 8:09 am Post subject: Messages stuck in Transmission Queue |
 |
|
Novice
Joined: 03 Feb 2011
Posts: 15
|
We have a Queue manager in windows 2000 server, from which the messages are being sent to unix server.
The messages are not being transferred to Unix server suddenly from Yesterday, all the messages are stuck in Transmission queue.
No changes have been done and MQ V6.0 is running on the server from long time.
In the source host Error logs are updated with Source log file attached.
2/2/2011 01:37:38 - Process(6852.1) User(MUSR_MQADMIN) Program(runmqchl.exe)
AMQ9206: Error sending data to host (XXXX).
EXPLANATION:
An error occurred sending data over TCP/IP to (XXX). This may be
due to a communications failure.
ACTION:
The return code from the TCP/IP(send) call was 10053 X('2745'). Record these
values and tell your systems administrator.
The destination server logs are updated with errors - attached in destination log.
A connection from host 'xxx' over TCP/IP timed out.
ACTION:
Check to see why data was not received in the expected time. Correct the
problem. Reconnect the channel, or wait for a retrying channel to reconnect
itself.
----- amqccita.c : 3546 -------------------------------------------------------
02/02/11 13:48:28 - Process(442830.1048) User(mqm) Program(amqrmppa)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'xx.TO.yy' ended abnormally.
ACTION:
Look at previous error messages for channel program 'xx.TO.yy' in
the error files to determine the cause of the failure.
----- amqrmrsa.c : 468 --------------------------------------------------------
02/02/11 13:51:45 - Process(442830.1017) User(mqm) Program(amqrmppa)
AMQ9208: Error on receive from host (yyyy).
EXPLANATION:
An error occurred receiving data from (xxxx) over TCP/IP. This may
be due to a communications failure.
ACTION:
The return code from the TCP/IP (read) call was 73 (X'49'). Record these values
and tell the systems administrator.
and there is a FDC created at Destination.
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Tuesday February 01 12:38:55 GMT 2011 |
| Host Name :- xxx (AIX 5.3) |
| PIDS :- 5724H7201 |
| LVLS :- 6.0.2.4 |
| Product Long Name :- WebSphere MQ for AIX |
| Vendor :- IBM |
| Probe Id :- CO052000 |
| Application Name :- MQM |
| Component :- cciTcpReceive |
| SCCS Info :- lib/comms/amqccita.c, 1.255.1.32 |
| Line Number :- 3437 |
| Build Date :- May 12 2008 |
| CMVC level :- p600-204-080509 |
| Build Type :- IKAP - (Production) |
| UserID :- 00000461 (mqm) |
| Program Name :- amqrmppa |
| Addressing mode :- 64-bit |
| Process :- 442830 |
| Thread :- 722 |
| Last HQC :- 0.0.0-0 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- rrcE_BAD_DATA_RECEIVED |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ9207 |
| Probe Severity :- 2 |
| Probe Description :- AMQ9207: The data received from host '172 |
| xxx' is not valid. |
| FDCSequenceNumber :- 0 |
| Comment1 :- xxx |
| Comment2 :- TCP/IP |
| Comment3 :- |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
ccxResponder
rrxResponder
rriAcceptSess
ccxReceive
cciTcpReceive
xcsFFST
MQM Trace History
{ xppInitialiseDestructorRegistrations
} xppInitialiseDestructorRegistrations rc=OK
{ xcsInitialize
-{ xcsRequestThreadMutexSem
-} xcsRequestThreadMutexSem rc=OK
-{ xcsIsEnvironment
-} xcsIsEnvironment rc=OK
-{ xcsGetEnvironmentString
-} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
-{ xcsReleaseThreadMutexSem
-} xcsReleaseThreadMutexSem rc=OK
} xcsInitialize rc=OK
{ cccGetMem
-{ xcsGetMem
-} xcsGetMem rc=OK
} cccGetMem rc=OK
{ cccCSRegister
-{ xcsRegComp
-} xcsRegComp rc=OK
} cccCSRegister rc=OK
{ xcsRequestThreadMutexSem
} xcsRequestThreadMutexSem rc=OK
{ xcsReleaseThreadMutexSem
} xcsReleaseThreadMutexSem rc=OK
{ ccxResponder
-{ ccxNetWorkInit
--{ ccxParseIni
--} ccxParseIni rc=OK
--{ cciLoadLibrary
---{ cccGetMem
----{ xcsGetMem
----} xcsGetMem rc=OK
---} cccGetMem rc=OK
---{ xcsLoadFunction
----{ xcsQueryValueForSubpool
----} xcsQueryValueForSubpool rc=OK
---} xcsLoadFunction rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
---} xcsQueryProcAddr rc=OK
---{ xcsQueryProcAddr
-{ xcsRequestThreadMutexSem
--} xcsRequestThreadMutexSem rc=OK
--{ xcsReleaseThreadMutexSem
--} xcsReleaseThreadMutexSem rc=OK
-} ccxNetWorkInit rc=OK
-{ ccxAcceptConv
--{ cciTcpAcceptConv
---{ ioctl
---} ioctl rc=OK
---{ cciTcpResolveAddress
---} cciTcpResolveAddress rc=OK
---{ cciTcpResolveAddress
---} cciTcpResolveAddress rc=OK
--} cciTcpAcceptConv rc=OK
-} ccxAcceptConv rc=OK
-{ rrxResponder
--{ cccGetConfig
--} cccGetConfig rc=OK
--{ rriAcceptSess
---{ cccGetMem
----{ xcsGetMem
----} xcsGetMem rc=OK
---} cccGetMem rc=OK
---{ ccxSetTimeOut
---} ccxSetTimeOut rc=OK
---{ ccxReceive
----{ cciTcpReceive
-----{ ccxAllocMem
------{ cccGetMem
-------{ xcsGetMem
-------} xcsGetMem rc=OK
------} cccGetMem rc=OK
-----} ccxAllocMem rc=OK
-----{ recv
-----} recv rc=Unknown(FFFF)
-----{ xcsWaitFd
------{ poll
------} poll rc=Unknown(1)
-----} xcsWaitFd rc=Unknown(1)
-----{ recv
-----} recv rc=Unknown(200)
-----{ xcsSleep
-----} xcsSleep rc=OK
-----{ ioctl
-----} ioctl rc=OK
-----{ recv
-----} recv rc=Unknown(96)
-----{ cciTcpGetNameandAddress
------{ cciTcpResolveHostname
------} cciTcpResolveHostname rc=OK
------{ cciTcpResolveAddress
------} cciTcpResolveAddress rc=OK
-----} cciTcpGetNameandAddress rc=OK
-----{ rrxError
-----} rrxError rc=rrcE_BAD_DATA_RECEIVED
-----{ xcsBuildDumpPtr
------{ xcsGetMem
-----{ xcsFFST
Dump of Transmission Segment Header
110574b50 47455420 2F204854 GET / HT
110574b60 54502F31 2E310D0A 41636365 70743A20 TP/1.1..Accept:
110574b70 6170706C 69636174 696F6E2F 766E642E application/vnd.
110574b80 6D732D65 7863656C 2C206170 706C6963 ms-excel, applic
110574b90 6174696F 6E2F766E 642E6D73 2D706F77 ation/vnd.ms-pow
110574ba0 6572706F 696E742C 20617070 6C696361 erpoint, applica
110574bb0 74696F6E 2F6D7377 6F72642C 20696D61 tion/msword, ima
110574bc0 67652F67 69662C20 696D6167 652F782D ge/gif, image/x-
110574bd0 78626974 6D61702C 20696D61 67652F6A xbitmap, image/j
110574be0 7065672C 20696D61 67652F70 6A706567 peg, image/pjpeg
110574bf0 2C206170 706C6963 6174696F 6E2F7861 , application/xa
110574c00 6D6C2B78 6D6C2C20 6170706C 69636174 ml+xml, applicat
110574c10 696F6E2F 766E642E 6D732D78 7073646F ion/vnd.ms-xpsdo
110574c20 63756D65 6E742C20 6170706C 69636174 cument, applicat
110574c30 696F6E2F 782D6D73 2D786261 702C2061 ion/x-ms-xbap, a
110574c40 70706C69 63617469 6F6E2F78 2D6D732D pplication/x-ms-
110574c50 6170706C 69636174 696F6E2C 20617070 application, app
110574c60 6C696361 74696F6E 2F782D73 686F636B lication/x-shock
110574c70 77617665 2D666C61 73682C20 2A2F2A0D wave-flash, */*.
110574c80 0A416363 6570742D 4C616E67 75616765 .Accept-Language
110574c90 3A207A68 2D686B0D 0A55412D 4350553A : zh-hk..UA-CPU:
110574ca0 20783836 0D0A4163 63657074 2D456E63 x86..Accept-Enc
110574cb0 6F64696E 673A2067 7A69702C 20646566 oding: gzip, def
110574cc0 6C617465 0D0A5573 65722D41 67656E74 late..User-Agent
110574cd0 3A204D6F 7A696C6C 612F342E 30202863 : Mozilla/4.0 (c
110574ce0 6F6D7061 7469626C 653B204D 53494520 ompatible; MSIE
110574cf0 372E303B 2057696E 646F7773 204E5420 7.0; Windows NT
110574d00 352E313B 20496E66 6F506174 682E323B 5.1; InfoPath.2;
110574d10 202E4E45 5420434C 5220322E 302E3530 .NET CLR 2.0.50
110574d20 3732373B 202E4E45 5420434C 5220312E 727; .NET CLR 1.
110574d30 312E3433 32323B20 2E4E4554 20434C52 1.4322; .NET CLR
110574d40 20332E30 2E303435 30362E33 303B202E 3.0.04506.30; .
110574d50 4E455420 434C5220 332E302E 34353036 NET CLR 3.0.4506
110574d60 2E323135 323B202E 4E455420 434C5220 .2152; .NET CLR
110574d70 332E352E 33303732 39290D0A 486F7374 3.5.30729)..Host
110574d80 3A203137 322E3136 2E31322E 38363A31 : 172.16.12.86:1
110574d90 34313532 0D0A4361 6368652D 436F6E74 4152..Cache-Cont
110574da0 726F6C3A 206D6178 2D737461 6C653D30 rol: max-stale=0
110574db0 0D0A436F 6E6E6563 74696F6E 3A204B65 ..Connection: Ke
110574dc0 65702D41 6C697665 0D0A582D 426C7565 ep-Alive..X-Blue
110574dd0 436F6174 2D566961 3A204431 32343533 Coat-Via: D12453
110574de0 38313738 34333035 37460D0A 0D0A 817843057F....
1) I have restarted the channel - which didn't solve the problem.
2) Moved the messages from Transmission queue to a TEST queue.
3) Rebooted the queue manager, added the Environment variable MQNOREMPOOL.
4) Rebooted the server.
But the messages are stil stuck in the transmission queue, can you please suggest any soultion ? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Feb 03, 2011 8:18 am Post subject: Re: Messages stuck in Transmission Queue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Navya wrote: |
| can you please suggest any soultion ? |
Get your network people to fix the network issue (the one causing the 10053 error).
2 other points:
- Please use code tags when posting output like that
- Don't post the whole FDC; it's only useful to IBM. The box at the top contains everything normal mortals can use.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Navya |
| Posted: Thu Feb 03, 2011 9:25 am Post subject: |
|
|
Novice
Joined: 03 Feb 2011
Posts: 15
|
Hi,
Thanks for your reply.
But i am able to ping the server. and Telnet with the MQ port from Source server.
So network team says no issues from their end. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 03, 2011 9:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Navya wrote: |
| So network team says no issues from their end. |
They always say that. Even if the CAT 5 isn't actually plugged into the router. 
Something is causing that 10053 - ask the network team what non-issue could be causing that.
When you telnet to the receiving port does the MCA report an error (as it should) and does the connection hold long enough for you to type into the telnet screen?
What version of WMQv6 are you using?
What Unix are you using?
What network changes didn't the network team make between it working and it not working? Because when there's a network issue it's a sure but that not only is it not a network issue but nothing's changed in the network.
Until a few days later they admit to "this tiny change that has nothing to do with anything you're using and wouldn't cause this" but when reversed causes your channels to miraculously restart.
What do the receiver channel logs say?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Navya |
| Posted: Thu Feb 03, 2011 10:19 am Post subject: |
|
|
Novice
Joined: 03 Feb 2011
Posts: 15
|
MQ Version : 6.0.2.1
And Telnet is coming up a with blank screen and I can't see any errors at the receiver end.
We are using AIX.
I will check with Networks if there are any recent changes.
And i am unable to understand the reason behind the Destionation queue manager's FDC. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 03, 2011 10:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Navya wrote: |
| MQ Version : 6.0.2.1 |
That's very old.
| Navya wrote: |
| And Telnet is coming up a with blank screen and I can't see any errors at the receiver end. |
If you try into the blank screen you should get receiver errors, because you're not a sender MCA. This would prove the receiver MCA is properly responsive.
| Navya wrote: |
| And i am unable to understand the reason behind the Destionation queue manager's FDC. |
I missed that the FDC was from the destination queue manager not the source. That's the FDC you should get if you type random stuff into the telnet screen. It means that something other than the sender MCA is sending data to the WMQ port on the receiver box. Like another application, or a network utility.
The same sort of thing that would cause the sender to fail with a 10053.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Feb 03, 2011 10:27 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
A quick search of google for Probe Id CO052000 will offer some ideas.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 03, 2011 10:28 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The FDC on the receiver side is because you telneted to the listener port and didn't enter in a proper MQ channel handshake (because you are a human, not a queue manager). Or someone else did.
If the telnet is functioning now, then start the channel again. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Feb 03, 2011 11:21 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Google's an amazing resource...
| Quote: |
Question/Problem: WSAECONNABORTED (10053) Software caused connection abort.
Answer/Solution: A connection abort was caused internal to your host machine. The software caused a connection abort because there is no space on the socket's queue and the socket cannot receive further connections.
WinSock description: The error can occur when the local network system aborts a connection. This would occur if WinSock aborts an established connection after data retransmission fails (receiver never acknowledges data sent on a datastream socket).
TCP/IP scenario: A connection will timeout if the local system doesn't receive an (ACK)nowledgement for data sent. It would also timeout if a (FIN)ish TCP packet is not ACK'd (and even if the FIN is ACK'd, it will eventually timeout if a FIN is not returned). |
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Navya |
| Posted: Fri Feb 04, 2011 7:37 am Post subject: |
|
|
Novice
Joined: 03 Feb 2011
Posts: 15
|
| Just came to know that we are able to transfer messages of small size like 1KB upto 18KB but when i tried 100KB it is not getting transferred again. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 04, 2011 8:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Navya wrote: |
| Just came to know that we are able to transfer messages of small size like 1KB upto 18KB but when i tried 100KB it is not getting transferred again. |
What did your network people say about the 10053?
Do you have any packet shaping technology that could be rejecting larger messages?
(I used the term guardedly - 100KB is not "large")
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Feb 04, 2011 8:19 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
This also sounds a bit like a known issue with 6.0.2.4, at least in the back of my brain that fails to forget things.
So I'd review the fixes since 6.0.2.4 to the current level of 6.0.2.x or consider migrating to v7. |
|
| Back to top |
|
|
| Navya |
| Posted: Fri Feb 04, 2011 8:25 am Post subject: |
|
|
Novice
Joined: 03 Feb 2011
Posts: 15
|
Hi Vitor,
I can't see 10053 or any other channel disconnect errors when there are no messages in the channel or when the message size is very less.
I think if the message is just above 100 it is taking long time to transfer and by the time the receiver channel is getting timed out causing the abnormal end of Channel.
so, do you think changing the channel parameters could help ? 
Thanks,
Navya.
[/quote] |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 04, 2011 8:34 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Navya wrote: |
| I think if the message is just above 100 it is taking long time to transfer and by the time the receiver channel is getting timed out causing the abnormal end of Channel. |
I think that's plausible.
| Navya wrote: |
| so, do you think changing the channel parameters could help ? |
I don't see that it would. You might be able to change something in the network level that would prevent the TCP/IP resetting (which is the actual problem).
I'd also seriously think about applying maintenance. I mentioned above that the 2 versions you've quoted are old (6.0.2.1/4) and if mqjeff has a faint memory of it being fixed in a later version that's strong enough to go with.
You could also raise a PMR if you just want the specific fix (if any) to this problem, but why not come up to the most recent version?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Feb 04, 2011 9:11 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Navya wrote: |
MQ Version : 6.0.2.1
And Telnet is coming up a with blank screen and I can't see any errors at the receiver end.
We are using AIX.
I will check with Networks if there are any recent changes.
And i am unable to understand the reason behind the Destionation queue manager's FDC. |
Not quite what your FDC says:
And keep in mind that before 6.0.2.8 there were also some errors with message compression where a channel would be in running status but the messages were not being transmitted. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
