| Author |
Message
|
| toshi1234 |
 Posted: Thu Jul 21, 2016 3:50 am Post subject: An error occurred receiving data AMQ9207 and AMQ9209 |
 |
|
Novice
Joined: 15 Dec 2015
Posts: 21
|
Hi all
We are having JMS/WAS application (Linux) who trying to put message in queue .
Some time back the JMS App could not able to put message in a queue;
When restarting the server connection channel all goes well but its occurring frequently no logs at application side.
07/20/2016 06:49:26 PM - Process(21301.6 User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9209: Connection to host '' closed.
EXPLANATION:
An error occurred receiving data from '' over TCP/IP. The connection
to the remote host has unexpectedly terminated.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3473 -------------------------------------------------------
07/20/2016 06:49:26 PM - Process(21301.69) User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9209: Connection to host '' closed.
EXPLANATION:
An error occurred receiving data from '' over TCP/IP. The connection
to the remote host has unexpectedly terminated.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3473 -------------------------------------------------------
07/20/2016 07:14:34 PM - Process(21301.8 User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9207: The data received from host '10 ()' is not valid.
EXPLANATION:
Incorrect data format received from host '10 ()' over TCP/IP. It
may be that an unknown host is attempting to send data. An FFST file has been
generated containing the invalid data received.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3620 -------------------------------------------------------
07/20/2016 07:14:34 PM - Process(21301.8 User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9492: The TCP/IP responder program encountered an error.
EXPLANATION:
The responder program was started but detected an error.
ACTION:
Look at previous error messages in the error files to determine the error
encountered by the responder program.
----- amqrmrsa.c : 524 --------------------------------------------------------
And FDC file I see
[img]
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- rrcE_BAD_DATA_RECEIVED |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ9207 |
| Probe Severity :- 2 |
| Probe Description :- AMQ9207: The data received from host '10 [/img]
for this i changed the SHARECNV property to 0 on server connection channel but same issue occurred recently.
Unable to find the RCA for this issue please share your suggestions it will be very helpful
Thanks.. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Jul 21, 2016 3:57 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Perhaps someone is trying to telnet to the MQ listener port.
Perhaps something is trying to open a TCP connection to the MQ listener port, to find out if the listener is open.
Perhaps something is doing a port scan.
Perhaps the FFST/FDC file has a slight bit of more information.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| toshi1234 |
| Posted: Thu Jul 21, 2016 5:12 am Post subject: |
|
|
Novice
Joined: 15 Dec 2015
Posts: 21
|
Hi mqjeff
Thanks for quick reply this is FDC file generated every time
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Wed July 20 2016 18:41:26 IST |
| UTC Time :- 1469020286.292926 |
| UTC Time Offset :- 330 (IST) |
| Host Name :- interface |
| Operating System :- Linux 2.6.32-573.el6.x86_64 |
| PIDS :- 5724H7230 |
| LVLS :- 7.5.0.5 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| Installation Path :- /opt/mqm |
| Installation Name :- Installation1 (1) |
| License Type :- Production |
| Probe Id :- CO603000 |
| Application Name :- MQM |
| Component :- cciInvalidDataReceived |
| SCCS Info :- /build/slot1/p750_P/src/lib/comms/amqccita.c, |
| Line Number :- 4295 |
| Build Date :- Apr 24 2015 |
| Build Level :- p750-005-150424 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 501 (mqm) |
| Real UserID :- 501 (mqm) |
| Program Name :- amqrmppa |
| Arguments :- -m "*** " |
| Addressing mode :- 64-bit |
| LANG :- en_US.UTF-8 |
| Process :- 125712 |
| Process(Thread) :- 42086 |
| Thread :- 253231 |
| ThreadingModel :- PosixThreads |
| UserApp :- FALSE |
| ConnId(1) IPCC :- 285577 |
| ConnId(3) QM-P :- 282334 |
| Last HQC :- 1.0.0-136584 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- rrcE_BAD_DATA_RECEIVED |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ9207 |
| Probe Severity :- 2 |
| Probe Description :- AMQ9207: The data received from host '*.*.*.*' on |
| channel '????' is not valid. |
| FDCSequenceNumber :- 0 |
| Comment1 :- |
| Comment2 :- TCP/IP |
| Comment3 :- ???? |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
ccxResponder
rrxResponder
rriAcceptSess
ccxReceive
cciTcpReceive
cciInvalidDataReceived
xcsFFST
MQM Trace History
-} xcsRequestThreadMutexSem rc=OK
-{ xcsConnectExistingSharedSubpool
--{ xstGetSubpoolsLock
---{ xstAccessSubpoolsLock
----{ xcsQueryValue
----} xcsQueryValue rc=OK
---} xstAccessSubpoolsLock rc=OK
Data: 0x01200026 0x00000001
--} xstGetSubpoolsLock rc=OK
Data: 0x00000001 0x00045b89 0x0003dd2f
--{ xcsAllocateQuickCell
--} xcsAllocateQuickCell rc=OK
--{ xstReleaseSubpoolsLock
---{ xstAccessSubpoolsLock
----{ xcsQueryValue
----} xcsQueryValue rc=OK
---} xstAccessSubpoolsLock rc=OK
--} xstReleaseSubpoolsLock rc=OK
--{ xlsThreadInitialization
---{ xlsThreadPreInitialization
----{ xcsAllocateQuickCell
----} xcsAllocateQuickCell rc=OK
----{ xcsQueryProcessDetails
----} xcsQueryProcessDetails rc=OK
---} xlsThreadPreInitialization rc=OK
--} xlsThreadInitialization rc=OK
--{ xcsConnectSharedMemSet
---{ xcsRequestThreadMutexSem
---} xcsRequestThreadMutexSem rc=OK
---{ xcsRequestThreadMutexSem
---} xcsRequestThreadMutexSem rc=OK
---{ xcsReleaseThreadMutexSem
---} xcsReleaseThreadMutexSem rc=OK
---{ xcsGetMemFn
---} xcsGetMemFn rc=OK
---{ xcsReleaseThreadMutexSem
---} xcsReleaseThreadMutexSem rc=OK
--} xcsConnectSharedMemSet rc=OK
-} xcsConnectExistingSharedSubpool rc=OK
-{ xcsGetEnvironmentString
-} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
-{ xcsReleaseThreadMutexSem
-} xcsReleaseThreadMutexSem rc=OK
} xcsInitialize rc=OK
{ ccxResponder
-{ cccGetConfig
--{ xcsRequestThreadMutexSem
--} xcsRequestThreadMutexSem rc=OK
--{ xcsReleaseThreadMutexSem
--} xcsReleaseThreadMutexSem rc=OK
-} cccGetConfig rc=OK
-{ ccxNetWorkInit
--{ ccxParseIni
--} ccxParseIni rc=OK
--{ cciLoadLibrary
---{ cccGetMem
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
---} cccGetMem rc=OK
--} cciLoadLibrary rc=OK
--{ xcsRequestThreadMutexSem
--} xcsRequestThreadMutexSem rc=OK
--{ xcsReleaseThreadMutexSem
--} xcsReleaseThreadMutexSem rc=OK
-} ccxNetWorkInit rc=OK
-{ ccxAcceptConv
--{ cciTcpAcceptConv
---{ ioctl
---} ioctl rc=OK
---{ cciTcpGetPeerName
----{ cciTcpResolveAddress
----} cciTcpResolveAddress rc=OK
---} cciTcpGetPeerName rc=OK
---{ cciTcpResolveAddress
---} cciTcpResolveAddress rc=OK
--} cciTcpAcceptConv rc=OK
--{ ccxCheckConvBlocked
---{ rsxAccessCache
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
----{ xcsConnectSharedMemSet
-----{ xcsRequestThreadMutexSem
-----} xcsRequestThreadMutexSem rc=OK
-----{ xcsRequestThreadMutexSem
-----} xcsRequestThreadMutexSem rc=OK
-----{ xcsReleaseThreadMutexSem
-----} xcsReleaseThreadMutexSem rc=OK
-----{ xcsGetMemFn
-----} xcsGetMemFn rc=OK
-----{ xcsReleaseThreadMutexSem
-----} xcsReleaseThreadMutexSem rc=OK
----} xcsConnectSharedMemSet rc=OK
----{ xcsRequestMutexSem
----} xcsRequestMutexSem rc=OK
----{ xcsReleaseMutexSem
----} xcsReleaseMutexSem rc=OK
---} rsxAccessCache rc=OK
---{ rsxReadLock
----{ xlsRWMutexRequest
-----{ xlsRWLockState
-----} xlsRWLockState rc=OK
-----{ xlsRWUnlockState
-----} xlsRWUnlockState rc=OK
----} xlsRWMutexRequest rc=OK
---} rsxReadLock rc=OK
---{ rsxIsAddressAllowed
----{ rsxReadUnlock
----} rsxReadUnlock rc=OK
---} rsxIsAddressAllowed rc=OK
---{ rsxReadUnlock
----{ xlsRWMutexRelease
-----{ xlsRWLockState
-----} xlsRWLockState rc=OK
-----{ xlsRWUnlockState
-----} xlsRWUnlockState rc=OK
----} xlsRWMutexRelease rc=OK
---} rsxReadUnlock rc=OK
---{ rsxReleaseCache
----{ xcsDisconnectSharedMemSet
-----{ xcsRequestThreadMutexSem
-----} xcsRequestThreadMutexSem rc=OK
-----{ xcsFreeMemFn
-----} xcsFreeMemFn rc=OK
-----{ xcsReleaseThreadMutexSem
-----} xcsReleaseThreadMutexSem rc=OK
----} xcsDisconnectSharedMemSet rc=OK
----{ xcsFreeMemFn
----} xcsFreeMemFn rc=OK
---} rsxReleaseCache rc=OK
--} ccxCheckConvBlocked rc=OK
-} ccxAcceptConv rc=OK
-{ rrxResponder
--{ cccGetConfig
--} cccGetConfig rc=OK
--{ rriAcceptSess
---{ cccGetMem
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
---} cccGetMem rc=OK
---{ xcsGetNextTraceIdent
----{ xcsRequestThreadMutexSem
----} xcsRequestThreadMutexSem rc=OK
----{ xcsReleaseThreadMutexSem
----} xcsReleaseThreadMutexSem rc=OK
---} xcsGetNextTraceIdent rc=OK
---{ xcsSetTraceIdent
---} xcsSetTraceIdent rc=OK
---{ ccxSetTimeOut
---} ccxSetTimeOut rc=OK
---{ cccGetMem
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
---} cccGetMem rc=OK
---{ ccxReceive
----{ cciTcpReceive
-----{ ccxAllocMem
------{ cccGetMem
-------{ xcsGetMemFn
-------} xcsGetMemFn rc=OK
------} cccGetMem rc=OK
-----} ccxAllocMem rc=OK
-----{ recv
-----} recv rc=Unknown(B)
Data: 0xffffffff
-----{ cciTcpWaitReceive
------{ xcsWaitFd
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=Unknown(1)
------} xcsWaitFd rc=Unknown(1)
-----} cciTcpWaitReceive rc=OK
-----{ recv
-----} recv rc=OK
Data: 0x00000005
-----{ recv
-----} recv rc=Unknown(B)
Data: 0xffffffff
-----{ cciTcpWaitReceive
------{ xcsWaitFd
-------{ poll
-------} poll rc=Unknown(1)
------} xcsWaitFd rc=Unknown(1)
-----} cciTcpWaitReceive rc=OK
-----{ recv
-----} recv rc=OK
Data: 0x00000001
-----{ recv
-----} recv rc=Unknown(B)
Data: 0xffffffff
-----{ cciTcpWaitReceive
------{ xcsWaitFd
-------{ poll
-------} poll rc=Unknown(1)
------} xcsWaitFd rc=Unknown(1)
-----} cciTcpWaitReceive rc=OK
-----{ recv
-----} recv rc=OK
Data: 0x00000005
-----{ cciBadDataFFSTCheck
------{ xcsIsEnvironment
------} xcsIsEnvironment rc=OK
-----} cciBadDataFFSTCheck rc=Unknown(1)
-----{ cciInvalidDataReceived
------{ xcsSleep
------} xcsSleep rc=OK
------{ ioctl
------} ioctl rc=OK
------{ recv
------} recv rc=OK
Data: 0x0000000c
------{ cciTcpGetNameandAddress
-------{ cciTcpResolveHostname
--------{ xcsIsEnvironment
--------} xcsIsEnvironment rc=OK
-------} cciTcpResolveHostname rc=OK
-------{ cciTcpResolveAddress
--------{ rrxError
--------} rrxError rc=rrcE_HOST_NOT_AVAILABLE
-------} cciTcpResolveAddress rc=rrcE_HOST_NOT_AVAILABLE
------} cciTcpGetNameandAddress rc=rrcE_HOST_NOT_AVAILABLE
------{ rrxError
------} rrxError rc=rrcE_BAD_DATA_RECEIVED
------{ xcsBuildDumpPtr
-------{ xcsGetMemFn
-------} xcsGetMemFn rc=OK
------} xcsBuildDumpPtr rc=OK
------{ xcsFFST
Dump of Transmission Segment Header
0x7f83dc000940 FFF4FFFD 0604FFED FFFD060D 0A0D0A0D ................
0x7f83dc000950 0AFFF4FF FD0604 .......
Thread Control Block (xihTHREAD pCtl)
0x1e11290 58494854 C8100000 2FDD0300 66A40000 XIHT..../...f...
0x1e112a0 01000000 10EB0100 10EB0100 00000000 ................
0x1e112b0 E009E201 00000000 0083E001 00000000 ................
0x1e112c0 00000000 00000000 00000000 00000000 ................ |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 21, 2016 5:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| toshi1234 wrote: |
| | Probe Id :- CO603000 |
Mr Google offers this as the first hit:
http://www-01.ibm.com/support/docview.wss?uid=swg21413653
| Quote: |
| WebSphere MQ detects that bad data is being sent |
I endorse the 3 possible causes listed by my most worthy associate in his earlier post:
| mqjeff wrote: |
Perhaps someone is trying to telnet to the MQ listener port.
Perhaps something is trying to open a TCP connection to the MQ listener port, to find out if the listener is open.
Perhaps something is doing a port scan.
|
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Jul 21, 2016 12:55 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
The FDC file should contain the bad data as well. This should help to identify what made the connection. Can you show us that part of the FDC please?
Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| toshi1234 |
| Posted: Sat Jul 23, 2016 10:29 pm Post subject: |
|
|
Novice
Joined: 15 Dec 2015
Posts: 21
|
Yes hughson , it's containing the data as well I have posted that in the previous post ... At what points data will be populated in FDC files and every time after starting the SERVER CONN channel its goes well unable to find the reason why its happening and cannot find the RCA for this issue need help pls .. 
Thanks |
|
| Back to top |
|
|
| hughson |
| Posted: Sat Jul 23, 2016 10:41 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
Apologies - I missed it - couldn't see the wood for the trees!
Here it is.
| toshi1234 wrote: |
Dump of Transmission Segment Header
0x7f83dc000940 FFF4FFFD 0604FFED FFFD060D 0A0D0A0D ................
0x7f83dc000950 0AFFF4FF FD0604 ....... |
Certainly this is not valid MQ data. It also doesn't look like HTTP data either which is a common one to see with some port scanners. A little web searching does suggest that this is some kind of port scanner though as I see the same set of bytes [FFF4 FFFD 06] in various fora.
Anyone else recognise this data as being from a particular port scanner?
The question to toshi1234 is what do you know about the IP address this data is coming from? Is it a port scanner?
Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| toshi1234 |
| Posted: Mon Jul 25, 2016 9:13 pm Post subject: |
|
|
Novice
Joined: 15 Dec 2015
Posts: 21
|
Apologies for late reply
Its the application ip where it is running, |
|
| Back to top |
|
|
| hughson |
| Posted: Mon Jul 25, 2016 9:32 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| toshi1234 wrote: |
| Its the application ip where it is running, |
What kind of application is it? And what else is running on that machine apart from the application?
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| toshi1234 |
| Posted: Mon Jul 25, 2016 10:13 pm Post subject: |
|
|
Novice
Joined: 15 Dec 2015
Posts: 21
|
| It is a java application and no other application is running on that machine.. |
|
| Back to top |
|
|
| hughson |
| Posted: Tue Jul 26, 2016 1:53 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
So what else does this Java application do apart from MQ stuff?
I would advise TCP tracing to confirm exactly where this data is coming from on that machine.
Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| toshi1234 |
| Posted: Tue Jul 26, 2016 2:22 am Post subject: |
|
|
Novice
Joined: 15 Dec 2015
Posts: 21
|
It places the xml file in the remote queue and that file is transferred to local queue of another queue manager where that xml file get processed.
Thanks |
|
| Back to top |
|
|
| hughson |
| Posted: Tue Jul 26, 2016 3:09 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
So what other TCP traffic is there apart from the MQ traffic? We've suggested a port scanner. What else is on that machine?
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 26, 2016 4:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hughson wrote: |
| So what other TCP traffic is there apart from the MQ traffic? We've suggested a port scanner. What else is on that machine? |
It's unlikely that whatever's causing this is on the machine in question. Staying with the example of a port scanner (as the most likely suspect) that's likely to be a separate machine somewhere else in the network, deployed as part of the organization's vulnerability detection.
@toshi1234,
In case it's not completely clear, the prevailing theory is that your machine is being pinged with non-MQ traffic from elsewhere in the network. The channel doesn't understand it and throws an FDC. This isn't a problem with your machine, the MQ installed on it or the application running there that's trying to use MQ.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| hughson |
| Posted: Tue Jul 26, 2016 9:06 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
The bad data is coming from the IP address shown in the error message and FDC. It is that machine I'm asking about.
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
|
|
