ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » An error occurred receiving data AMQ9207 and AMQ9209

Post new topic  Reply to topic Goto page 1, 2  Next
 An error occurred receiving data AMQ9207 and AMQ9209 « View previous topic :: View next topic » 
Author Message
toshi1234
PostPosted: Thu Jul 21, 2016 3:50 am    Post subject: An error occurred receiving data AMQ9207 and AMQ9209 Reply with quote

Novice

Joined: 15 Dec 2015
Posts: 21

Hi all

We are having JMS/WAS application (Linux) who trying to put message in queue .

Some time back the JMS App could not able to put message in a queue;

When restarting the server connection channel all goes well but its occurring frequently no logs at application side.

07/20/2016 06:49:26 PM - Process(21301.6 User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9209: Connection to host '' closed.

EXPLANATION:
An error occurred receiving data from '' over TCP/IP. The connection
to the remote host has unexpectedly terminated.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3473 -------------------------------------------------------
07/20/2016 06:49:26 PM - Process(21301.69) User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9209: Connection to host '' closed.

EXPLANATION:
An error occurred receiving data from '' over TCP/IP. The connection
to the remote host has unexpectedly terminated.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3473 -------------------------------------------------------
07/20/2016 07:14:34 PM - Process(21301.8 User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9207: The data received from host '10 ()' is not valid.

EXPLANATION:
Incorrect data format received from host '10 ()' over TCP/IP. It
may be that an unknown host is attempting to send data. An FFST file has been
generated containing the invalid data received.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3620 -------------------------------------------------------
07/20/2016 07:14:34 PM - Process(21301.8 User(mqm) Program(amqrmppa)
Host(mqm-live-primary)
AMQ9492: The TCP/IP responder program encountered an error.

EXPLANATION:
The responder program was started but detected an error.
ACTION:
Look at previous error messages in the error files to determine the error
encountered by the responder program.
----- amqrmrsa.c : 524 --------------------------------------------------------

And FDC file I see

[img]
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- rrcE_BAD_DATA_RECEIVED |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ9207 |
| Probe Severity :- 2 |
| Probe Description :- AMQ9207: The data received from host '10 [/img]


for this i changed the SHARECNV property to 0 on server connection channel but same issue occurred recently.

Unable to find the RCA for this issue please share your suggestions it will be very helpful

Thanks..
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Jul 21, 2016 3:57 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Perhaps someone is trying to telnet to the MQ listener port.

Perhaps something is trying to open a TCP connection to the MQ listener port, to find out if the listener is open.

Perhaps something is doing a port scan.

Perhaps the FFST/FDC file has a slight bit of more information.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
toshi1234
PostPosted: Thu Jul 21, 2016 5:12 am    Post subject: Reply with quote

Novice

Joined: 15 Dec 2015
Posts: 21

Hi mqjeff

Thanks for quick reply this is FDC file generated every time

+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Wed July 20 2016 18:41:26 IST |
| UTC Time :- 1469020286.292926 |
| UTC Time Offset :- 330 (IST) |
| Host Name :- interface |
| Operating System :- Linux 2.6.32-573.el6.x86_64 |
| PIDS :- 5724H7230 |
| LVLS :- 7.5.0.5 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| Installation Path :- /opt/mqm |
| Installation Name :- Installation1 (1) |
| License Type :- Production |
| Probe Id :- CO603000 |
| Application Name :- MQM |
| Component :- cciInvalidDataReceived |
| SCCS Info :- /build/slot1/p750_P/src/lib/comms/amqccita.c, |
| Line Number :- 4295 |
| Build Date :- Apr 24 2015 |
| Build Level :- p750-005-150424 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 501 (mqm) |
| Real UserID :- 501 (mqm) |
| Program Name :- amqrmppa |
| Arguments :- -m "*** " |
| Addressing mode :- 64-bit |
| LANG :- en_US.UTF-8 |
| Process :- 125712 |
| Process(Thread) :- 42086 |
| Thread :- 253231 |
| ThreadingModel :- PosixThreads |
| UserApp :- FALSE |
| ConnId(1) IPCC :- 285577 |
| ConnId(3) QM-P :- 282334 |
| Last HQC :- 1.0.0-136584 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- rrcE_BAD_DATA_RECEIVED |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ9207 |
| Probe Severity :- 2 |
| Probe Description :- AMQ9207: The data received from host '*.*.*.*' on |
| channel '????' is not valid. |
| FDCSequenceNumber :- 0 |
| Comment1 :- |
| Comment2 :- TCP/IP |
| Comment3 :- ???? |
| |
+-----------------------------------------------------------------------------+

MQM Function Stack
ccxResponder
rrxResponder
rriAcceptSess
ccxReceive
cciTcpReceive
cciInvalidDataReceived
xcsFFST

MQM Trace History
-} xcsRequestThreadMutexSem rc=OK
-{ xcsConnectExistingSharedSubpool
--{ xstGetSubpoolsLock
---{ xstAccessSubpoolsLock
----{ xcsQueryValue
----} xcsQueryValue rc=OK
---} xstAccessSubpoolsLock rc=OK
Data: 0x01200026 0x00000001
--} xstGetSubpoolsLock rc=OK
Data: 0x00000001 0x00045b89 0x0003dd2f
--{ xcsAllocateQuickCell
--} xcsAllocateQuickCell rc=OK
--{ xstReleaseSubpoolsLock
---{ xstAccessSubpoolsLock
----{ xcsQueryValue
----} xcsQueryValue rc=OK
---} xstAccessSubpoolsLock rc=OK
--} xstReleaseSubpoolsLock rc=OK
--{ xlsThreadInitialization
---{ xlsThreadPreInitialization
----{ xcsAllocateQuickCell
----} xcsAllocateQuickCell rc=OK
----{ xcsQueryProcessDetails
----} xcsQueryProcessDetails rc=OK
---} xlsThreadPreInitialization rc=OK
--} xlsThreadInitialization rc=OK
--{ xcsConnectSharedMemSet
---{ xcsRequestThreadMutexSem
---} xcsRequestThreadMutexSem rc=OK
---{ xcsRequestThreadMutexSem
---} xcsRequestThreadMutexSem rc=OK
---{ xcsReleaseThreadMutexSem
---} xcsReleaseThreadMutexSem rc=OK
---{ xcsGetMemFn
---} xcsGetMemFn rc=OK
---{ xcsReleaseThreadMutexSem
---} xcsReleaseThreadMutexSem rc=OK
--} xcsConnectSharedMemSet rc=OK
-} xcsConnectExistingSharedSubpool rc=OK
-{ xcsGetEnvironmentString
-} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
-{ xcsReleaseThreadMutexSem
-} xcsReleaseThreadMutexSem rc=OK
} xcsInitialize rc=OK
{ ccxResponder
-{ cccGetConfig
--{ xcsRequestThreadMutexSem
--} xcsRequestThreadMutexSem rc=OK
--{ xcsReleaseThreadMutexSem
--} xcsReleaseThreadMutexSem rc=OK
-} cccGetConfig rc=OK
-{ ccxNetWorkInit
--{ ccxParseIni
--} ccxParseIni rc=OK
--{ cciLoadLibrary
---{ cccGetMem
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
---} cccGetMem rc=OK
--} cciLoadLibrary rc=OK
--{ xcsRequestThreadMutexSem
--} xcsRequestThreadMutexSem rc=OK
--{ xcsReleaseThreadMutexSem
--} xcsReleaseThreadMutexSem rc=OK
-} ccxNetWorkInit rc=OK
-{ ccxAcceptConv
--{ cciTcpAcceptConv
---{ ioctl
---} ioctl rc=OK
---{ cciTcpGetPeerName
----{ cciTcpResolveAddress
----} cciTcpResolveAddress rc=OK
---} cciTcpGetPeerName rc=OK
---{ cciTcpResolveAddress
---} cciTcpResolveAddress rc=OK
--} cciTcpAcceptConv rc=OK
--{ ccxCheckConvBlocked
---{ rsxAccessCache
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
----{ xcsConnectSharedMemSet
-----{ xcsRequestThreadMutexSem
-----} xcsRequestThreadMutexSem rc=OK
-----{ xcsRequestThreadMutexSem
-----} xcsRequestThreadMutexSem rc=OK
-----{ xcsReleaseThreadMutexSem
-----} xcsReleaseThreadMutexSem rc=OK
-----{ xcsGetMemFn
-----} xcsGetMemFn rc=OK
-----{ xcsReleaseThreadMutexSem
-----} xcsReleaseThreadMutexSem rc=OK
----} xcsConnectSharedMemSet rc=OK
----{ xcsRequestMutexSem
----} xcsRequestMutexSem rc=OK
----{ xcsReleaseMutexSem
----} xcsReleaseMutexSem rc=OK
---} rsxAccessCache rc=OK
---{ rsxReadLock
----{ xlsRWMutexRequest
-----{ xlsRWLockState
-----} xlsRWLockState rc=OK
-----{ xlsRWUnlockState
-----} xlsRWUnlockState rc=OK
----} xlsRWMutexRequest rc=OK
---} rsxReadLock rc=OK
---{ rsxIsAddressAllowed
----{ rsxReadUnlock
----} rsxReadUnlock rc=OK
---} rsxIsAddressAllowed rc=OK
---{ rsxReadUnlock
----{ xlsRWMutexRelease
-----{ xlsRWLockState
-----} xlsRWLockState rc=OK
-----{ xlsRWUnlockState
-----} xlsRWUnlockState rc=OK
----} xlsRWMutexRelease rc=OK
---} rsxReadUnlock rc=OK
---{ rsxReleaseCache
----{ xcsDisconnectSharedMemSet
-----{ xcsRequestThreadMutexSem
-----} xcsRequestThreadMutexSem rc=OK
-----{ xcsFreeMemFn
-----} xcsFreeMemFn rc=OK
-----{ xcsReleaseThreadMutexSem
-----} xcsReleaseThreadMutexSem rc=OK
----} xcsDisconnectSharedMemSet rc=OK
----{ xcsFreeMemFn
----} xcsFreeMemFn rc=OK
---} rsxReleaseCache rc=OK
--} ccxCheckConvBlocked rc=OK
-} ccxAcceptConv rc=OK
-{ rrxResponder
--{ cccGetConfig
--} cccGetConfig rc=OK
--{ rriAcceptSess
---{ cccGetMem
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
---} cccGetMem rc=OK
---{ xcsGetNextTraceIdent
----{ xcsRequestThreadMutexSem
----} xcsRequestThreadMutexSem rc=OK
----{ xcsReleaseThreadMutexSem
----} xcsReleaseThreadMutexSem rc=OK
---} xcsGetNextTraceIdent rc=OK
---{ xcsSetTraceIdent
---} xcsSetTraceIdent rc=OK
---{ ccxSetTimeOut
---} ccxSetTimeOut rc=OK
---{ cccGetMem
----{ xcsGetMemFn
----} xcsGetMemFn rc=OK
---} cccGetMem rc=OK
---{ ccxReceive
----{ cciTcpReceive
-----{ ccxAllocMem
------{ cccGetMem
-------{ xcsGetMemFn
-------} xcsGetMemFn rc=OK
------} cccGetMem rc=OK
-----} ccxAllocMem rc=OK
-----{ recv
-----} recv rc=Unknown(B)
Data: 0xffffffff
-----{ cciTcpWaitReceive
------{ xcsWaitFd
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=OK
-------{ poll
-------} poll rc=Unknown(1)
------} xcsWaitFd rc=Unknown(1)
-----} cciTcpWaitReceive rc=OK
-----{ recv
-----} recv rc=OK
Data: 0x00000005
-----{ recv
-----} recv rc=Unknown(B)
Data: 0xffffffff
-----{ cciTcpWaitReceive
------{ xcsWaitFd
-------{ poll
-------} poll rc=Unknown(1)
------} xcsWaitFd rc=Unknown(1)
-----} cciTcpWaitReceive rc=OK
-----{ recv
-----} recv rc=OK
Data: 0x00000001
-----{ recv
-----} recv rc=Unknown(B)
Data: 0xffffffff
-----{ cciTcpWaitReceive
------{ xcsWaitFd
-------{ poll
-------} poll rc=Unknown(1)
------} xcsWaitFd rc=Unknown(1)
-----} cciTcpWaitReceive rc=OK
-----{ recv
-----} recv rc=OK
Data: 0x00000005
-----{ cciBadDataFFSTCheck
------{ xcsIsEnvironment
------} xcsIsEnvironment rc=OK
-----} cciBadDataFFSTCheck rc=Unknown(1)
-----{ cciInvalidDataReceived
------{ xcsSleep
------} xcsSleep rc=OK
------{ ioctl
------} ioctl rc=OK
------{ recv
------} recv rc=OK
Data: 0x0000000c
------{ cciTcpGetNameandAddress
-------{ cciTcpResolveHostname
--------{ xcsIsEnvironment
--------} xcsIsEnvironment rc=OK
-------} cciTcpResolveHostname rc=OK
-------{ cciTcpResolveAddress
--------{ rrxError
--------} rrxError rc=rrcE_HOST_NOT_AVAILABLE
-------} cciTcpResolveAddress rc=rrcE_HOST_NOT_AVAILABLE
------} cciTcpGetNameandAddress rc=rrcE_HOST_NOT_AVAILABLE
------{ rrxError
------} rrxError rc=rrcE_BAD_DATA_RECEIVED
------{ xcsBuildDumpPtr
-------{ xcsGetMemFn
-------} xcsGetMemFn rc=OK
------} xcsBuildDumpPtr rc=OK
------{ xcsFFST

Dump of Transmission Segment Header
0x7f83dc000940 FFF4FFFD 0604FFED FFFD060D 0A0D0A0D ................
0x7f83dc000950 0AFFF4FF FD0604 .......


Thread Control Block (xihTHREAD pCtl)
0x1e11290 58494854 C8100000 2FDD0300 66A40000 XIHT..../...f...
0x1e112a0 01000000 10EB0100 10EB0100 00000000 ................
0x1e112b0 E009E201 00000000 0083E001 00000000 ................
0x1e112c0 00000000 00000000 00000000 00000000 ................
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jul 21, 2016 5:22 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

toshi1234 wrote:
| Probe Id :- CO603000


Mr Google offers this as the first hit:

http://www-01.ibm.com/support/docview.wss?uid=swg21413653

Quote:
WebSphere MQ detects that bad data is being sent


I endorse the 3 possible causes listed by my most worthy associate in his earlier post:

mqjeff wrote:

Perhaps someone is trying to telnet to the MQ listener port.

Perhaps something is trying to open a TCP connection to the MQ listener port, to find out if the listener is open.

Perhaps something is doing a port scan.

_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Jul 21, 2016 12:55 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

The FDC file should contain the bad data as well. This should help to identify what made the connection. Can you show us that part of the FDC please?

Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
toshi1234
PostPosted: Sat Jul 23, 2016 10:29 pm    Post subject: Reply with quote

Novice

Joined: 15 Dec 2015
Posts: 21

Yes hughson , it's containing the data as well I have posted that in the previous post ... At what points data will be populated in FDC files and every time after starting the SERVER CONN channel its goes well unable to find the reason why its happening and cannot find the RCA for this issue need help pls ..

Thanks
Back to top
View user's profile Send private message
hughson
PostPosted: Sat Jul 23, 2016 10:41 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

Apologies - I missed it - couldn't see the wood for the trees!

Here it is.

toshi1234 wrote:
Dump of Transmission Segment Header
0x7f83dc000940 FFF4FFFD 0604FFED FFFD060D 0A0D0A0D ................
0x7f83dc000950 0AFFF4FF FD0604 .......


Certainly this is not valid MQ data. It also doesn't look like HTTP data either which is a common one to see with some port scanners. A little web searching does suggest that this is some kind of port scanner though as I see the same set of bytes [FFF4 FFFD 06] in various fora.

Anyone else recognise this data as being from a particular port scanner?

The question to toshi1234 is what do you know about the IP address this data is coming from? Is it a port scanner?

Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
toshi1234
PostPosted: Mon Jul 25, 2016 9:13 pm    Post subject: Reply with quote

Novice

Joined: 15 Dec 2015
Posts: 21

Apologies for late reply

Its the application ip where it is running,
Back to top
View user's profile Send private message
hughson
PostPosted: Mon Jul 25, 2016 9:32 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

toshi1234 wrote:
Its the application ip where it is running,

What kind of application is it? And what else is running on that machine apart from the application?
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
toshi1234
PostPosted: Mon Jul 25, 2016 10:13 pm    Post subject: Reply with quote

Novice

Joined: 15 Dec 2015
Posts: 21

It is a java application and no other application is running on that machine..
Back to top
View user's profile Send private message
hughson
PostPosted: Tue Jul 26, 2016 1:53 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

So what else does this Java application do apart from MQ stuff?

I would advise TCP tracing to confirm exactly where this data is coming from on that machine.

Cheers
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
toshi1234
PostPosted: Tue Jul 26, 2016 2:22 am    Post subject: Reply with quote

Novice

Joined: 15 Dec 2015
Posts: 21

It places the xml file in the remote queue and that file is transferred to local queue of another queue manager where that xml file get processed.

Thanks
Back to top
View user's profile Send private message
hughson
PostPosted: Tue Jul 26, 2016 3:09 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

So what other TCP traffic is there apart from the MQ traffic? We've suggested a port scanner. What else is on that machine?
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Vitor
PostPosted: Tue Jul 26, 2016 4:20 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

hughson wrote:
So what other TCP traffic is there apart from the MQ traffic? We've suggested a port scanner. What else is on that machine?


It's unlikely that whatever's causing this is on the machine in question. Staying with the example of a port scanner (as the most likely suspect) that's likely to be a separate machine somewhere else in the network, deployed as part of the organization's vulnerability detection.

@toshi1234,

In case it's not completely clear, the prevailing theory is that your machine is being pinged with non-MQ traffic from elsewhere in the network. The channel doesn't understand it and throws an FDC. This isn't a problem with your machine, the MQ installed on it or the application running there that's trying to use MQ.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
hughson
PostPosted: Tue Jul 26, 2016 9:06 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

The bad data is coming from the IP address shown in the error message and FDC. It is that machine I'm asking about.
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General Discussion » An error occurred receiving data AMQ9207 and AMQ9209
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.