| Author |
Message
|
| mqtechie |
 Posted: Mon May 11, 2009 12:04 am Post subject: MQ AMQ9209 and AMQ9208 issue affected by router IPSEC |
 |
|
Newbie
Joined: 10 May 2009
Posts: 6
|
Hi,
I have an issue with MQ that has router implemented with IPSEC. There are routers (with IPSEC implemented) between the sender and receiver channel.
The system of the sender and receiver are as follows.
Sender: WS MQ 6.0.0.0 (AIX)
Receiver: WS MQ 6.0.2.6 (Windows)
Both sender and receiver system can be telnet to each other and vice versa. Thus, the IPSEC policy was setup correctly.
However, with the IPSEC implemented in the router, we receive the following problems with AMQ9209 and AMQ9208.
To resolve this, we just remove the IPSEC in the router and all works fine now.
Is the latency in the IPSEC cause the impact to the MQ messaging? Or is it something in the IPSEC that cause this issue? Can someone help to provide some logical explanations on this case?
Thanks
From Sender:
16:49:11.622709 376884.1 RetCode = 20009209, rc1 = 0, rc2 = 0,
Comment1 = '203 (nnn.nnn.nn.nnn)', Comment2 = 'TCP/IP', Comment3= '', File=
'./amqccita.c', Line= '3105'
16:49:11.622715 376884.1 ------}! rrxError rc=rrcE_CONNECTION_CLOSED
16:49:11.622721 376884.1 *pointer(1100bb0d8)
16:49:11.622729 376884.1 ------{ ccxFreeMem
16:49:11.622736 376884.1 ------} ccxFreeMem rc=OK
16:49:11.622742 376884.1 RetCode (rrcE_CONNECTION_CLOSED)
16:49:11.622748 376884.1 -----}! cciTcpReceive
rc=rrcE_CONNECTION_CLOSED
16:49:11.622754 376884.1 ----}! ccxReceive rc=rrcE_CONNECTION_CLOSED
16:49:11.622761 376884.1 ----{ rriCommsError
16:49:11.622766 376884.1 ----} rriCommsError rc=OK
16:49:11.622772 376884.1 ---}! rriConfirm rc=rrcE_CONNECTION_CLOSED
16:49:11.622778 376884.1 --}! rriSendData rc=rrcE_CONNECTION_CLOSED
-------------------------------------------------------------------------------
05/05/09 16:49:11 - Process(376884.1) User(mqm) Program(runmqchl_nd)
AMQ9209: Connection to host '203 (nnn.nnn.nn.nnn)' closed.
EXPLANATION:
An error occurred receiving data from '203 (nnn.nnn.nn.nnn)' over TCP/IP.
The
connection to the remote host has unexpectedly terminated.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3105
-------------------------------------------------------
05/05/09 16:49:11 - Process(376884.1) User(mqm) Program(runmqchl_nd)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'QM.S01' ended abnormally.
ACTION:
Look at previous error messages for channel program 'QM.S01' in
the
error files to determine the cause of the failure.
From the receiver:
00214877 14:56:24.949290 2844.9 ---------{ rrxError
00214878 14:56:24.949312 2844.9 RetCode = 20009202, rc1 = 0, rc2 =
0, Comment1 = 'xxx.xx.xx.xx', Comment2 = 'TCP/IP', Comment3= '', File=
'F:\build\p600_P\src\lib\comms\amqcrhna.c', Line= '439'
00214879 14:56:24.949318 2844.9 ---------}! rrxError
(rc=rrcE_HOST_NOT_AVAILABLE)
0021487A 14:56:24.949332 2844.9 AddrName: ''
0021487B 14:56:24.949337 2844.9 Hostname: 'xxx.xx.xx.xx'
0021487C 14:56:24.949341 2844.9 --------}! cciTcpResolveAddress
(rc=rrcE_HOST_NOT_AVAILABLE)
0021487D 14:56:24.949348 2844.9 -------}! cciTcpGetNameandAddress
(rc=rrcE_HOST_NOT_AVAILABLE)
0021487E 14:56:24.949354 2844.9 -------{ rrxError
0021487F 14:56:24.949361 2844.9 RetCode = 20009208, rc1 = 10054, rc2
= 10054, Comment1 = 'xxx.xx.xx.xx', Comment2 = 'TCP/IP', Comment3=
' (recv)', File= 'F:\build\p600_P\src\lib\comms\amqccita.c', Line= '3255'
00214880 14:56:24.949366 2844.9 -------}! rrxError
(rc=rrcE_RECEIVE_FAILED)
00214881 14:56:24.949374 2844.9 *pointer(00860064)
00214882 14:56:24.949379 2844.9 -------{ ccxFreeMem
00214883 14:56:24.949385 2844.9 -------} ccxFreeMem (rc=OK)
00214884 14:56:24.949388 2844.9 RetCode (rrcE_RECEIVE_FAILED)
00214885 14:56:24.949393 2844.9 ------}! cciTcpReceive
(rc=rrcE_RECEIVE_FAILED)
00214886 14:56:24.949399 2844.9 -----}! ccxReceive
(rc=rrcE_RECEIVE_FAILED)
00214887 14:56:24.949457 2844.9 -----{ rriCommsError
00214888 14:56:24.949462 2844.9 -----} rriCommsError (rc=OK)
00214889 14:56:24.949465 2844.9 ----}! rriReceiveData
(rc=rrcE_RECEIVE_FAILED)
-------------------------------------------------------------------------------
5/5/2009 14:56:24 - Process(2844.9) User(MUSR_MQADMIN) Program
(amqrmppa.exe)
AMQ9208: Error on receive from host xxx.xx.xx.xx. (receiver)
EXPLANATION:
An error occurred receiving data from xxx.xx.xx.xx over TCP/IP. This may be
due
to a communications failure.
ACTION:
The return code from the TCP/IP (recv) call was 10054 (X'2746'). Record
these
values and tell the systems administrator.
----- amqccita.c : 3255
-------------------------------------------------------
5/5/2009 14:56:24 - Process(2844.9) User(MUSR_MQADMIN) Program
(amqrmppa.exe)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'QM.S01' ended abnormally.
ACTION:
Look at previous error messages for channel program 'QM.S01' in
the
error files to determine the cause of the failure. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon May 11, 2009 1:09 am Post subject: Re: MQ AMQ9209 and AMQ9208 issue affected by router IPSEC |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqtechie wrote: |
The return code from the TCP/IP (recv) call was 10054 (X'2746'). Record
these
values and tell the systems administrator.
|
The router is hanging up on the channel. Speak to your network people and ensure that the policies surrounding apparently domant connections are set ok. Also search this forum for discussions around KeepAlive
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqtechie |
| Posted: Mon May 11, 2009 1:21 am Post subject: |
|
|
Newbie
Joined: 10 May 2009
Posts: 6
|
Hi Vitor,
Thanks for your response.
My group of network people (3 people) have spent a whole day verifying the whole network and routers and they found no problems on the policies and network connections.
The KeepAlive feature is also acivated in both sender (6.0.0.0) and receiver (6.0.2.6) MQ.
So, we are puzzled. The last thing we did is just remove the IPSEC in the router and everything went well.
Hope someone can help to give some logical explanations to this case.
Thanks |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 11, 2009 3:23 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You said that with ipsecure telnet worked fine. Did it work if you tried to telnet to the MQ port?
Did ipsecure somehow filter out the MQ protocol or the MQ port?
Does it tag the data flowing through it with some additional data that is not understood by MQ? Does it encrypt the data?
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon May 11, 2009 3:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Is the IPSec tunnel timing out, or closing and reopening to renegotiate a secret key?
Does the channel start successfully with IPSEC enabled and then die later? Or fail to start in the first place? |
|
| Back to top |
|
|
| mqtechie |
| Posted: Mon May 11, 2009 5:16 am Post subject: |
|
|
Newbie
Joined: 10 May 2009
Posts: 6
|
Hi mqjeff/fjb_saber,
Thanks for your response. Questions and Answers are written below.
Question: You said that with ipsecure telnet worked fine. Did it work if you tried to telnet to the MQ port?
Answer: Yes. It worked when I tried to telnet to the MQ Port from sender to receiver and vice versa.
Question:
Did ipsecure somehow filter out the MQ protocol or the MQ port?
Answer: No. The network people checked and the policy was set correctly.
Question:
Does it tag the data flowing through it with some additional data that is not understood by MQ? Does it encrypt the data?
Answer: No. We just put some test message (e.g. testing123) on the queue for testing.
Question:
Is the IPSec tunnel timing out, or closing and reopening to renegotiate a secret key?
Answer: No. The secret keys are correct. Telnet able to work successfully.
Question:
Does the channel start successfully with IPSEC enabled and then die later? Or fail to start in the first place?
Answer: We did not try it because it is in production and we are not allowed to enabled it back until a logical explanation is found.
Note that the sender has no CSD installed. Version 6.0.0.0. Could it be one of the unfixed bug cause the issue?
Hope someone can help to give an logical explanation to this.
Thanks. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon May 11, 2009 5:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Yes, the back level could contribute.
If you can't recreate the problem - because you can't reenable it in production - then you can still tell us what DID happen. Did you enable IPSEC and have the channel start successfully at all? |
|
| Back to top |
|
|
| mqtechie |
| Posted: Mon May 11, 2009 6:20 am Post subject: |
|
|
Newbie
Joined: 10 May 2009
Posts: 6
|
Hi mqjeff,
I actually disable the IPSEC and the channel can communicate successfully. Just tested that if the IPSEC is enabled, the errors AMQ9209 and AMQ9208 came back.
Do you think which of the back level can contribute this issue?
Thanks |
|
| Back to top |
|
|
| belchman |
| Posted: Mon May 11, 2009 10:32 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
It is also possible that you router is set to clean up connections that have been inactive for X seconds and your discint property on your sender channel is Y seconds and X < Y.
This would cause those types of errors because to the MQ nodes, when the cleanup process cleans the "idle" threads it does so in a way that appears abrupt to MQ.
For example, our Firewall cleans up connections it thinks are unused or orphaned. The channels are running but no data is going across. When the FW closes the connection, an error is trapped and written to the qmgr logs because MQ thinks it was an ABEND.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Mon May 11, 2009 10:54 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
| The back level Jeff mentions is AIX at WMQ 6.0.0.0. Please apply 6.0.2.6 maintenance. Notice that maintenance also closes a security hole around setmqaut. |
|
| Back to top |
|
|
| belchman |
| Posted: Mon May 11, 2009 11:28 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
Can someone point me to something that explains "back level"?
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 11, 2009 2:40 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
back level means not current version of the software. The current version of the software contains hundreds of fixes that your version does not.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| nheng |
| Posted: Mon May 11, 2009 8:06 pm Post subject: hi |
|
|
Apprentice
Joined: 07 Dec 2007
Posts: 39
|
| IPsec may be encrypt data (DES , 3DES , IDEA , RC2 ,etc) .Plz disable its. |
|
| Back to top |
|
|
| mqtechie |
| Posted: Mon May 11, 2009 10:56 pm Post subject: Re: hi |
|
|
Newbie
Joined: 10 May 2009
Posts: 6
|
| nheng wrote: |
| IPsec may be encrypt data (DES , 3DES , IDEA , RC2 ,etc) .Plz disable its. |
Hi nheng,
Thanks for your comment.
Yes. We disable the IPSec and it works. The question is why? Need a logical explanation.
Thanks |
|
| Back to top |
|
|
| vol |
| Posted: Tue May 12, 2009 1:17 am Post subject: |
|
|
Acolyte
Joined: 01 Feb 2009
Posts: 69
|
reenable the IPSEC, and get your network guys to check what is happening to the packets sent out by the WMQ servers
9209 and 9208 are the same error; one is on the sending side and one on the receiver, and record the fact that expected data is not arriving.
Clearly the IPSEC is swallowing the data; get over your denial of this and check it. |
|
| Back to top |
|
|
|
|
