ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ Support1300 character Message not transmitted

Post new topicReply to topic Goto page 1, 2, 3, 4, 5, 6, 7  Next
1300 character Message not transmitted View previous topic :: View next topic
Author Message
flsb
PostPosted: Mon Oct 14, 2019 5:43 pm Post subject: 1300 character Message not transmitted Reply with quote

Apprentice

Joined: 01 Apr 2010
Posts: 38

Hi everyone,
My issue is similar to the thread "Messages stalled on XMITQ" but it is slightly different..
i usually can google and found the solution to my MQ problems but this issue really had me running in circles..

No FDC file created.. running MQ 9.1.0.3 in Linux RHEL7

Error in AMQERR log

Quote:
AMQ9259E: Connection timed out from host 'xxx'.

EXPLANATION:
A connection from host 'xxx' over TCP/IP timed out.
ACTION:
The select() [TIMEOUT] 60 seconds call timed out. Check to see why data was not
received in the expected time. Correct the problem. Reconnect the channel, or
wait for a retrying channel to reconnect itself.

AMQ9999E: Channel 'xxx.CHANNEL' to host 'xxx' ended
abnormally.

EXPLANATION:
The channel program running under process ID 26742 for channel
'xxx.CHANNEL' ended abnormally. The host name is 'xxx'; in
some cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure.


Other channels to the same location is running OK, and only this channel is with messages stuck in XMITQ.
It looks like the message is not able to fully transmit to the destination.
Unable to find amqsbcg in /opt/mqm/bin

We split the 1.3k characters message to smaller chunks and is able to send through but with altering the channel batchsz to 1.

Another weird thing is we currently migrated to MQ v9.1 from MQv5.3 (AIX).
While it works perfectly in MQv5.3 AIX previously.
WE have few parties with different channels and destination, but out of 10-20 parties, only 2 parties are having this issue..

I searched high and low and it all asked to contact network side.. but the channel is able to ping, it just data stuck in XMITQ. Network has found no issue at their end.

Would appreciate anybody that has experienced this before..
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Oct 14, 2019 6:24 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8534
Location: US: west coast, almost. Otherwise, enroute.

Did you use the exact same v5.3 channel definitions in the 9.1 qmgrs?

Post the sender side channel definition here.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
flsb
PostPosted: Mon Oct 14, 2019 6:47 pm Post subject: Reply with quote

Apprentice

Joined: 01 Apr 2010
Posts: 38

we create using the same definition..

DEFINE CHANNEL(xxx.CHANNEL) CHLTYPE(SVR) TRPTYPE(TCP) XMITQ(xxx.TRAN.QUEUE) DISCINT(0) CONNAME('xxx')


Quote:
CHANNEL(xxx.CHANNEL) CHLTYPE(SVR)
ALTDATE(2019-10-12) ALTTIME(09.18.17)
BATCHHB(0) BATCHINT(0)
BATCHLIM(5000) BATCHSZ(50)
CERTLABL( ) COMPHDR(NONE)
COMPMSG(NONE) CONNAME(xxx)
CONVERT(NO) DESCR( )
DISCINT(0) HBINT(300)
KAINT(AUTO) LOCLADDR( )
LONGRTY(999999999) LONGTMR(1200)
MAXMSGL(4194304) MCANAME( )
MCATYPE(PROCESS) MCAUSER( )
MODENAME( ) MONCHL(QMGR)
MSGDATA( ) MSGEXIT( )
NPMSPEED(FAST) PASSWORD( )
PROPCTL(COMPAT) RCVDATA( )
RCVEXIT( ) RESETSEQ(NO)
SCYDATA( ) SCYEXIT( )
SENDDATA( ) SENDEXIT( )
SEQWRAP(999999999) SHORTRTY(10)
SHORTTMR(60) SSLCAUTH(REQUIRED)
SSLCIPH( ) SSLPEER( )
STATCHL(QMGR) TPNAME( )
TRPTYPE(TCP) USEDLQ(YES)
USERID( ) XMITQ(xxx.TRAN.QUEUE)
Back to top
View user's profile Send private message
exerk
PostPosted: Mon Oct 14, 2019 11:21 pm Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6110

1300 characters, but what is the actual message size? Do you know what batch size is set at the receiving end? And do you really want your channels to run continuously once started, i.e. with no DISCINT?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
flsb
PostPosted: Tue Oct 15, 2019 12:53 am Post subject: Reply with quote

Apprentice

Joined: 01 Apr 2010
Posts: 38

it is just plain text..
total 1300 characters...
default MAXMSGL is 4Mb... i don't think the message can exceed 4Mb even with headers and encryption etc..
we have tried increasing to 10mb on Qmgr, channel and queue level.. and still fail to send..

The batchsz for both channel is set to 50... we tried to set as 1 but still failed to send..
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Oct 15, 2019 4:15 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8534
Location: US: west coast, almost. Otherwise, enroute.

flsb wrote:
it is just plain text..
total 1300 characters...
default MAXMSGL is 4Mb... i don't think the message can exceed 4Mb even with headers and encryption etc..
we have tried increasing to 10mb on Qmgr, channel and queue level.. and still fail to send..

The batchsz for both channel is set to 50... we tried to set as 1 but still failed to send..

How/where do you encrypt the message? In an MQ exit?

Please use one of the supplied utilities to display the message in the XMITQ. Post the results here.

The supplied utilities are found MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin (32-bit versions) MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin64 (64-bit versions)
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
hughson
PostPosted: Tue Oct 15, 2019 1:45 pm Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1283
Location: Bay of Plenty, New Zealand

The error you supply from the AMQERR01.LOG is from the SVR side of the channel I assume? Please can you also supply the error in the AMQERR01.LOG from the other end of the channel (and also tell us whether it is a RCVR or a RQSTR).

Thanks,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
gbaddeley
PostPosted: Tue Oct 15, 2019 3:28 pm Post subject: Re: 1300 character Message not transmitted Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2040
Location: Melbourne, Australia

flsb wrote:
Hi everyone,
AMQ9259E: Connection timed out from host 'xxx'.
EXPLANATION:
A connection from host 'xxx' over TCP/IP timed out.
ACTION:
The select() [TIMEOUT] 60 seconds call timed out. Check to see why data was not
received in the expected time. Correct the problem. Reconnect the channel, or
wait for a retrying channel to reconnect itself.
AMQ9999E: Channel 'xxx.CHANNEL' to host 'xxx' ended
abnormally.

... WE have few parties with different channels and destination, but out of 10-20 parties, only 2 parties are having this issue....
.... I searched high and low and it all asked to contact network side.. but the channel is able to ping, it just data stuck in XMITQ. Network has found no issue at their end....

select() timeout usually means that the network connection was lost at the TCP socket level. Nothing you can do in MQ will fix this. It is a network issue. Request your network team to run a packet trace between the two IPs. Check that firewalls or switches are not dropping the connection.
_________________
Glenn
Back to top
View user's profile Send private message
flsb
PostPosted: Tue Oct 15, 2019 5:53 pm Post subject: Reply with quote

Apprentice

Joined: 01 Apr 2010
Posts: 38

Quote:
How/where do you encrypt the message? In an MQ exit?

The supplied utilities are found MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin (32-bit versions) MQ_INSTALLATION_PATH\ Tools\C\Samples\Bin64 (64-bit versions))


is this windows? Im running in RHEL7
Installation path is /opt/mqm/bin and i am unable to locate this amqsbcg
Can i copy another amqsbcg from MQ7.5 into this server?

I encrypt the data on application level before putting the message into qremote
Back to top
View user's profile Send private message
flsb
PostPosted: Tue Oct 15, 2019 5:55 pm Post subject: Reply with quote

Apprentice

Joined: 01 Apr 2010
Posts: 38

hughson wrote:
The error you supply from the AMQERR01.LOG is from the SVR side of the channel I assume? Please can you also supply the error in the AMQERR01.LOG from the other end of the channel (and also tell us whether it is a RCVR or a RQSTR).

Thanks,
Morag


AMQERR01 at receiver's end

AMQ9788: Slow DNS lookup for address 'xxx'.

EXPLANATION:
An attempt to resolve address 'xxx' using the 'getnameinfo' function
call took 12 seconds to complete. This might indicate a problem with the DNS
configuration.
ACTION:
Ensure that DNS is correctly configured on the local system.

If the address was an IP address then the slow operation was a reverse DNS
lookup. Some DNS configurations are not capable of reverse DNS lookups and some
IP addresses have no valid reverse DNS entries. If the problem persists,
consider disabling reverse DNS lookups until the issue with the DNS can be
resolved.
----- amqcrhna.c : 785 --------------------------------------------------------
10/9/2019 10:34:37 - Process(6524.13237038) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(KULDCPWFLS01) Installation(FLS01)
VRMF(8.0.0.4) QMgr(queue.manager.aia)

AMQ9259: Connection timed out from host 'xxx'.

EXPLANATION:
A connection from host 'xxx' over TCP/IP timed out.
ACTION:
The select() [TIMEOUT] 360 seconds call timed out. Check to see why data was
not received in the expected time. Correct the problem. Reconnect the channel,
or wait for a retrying channel to reconnect itself.
----- amqccita.c : 4452 -------------------------------------------------------
10/9/2019 10:34:37 - Process(6524.13237038) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(KULDCPWFLS01) Installation(FLS01)
VRMF(8.0.0.4) QMgr(queue.manager.aia)

AMQ9999: Channel 'xxx.CHANNEL' to host 'xxx' ended abnormally.

EXPLANATION:
The channel program running under process ID 6524(1912) for channel
'xxx.CHANNEL' ended abnormally. The host name is 'xxx'; in
some cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
Back to top
View user's profile Send private message
flsb
PostPosted: Tue Oct 15, 2019 5:57 pm Post subject: Re: 1300 character Message not transmitted Reply with quote

Apprentice

Joined: 01 Apr 2010
Posts: 38

gbaddeley wrote:

select() timeout usually means that the network connection was lost at the TCP socket level. Nothing you can do in MQ will fix this. It is a network issue. Request your network team to run a packet trace between the two IPs. Check that firewalls or switches are not dropping the connection.


Receiver's side has run WIreshark and noted packet is receiving..
Firewall at my side is showing packet is sending..
it just seems like the message is not committing at receiver's end completely..
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Oct 15, 2019 8:29 pm Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20138
Location: LI,NY

Here is your problem
Quote:
EXPLANATION:
An attempt to resolve address 'xxx' using the 'getnameinfo' function
call took 12 seconds to complete. This might indicate a problem with the DNS
configuration.
ACTION:
Ensure that DNS is correctly configured on the local system.

If the address was an IP address then the slow operation was a reverse DNS lookup. Some DNS configurations are not capable of reverse DNS lookups and some IP addresses have no valid reverse DNS entries.
If the problem persists, consider disabling reverse DNS lookups until the issue with the DNS can be resolved.


Have the receiving side disable reverse DNS lookup, and also disable it on your side.

Enjoy
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
flsb
PostPosted: Wed Oct 16, 2019 12:15 am Post subject: Reply with quote

Apprentice

Joined: 01 Apr 2010
Posts: 38

fjb_saper wrote:
Here is your problem
Quote:
EXPLANATION:
An attempt to resolve address 'xxx' using the 'getnameinfo' function
call took 12 seconds to complete. This might indicate a problem with the DNS
configuration.
ACTION:
Ensure that DNS is correctly configured on the local system.

If the address was an IP address then the slow operation was a reverse DNS lookup. Some DNS configurations are not capable of reverse DNS lookups and some IP addresses have no valid reverse DNS entries.
If the problem persists, consider disabling reverse DNS lookups until the issue with the DNS can be resolved.


Have the receiving side disable reverse DNS lookup, and also disable it on your side.

Enjoy


i will try to get them to disable reverse DNS lookup..
ALTER QMGR REVDNS(DISABLED)

but the problem only happen on one of the channel where large 1.3k dat is sent..
it is ok after we split the message into smaller chunks and batchsz(1)
Back to top
View user's profile Send private message
HubertKleinmanns
PostPosted: Wed Oct 16, 2019 12:56 am Post subject: Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 724
Location: Germany

flsb wrote:
i will try to get them to disable reverse DNS lookup..
ALTER QMGR REVDNS(DISABLED)

but the problem only happen on one of the channel where large 1.3k dat is sent..
it is ok after we split the message into smaller chunks and batchsz(1)


Hmm, 1.3 k is not really large. I worked with MQ Systems, which transferred messages of several 100 MB (with messsage segmentation ) within one batch without problems . Even the MQ header has about 500 b.

When you split the message:

- Which size did these spitted messages have?
- Did it work when they where sent within one batch (e. g. 130 b and BATCHSZ 100)?
- Could you sent the bulk of splitted messages again and deliver the output of the following command before and after the transfer:
Code:
dis chs(<your Sender channel>) all


Some further questions:

- Does 'xxx' stand for an IP address or a host name?
- What's about the log size at your parties side? The content of "qm.ini" of your parties would be helpful.
- What's about the load on the parties side?

In Addition: I don't think, that DNS use used on a runnung channel. I assume, the DNS resolution only occurs when the channel starts.
_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
hughson
PostPosted: Wed Oct 16, 2019 1:08 am Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1283
Location: Bay of Plenty, New Zealand

I concur that the DNS problem is a red herring - only happens at channel start, not part of message delivery.

The fact that both ends of the channel are timing out (as per the error messages you have provided), this suggests that some part of the networking stack is not happy with your socket. I would be inclined to begin with the firewall as that would be a very common cause of such behaviour.

Perhaps there is something in your message it doesn't like the look of.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:
Post new topicReply to topic Goto page 1, 2, 3, 4, 5, 6, 7  Next Page 1 of 7

MQSeries.net Forum IndexGeneral IBM MQ Support1300 character Message not transmitted
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.