| Author |
Message
|
| rags132 |
 Posted: Thu Dec 02, 2021 6:46 am Post subject: MQPUT taking 4 to 5mins when network is disconnected |
 |
|
Newbie
Joined: 02 Dec 2021
Posts: 3
|
Our MQ client runs in AWS cloud and actual MQ server runs in on-perm datacenter. For testing purpose, our network team disconnected network between AWS cloud and on-perm datacenter and network cut was only for 10 secs. During that exercise, we have observed MQPUTs were taking 4 to 5minutes to return 2009 error code.
I believe these MQPUT calls were dropped in n/w and not reached queue manager. Can you please suggest is there a way we can make MQPUT calls timed-out (say less than 1 minute)?
Thanks in advance. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Dec 02, 2021 7:32 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
MQ Client software on client platform returns to app a Reason Code indicating failure of the connection. MQRC_CONNECTION_BROKEN (2009) for example. Does the app catch this (and other) errors? Look in the errors directory on the client platform for errors.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Thu Dec 02, 2021 11:48 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
MQ relies on the OS network TCP timeout value. The default is usually 10 minutes. You can change it but it will affect all applications running on that OS.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| rags132 |
| Posted: Thu Dec 02, 2021 12:30 pm Post subject: |
|
|
Newbie
Joined: 02 Dec 2021
Posts: 3
|
Thanks for the response bruce2359.
Actually we are using IBM MQ java API library to make MQPUT calls from the application. Application is receiving the exception/error messages for MQPUT calls only after 4 or 5mins until then all threads are hung state. At least if MQPUT gets timed-out then we could have re-tried the MQPUT operation with new MQ connection.
Last edited by rags132 on Thu Dec 02, 2021 12:47 pm; edited 2 times in total |
|
| Back to top |
|
|
| rags132 |
| Posted: Thu Dec 02, 2021 12:33 pm Post subject: |
|
|
Newbie
Joined: 02 Dec 2021
Posts: 3
|
Thanks for the response Roger.
I will try TCP timeout setting on OS level and check whether MQPUT timeout based on TCP timeout. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Dec 02, 2021 2:24 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| rags132 wrote: |
Thanks for the response bruce2359.
Actually we are using IBM MQ java API library to make MQPUT calls from the application. Application is receiving the exception/error messages for MQPUT calls only after 4 or 5mins until then all threads are hung state. At least if MQPUT gets timed-out then we could have re-tried the MQPUT operation with new MQ connection. |
Please explain more precisely how you are using the IBM MQ Java library to make the MQPUT calls.
Post the commands and parameters here.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Dec 02, 2021 6:51 pm Post subject: Re: MQPUT taking 4 to 5mins when network is disconnected |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| rags132 wrote: |
Our MQ client runs in AWS cloud and actual MQ server runs in on-perm datacenter. For testing purpose, our network team disconnected network between AWS cloud and on-perm datacenter and network cut was only for 10 secs. During that exercise, we have observed MQPUTs were taking 4 to 5minutes to return 2009 error code.
I believe these MQPUT calls were dropped in n/w and not reached queue manager. Can you please suggest is there a way we can make MQPUT calls timed-out (say less than 1 minute)?
Thanks in advance. |
TCP/IP is notoriously bad at informing the owner of the socket that there has been a break in the connection. Because of this, MQ has a number of tools that it uses to determine that this break has happened based on the fact that it can expect to receive traffic on its socket at certain times and when this traffic doesn't appear, it can assume a break. One of these tools is the heartbeat flow. By default the heartbeat flow happens every 300 seconds (5 minutes) and a little additional wait is added to that, because latency.
So if you want to detect network breaks more quickly, reduce the heartbeat interval in use. This is a negotiated attribute of a channel. I think you may only need to reduce it at the SVRCONN, since the Java client sets a really big number by default if my memory serves me correctly.
| Code: |
| ALTER CHANNEL(svrconn-name) CHLTYPE(SVRCONN) HBINT(60) |
You can see the negotiated value using MQSC command:
| Code: |
| DISPLAY CHSTATUS(svrconn-name) HBINT |
while your app is running.
Setting it to 60 seconds will mean it will send a small packet across the channel every 60 seconds, unless there has already been some other traffic. So if your application is sending data more frequently than every 60 seconds, reducing HBINT from 300 seconds down to 60 seconds will make no difference to your traffic, but will make a huge difference to your detection of network breaks.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Dec 03, 2021 7:25 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Did any of the MQPUTs succeed? Any app would have first successfully issued an MQCONNect to the qmgr, then an MQOPEN to the queue, all before an MQPUT.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 03, 2021 4:15 pm Post subject: Re: MQPUT taking 4 to 5mins when network is disconnected |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Morag,
Are you saying a heartbeat that doesn't make the roundtrip will cause MQ to understand their is a network break faster than a failed MQPUT?
Also, you mention that heartbeats won't be sent if the app is otherwise sending data more frequently than HBINT. What if the app is issuing multiple MQPUTs every minute, all are taking 4-5 minutes to time out and HBINT = 60. Does this mean HBINT=60 will not be of any help in these MQPUTs taking so long to timeout?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Dec 03, 2021 5:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
How does the OP know that it was specifically an MQPUT call that failed, and not an MQCONN? I’m not familiar with the IBM-supplied Java samples.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Dec 03, 2021 5:23 pm Post subject: Re: MQPUT taking 4 to 5mins when network is disconnected |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| PeterPotkay wrote: |
| Are you saying a heartbeat that doesn't make the roundtrip will cause MQ to understand their is a network break faster than a failed MQPUT? |
Yes, a heartbeat negotiation in a channel is a promise to send either a "real" flow or a heartbeat flow, every n seconds. If the end sitting in a TCP receive() call doesn't get some data in that time frame (plus a bit for latency) then it will pop out of the receive and complain about the loss of connectivity. The wait time is the heartbeat negotiated value adjusted based on receive time out (another setting), but easy enough to just think about it as heartbeat plus a bit.
| PeterPotkay wrote: |
| Also, you mention that heartbeats won't be sent if the app is otherwise sending data more frequently than HBINT. What if the app is issuing multiple MQPUTs every minute, all are taking 4-5 minutes to time out and HBINT = 60. Does this mean HBINT=60 will not be of any help in these MQPUTs taking so long to timeout? |
I'm not sure how the app can have more than one outstanding MQPUT. Either the app is sending multiple MQPUTs per minute, or the one MQPUT they sent 5 minutes ago, eventually times out. I understood the timeout to mean that the MQPUT call eventually returned with a 2009. Perhaps I misunderstood.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 03, 2021 6:04 pm Post subject: Re: MQPUT taking 4 to 5mins when network is disconnected |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| hughson wrote: |
Yes, a heartbeat negotiation in a channel is a promise to send either a "real" flow or a heartbeat flow, every n seconds. If the end sitting in a TCP receive() call doesn't get some data in that time frame (plus a bit for latency) then it will pop out of the receive and complain about the loss of connectivity. The wait time is the heartbeat negotiated value adjusted based on receive time out (another setting), but easy enough to just think about it as heartbeat plus a bit.
|
But wouldn't the pending MQPUT supersede the send of a heartbeat?
Are you saying the HBINT value is also used by the pending MQPUT to determine how long it should wait before giving up?
| hughson wrote: |
I'm not sure how the app can have more than one outstanding MQPUT.
|
I was thinking one hConn with multiple hObj.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Dec 03, 2021 10:22 pm Post subject: Re: MQPUT taking 4 to 5mins when network is disconnected |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| PeterPotkay wrote: |
| hughson wrote: |
Yes, a heartbeat negotiation in a channel is a promise to send either a "real" flow or a heartbeat flow, every n seconds. If the end sitting in a TCP receive() call doesn't get some data in that time frame (plus a bit for latency) then it will pop out of the receive and complain about the loss of connectivity. The wait time is the heartbeat negotiated value adjusted based on receive time out (another setting), but easy enough to just think about it as heartbeat plus a bit.
|
But wouldn't the pending MQPUT supersede the send of a heartbeat?
Are you saying the HBINT value is also used by the pending MQPUT to determine how long it should wait before giving up? |
Here's how I imagine it works:-
- MQ Client application does MQPUT call
- MQ Client code sends flow to the server-conn asking for MQPUT to be done
- MQ Client code hangs out a TCP receive() call waiting for the response flow from the result of the MQPUT call. This receive only waits for a finite amount of time before giving up - it is this timeout we want to shorten. It is this timeout that the OP is seeing taking 4-5 minutes to timeout.
- Assuming network is not broken, TCP receive() pops out with response, and MQPUT call returns to application
- MQ Client application inspects CompCode and Reason code of completed MQPUT call, then round we go again
| PeterPotkay wrote: |
| hughson wrote: |
I'm not sure how the app can have more than one outstanding MQPUT.
|
I was thinking one hConn with multiple hObj. |
A single hConn can only be doing one MQ API call at a time. You could have multiple hConns all going down one SHARED CONV channel. The client heartbeating (from V7 onwards) is done by a separate thread on behalf of all conversations, i.e. there is only one heartbeat controller, not one per hConn.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
|
|
