| Author |
Message
|
| vsathyan |
 Posted: Fri Aug 28, 2015 8:01 pm Post subject: MQ Listener Denial of Service - How to fix? |
 |
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
Currently, we seem to have a DOS operation on a MQ queue manager listener.
We have an Oracle Service Bus (OSB) application deployed on a server. This service has reconnection logic to MQ.
We have set the MAXINST and MAXINSTC parameters after discussing with the application teams. The DISCINT is also set to 30 minutes for each server connection channel. However, some how the max instances are being reached for a channel, though we have set the values 20% more than the agreed numbers.
The max instances could reach on a channel due to the following reasons.
1. Application making more connections on the channel than set in MAXINSTC. Say it does not close the previous connections properly.
2. Application closes the connection abruptly and it is active at MQ and is released after the DISCINT expires. Until then the connection is held and a new connection attempt is made by the application.
3. May be any other reason I'm missing.
What we suspect it, after the channel instances reach the max, the application still can reach the MQ listener port to open a socket. When it opens a socket connection and sends the request to the queue manager, it is being rejected as the max instances have reached. Now the application is trying to make a connection again and gets rejected. This is happening indefinitely. And also this service is running in multiple servers, the request is coming to the MQ listener at the same time from multiple clients in an infinite loop and finally degrading the performance of the queue manager, eventually abending it.
We are killing the processes and restarting the queue manager. This doesnt seem to be a fair play.
http://www-01.ibm.com/support/docview.wss?uid=swg21670374
The above link talks about TCP (inetd) listener and says the DOS may not happen on a MQ listener (runmqlsr). However, we have runmqlsr process only.
It also says this is fixed in 7.5.0.4 and above. We are at 7.5.0.5, Linux.
Is there a way to prevent the queue manager from these DOS attempts on listener?
Appreciate your advise. Thanks in advance.
Regards,
vsathyan
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004 |
|
| Back to top |
|
 |
| zpat |
| Posted: Fri Aug 28, 2015 11:07 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
The obvious thing is to fix the application so it does not do this excessive reconnection.
You might be able to mitigate the effect by defining another listener on a different port and using that port for this application.
If you actually want to totally block the connections - use CHLAUTH rules or BlockIP2 exit.
What you have is not a DOS attack (which you could simply block), it is a badly designed application.
That IBM technote does not say that a MQ listener cannot suffer any kind of DOS issue, it merely says that particular vulnerability is not present.
I suggest you run a MQ trace and see what sort of MQI calls it is making and tell the application developers or vendor to get it fixed ASAP.
My question is - is there any point in allowing the application to connect at all if it is going to cause these problems? - if not then just block it entirely.
You are looking for some sort of "slow down" control (pacing) - this is not part of the MQ product - although I once suggested it to IBM.
You could code your own exit to detect this client and enforce a wait period between each connection (or modify BlockIP2 source to do this) but I would still recommend a separate listener.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| vsathyan |
| Posted: Sat Aug 29, 2015 12:17 am Post subject: |
|
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
This is a class 1 application in our production environment and we cannot block this application. Though we have a dedicated queue manager for this application (no other apps are using this), we do have separate ports in this queue manager.
Indeed we separated ports for cluster receiver some time ago, when we had only one port and due to this issue, it turned worse when cluster resolution also failed as the port was busy.
The concern is - even if we have separate ports for each server connection channel, once the channel reaches max instances and it saturates a port, the queue manager performance still degrades. Because the connection request reaches the MQ listener, we cannot control the connections at a socket level, it is difficult to stop connection request when it reaches the port and socket is opened. 
In other words, having multiple ports/listeners also is not going to help here because, if one port is saturated by connection requests, it degrades the performance and we saw TCP backlogs on other listeners as well.
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004 |
|
| Back to top |
|
|
| zpat |
| Posted: Sat Aug 29, 2015 1:04 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
Fix the application, raise a top priority call with the vendor or developers.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Aug 29, 2015 2:01 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| zpat wrote: |
| Fix the application, raise a top priority call with the vendor or developers. |
And find out who signed off the load testing. If the application did not show these problems during those tests then something has changed.
Find out if there have been any 'tweaks' to the application recently.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Aug 29, 2015 2:05 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
If the application is home grown, make sure it attempts to close any connection even if an exception is thrown upon connection attempt...
Might also be easier to set your connection pools at application level so that you get a pool exhausted message an no connection attempt is being made until you have an unused connection in your pool....
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tczielke |
| Posted: Sat Aug 29, 2015 4:16 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
For MQ client applications that cause issues by trying to do MQCONN right after they get an MQCONN error (and over and over again), we have the client application build in a small tolerable delay (i.e. 5 seconds) before attempting the next MQCONN when receiving an MQCONN error. Obviously, just turning right around and trying to do an MQCONN when you get a failure and doing this in a tight loop over and over again is not a good idea.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| zpat |
| Posted: Sat Aug 29, 2015 5:18 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
Yes, I tell ours to implement at least 15 second delay before a reconnection attempt.
However clearly there is no need to reconnect often - sounds like the application was coded by monkeys with no understanding of MQ.
Connect once, Open queues once, the perform all the MQGET and MQPUTs you like. Keep the session handle for as long as you want. Only need to close queues and disconnect when closing the application down.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| vsathyan |
| Posted: Sat Aug 29, 2015 5:38 am Post subject: |
|
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
I sincerely thank all of you for your valuable time and inputs. I'm trying to reproduce the problem locally using thread(s) to make infinite connection attempts on the MQ listener port.
I'll setup a meeting and work with the application architects to fine tune it wherever possible. I'll update this thread once the issue has a permanent fix.
Regards,
vsathyan
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004 |
|
| Back to top |
|
|
| zpat |
| Posted: Sat Aug 29, 2015 6:32 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
Fine tune?
Surely you mean fix the terrible and basic design flaw?
This is not tuning. This is basic understanding of how to code MQI applications.
You do not connect to the QM for each message.
You do not open the queue for each message.
You do not repeatedly issue MQGET without MQGMO_WAIT on empty queues.
In many cases just adding MQGMO_WAIT with say a 30 second max wait time will help. This does not add latency when messages are available, but it reduces the monumental stupidity of repeatedly and rapidly accessing an empty queue (compounded of course by reconnecting each time).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| vsathyan |
| Posted: Sat Aug 29, 2015 6:44 am Post subject: |
|
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
Hi zpat,
As i mentioned earlier, this is OSB (Oracle Service Bus) service connecting to MQ. It may be connecting to MQ using an adapter, a client whatever, needs to be reviewed thoroughly.
I will do a deep dive on how the connections are being made to MQ, what is the error handling when connections are full on a channel, what is the delay in reconnection logic etc. I'll surely discuss all the aspects listed here with the app teams, we will reproduce the issue, fix it and then update this thread with the results. 
Thanks again,
vsathyan
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004 |
|
| Back to top |
|
|
| vsathyan |
| Posted: Sat Aug 29, 2015 6:47 am Post subject: |
|
|
Centurion
Joined: 10 Mar 2014
Posts: 121
|
| zpat wrote: |
(compounded of course by reconnecting each time). |
The problem more here is trying to open a socket listener infinitely without a delay on a MQ listener port when the max channel instances have been reached, rather than reconnecting each time.
The former is worse than the latter.
Regards,
vsathyan
_________________
Custom WebSphere MQ Tools Development C# & Java
WebSphere MQ Solution Architect Since 2011
WebSphere MQ Admin Since 2004 |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Aug 29, 2015 7:37 am Post subject: Re: MQ Listener Denial of Service - How to fix? |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9399
Location: US: west coast, almost. Otherwise, enroute.
|
| vsathyan wrote: |
| Currently, we seem to have a DOS operation on a MQ queue manager listener. |
A Denial of Service attack presumes an evil intent of a hacker. As my worthy colleague pointed out, what you seem to have is a poorly written/configured internal application.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| tczielke |
| Posted: Sat Aug 29, 2015 7:42 am Post subject: Re: MQ Listener Denial of Service - How to fix? |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
| bruce2359 wrote: |
| vsathyan wrote: |
| Currently, we seem to have a DOS operation on a MQ queue manager listener. |
A Denial of Service attack presumes an evil intent of a hacker. As my worthy colleague pointed out, what you seem to have is a poorly written/configured internal application. |
Maybe a Friendly Fire attack? 
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 31, 2015 4:23 am Post subject: Re: MQ Listener Denial of Service - How to fix? |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| tczielke wrote: |
| bruce2359 wrote: |
| vsathyan wrote: |
| Currently, we seem to have a DOS operation on a MQ queue manager listener. |
A Denial of Service attack presumes an evil intent of a hacker. As my worthy colleague pointed out, what you seem to have is a poorly written/configured internal application. |
Maybe a Friendly Fire attack? |
Problem Exists Between Programmer And Production?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
