| Author |
Message
|
| tleichen |
 Posted: Fri May 02, 2008 12:11 pm Post subject: Problem with JMS pub/sub applications |
 |
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
We have some JMS pub/sub applications that are (supposedly) doing small infrequent messaging over a period of time. From my understanding of the particular JMS app (which is almost nothing) it is supposed to run only once in a while when data is available; and then process it. It is not a long-running application. BTW, these are JMS apps connecting over a client channel.
The problem we are seeing on the channel display is that we have multiple connection threads that appear to be long-running. Some have been going for days. When I watch the channel display, these long-running threads are sending and receiving much higher volumes of data than the other open connections. At a refresh rate of 15 seconds I am seeing 12 messages going through in each interval. The bytes sent per interval is 6K, which is an average of 512 bytes per message. We've been told over and over again by the application people that their apps do not do that. But, the channel status does not lie.
My problem is that despite the fact that I can look at these threads and see what they're doing, I can't do anything about it, short of shutting down the queue manager. Is there any way of controlling an individual connection, i.e., pause it, stop it, kill it, etc? 
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Fri May 02, 2008 12:14 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
The app is not sending messages, this is the connection pool keeping the channels going so it has an established connection ready to go when a message does arrive.
The messages count on a client channel is very misleading. Do an MQOPEN from a client and the message count will go up by 2 - the first is the MQOPEN request, the 2nd is the MQOPEN results.
Do 1000 MQGETs all of which return a 2033. Your client channels message count will be 2000 higher.
Now that you know that, is there any reason you want to shutdown these channels?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| tleichen |
| Posted: Fri May 02, 2008 12:31 pm Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
So, in order to keep the channels up, it needs to send and receive a data packet of 512 bytes practically every second?!? 
Also, there are many other connections that are just as long-lived, but have practically no volume and appear to be practically nonactive as far as data flow. I might believe that these are what you had mentioned before, but these high-volume things have me worried. The flow is at a pretty much constant rate. I figured out that this would be putting out over 2GB of data per day in each direction on each of these "runaway" connections. Currently we have four of them and three out of the four are from the same client(IP). That's an awful lot of data just to keep the channel up!!
Another thing I noticed is that the channel substate on the high volume ones always reads "In MQGET", while the others all say "Receiving". What is the significance of this?
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri May 02, 2008 1:16 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
A channel connection that does and MQCONN an MQOPEN and then waits in the pool will not be sending any "messages" back and forth. A channel used by an MDB that is constantly issuing MQGETs to see if there are any new messages will be. There are settings in the JMS config giles that determine how often you poll and how many connections you have overall.
| Quote: |
| I figured out that this would be putting out over 2GB of data per day in each direction on each of these "runaway" connections. Currently we have four of them and three out of the four are from the same client(IP). That's an awful lot of data just to keep the channel up!! |
Is this a problem?
Yes, JMS is a resource PIG. Turn an MQ API exit on a QM that supports lots of JMS connections and be amazed at what your QM is actually dealing with. (lots of polling, lots of opening and closing of queues as well as the QM object, lots of MQINQ calls, etc)
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bower5932 |
| Posted: Sat May 03, 2008 1:45 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
|
| Back to top |
|
|
| tleichen |
| Posted: Mon May 05, 2008 6:35 am Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
Thanks for both of the responses, guys.
I'm still confused at what it means though when there is a connection handle that has passed a small number of messages and the last message was five days ago and the connection is still there! What causes this? 
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 05, 2008 8:46 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| PeterPotkay wrote: |
| A channel connection that does an MQCONN an MQOPEN and then waits in the pool will not be sending any "messages" back and forth. |
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| tleichen |
| Posted: Mon May 05, 2008 11:42 am Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
That makes sense, Peter. But, last Friday, these "heavy hitters" were doing 12 messages per 15-second cycle like clockwork. Now, they are each doing 138 per cycle. This volume increase is concerning us, as we have no way to account for it yet. Any thoughts?
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 05, 2008 1:04 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Are you saying that when you do a Channel Status for one of these SVRCONN channel instances you were seeing 12 "messages" every 15 seconds and now you see 138 "messages" every 15 seconds?
If yes, either the app is sending more real application messages thru the queues or someone tweaked a JMS setting and the JMS app is polling the queue a lot more frequently than it used to. JMS does support the concept of MQGET with wait. There is no need to tell your MDBs to poll once a second if you want to pick up application messages ASAP.
Actually another more likely scenario is you are using Message Selectors and the queue depth is rising with stale messages. When JMS says give me a message with "ABC123" in it its actually doing a browse, getting a copy of the 1st message, inspecting at the client side, and if doesn't match, doing a browse next. As the q depth grows if your matching message is at the bottom of the q each get will get slower and slower and s-l-o-w-e-r as you you to do more and more browses just to get the one message you want. You can eliminate this behavior if you are getting by MQ Message ID and Correl ID. You have to use the 48 hex representation of either ID (or both together) and prefix it with "ID:". This makes the selective get occur at the QM side like a typical selective MQGET by a non JMS app. The QM does the selection for you off of its indexed queue so its MUCH faster.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| tleichen |
| Posted: Tue May 06, 2008 10:31 am Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
Yes, I think that's exactly what's happening! Is there any way to get rid of those "stale messages"? I tried the Cleanup utility, but it did not seem to do anything.
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue May 06, 2008 10:52 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
It should be trivial for an MQ Admin to remove some messages from a queue.
Tell the app to start using Expiry to avoid this situation.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
