| Author |
Message
|
| jshailes |
 Posted: Mon Sep 07, 2009 6:40 am Post subject: Queue Emptying Unexpectedly |
 |
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
I'm having a few problems retrieving messages using jms from our queue.
We have a queue that is populated with messages from an external site through mqipt. The queue depth is 100,000. The server is struggling to keep up with the 4000 messages/min as we only read 500/min but it doesn't seem to explain the following problem:
UpTime (mins),MessagesRecieved,QueueDepth
1,4000,3500
2,8000,7000
3,12000,10500
4,16000,14000
5,20000,3200
For some unknown reason the queue depth drops significantly after 4-5 mins. Does anyone have any suggestions for what might be causing this? |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Mon Sep 07, 2009 6:48 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
The messages are expiring, or the messages are being consumed by another application.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| sumit |
| Posted: Mon Sep 07, 2009 6:50 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Is there any expiry interval set in the messages?
_________________
Regards
Sumit |
|
| Back to top |
|
|
| jshailes |
| Posted: Mon Sep 07, 2009 6:57 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
We talked about that and dismissed it on the basis that messages would expire on a linear basis as opposed to lots of messages being removed after 4 minutes?
As above it seems unlikely that we have an application ourselves that reads lots of messages such large intervals - I'm sure we've tried this but I'll restart the server just to make sure. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Sep 07, 2009 7:04 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| We talked about that and dismissed it |
Perhaps too quickly. Expired messages are purged from queues when the responsible qmgr component gets around to it.
EDIT: More specifically, expired messages are not deleted from queues as they expire; rather, they are deleted when the process that deletes expired messages comes to life - which is not linear. The subject of when the process comes to life has been discussed here many times.
In any case, a consuming application can never MQGET an expired message.
You have described symptoms. So, what exactly is the problem? Server too slow consuming messages?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jshailes |
| Posted: Mon Sep 07, 2009 7:26 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
wuw, ok finally making sense  thanks very much.
Our problem is definitely that we are consuming messages too slowly. Our single threaded app is running on tomcat. We pull 1 message from queue, do a bunch of processing on it and then loop. Expiry is 100s. We only get 12% of messages since the processing takes too long. The only solution I can think of is to move to an app server? |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Mon Sep 07, 2009 7:34 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| jshailes wrote: |
| The only solution I can think of is to move to an app server? |
Maybe, depends where your delay is. If your app its constantly opening connections to the QM time and time again then this will impact your performance and so yes you will benefit from the scalability and pooling features provided by an app server, but its worth proving this before you move as it would be a shame to buy the app server just to realise the delay was in your processing!
If this data is important then why the short expiry? If you are expiring the messages and you dont process them in the expiry period then im guessing its not worth processing them at all (else maybe look at making them never expire / expire over a longer period)?
Last edited by WMBDEV1 on Mon Sep 07, 2009 7:37 am; edited 1 time in total |
|
| Back to top |
|
|
| sumit |
| Posted: Mon Sep 07, 2009 7:36 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| jshailes wrote: |
| Expiry is 100s. |
There you go. Set the a suitable time interval and try again.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Sep 07, 2009 7:46 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| sumit wrote: |
| jshailes wrote: |
| Expiry is 100s. |
There you go. Set the a suitable time interval and try again. |
| jshailes wrote: |
| We have a queue that is populated with messages from an external site through mqipt |
The poster may not have that ability...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Sep 07, 2009 7:46 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Expiry is a period of time expressed in tenths of a second, set by the application that puts the message.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jshailes |
| Posted: Mon Sep 07, 2009 8:02 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
Thanks Bruce - I should've come back to the forum before trying to work that out!
We don't have control over the expiry, however the messages could be read and put onto a separate queue - this doesn't really solve our problem since we will be receiving messages at a rate of 4000/s 24/7. We simply need to process quicker.
The reason I thought an app server was the only way forward was that we could have multiple processes doing the work?
At the moment we have an app in tomcat - since it takes almost no time to read a message from the queue but approx 500ms to process. We can't significantly minimise the processing time since there is a lot of logic involved, so my thinking was that an app server can have the application executing many times as opposed to modifying the application to use threading? |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Sep 07, 2009 8:11 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
As an interim at least, either a multi-threaded application, or a number of instances of a non-threaded application, may solve the problem of expiry, but what other resources are exposed as part of the processing cycle? Why does it take 500ms to process a message? You may resolve your GET issue, but will you then be moving your bottleneck elsewhere?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| jshailes |
| Posted: Mon Sep 07, 2009 8:20 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
The messages are of trains moving around a rail network so we need to make associations between messages in order to create the journey. The process of creating this journey, calculating its lateness after each message, and various other info is costly on resources - currently approximately 500ms. These calculations are our bottleneck.
I'm not all that interested in an interim, I need to get started on a long term solution. Was my reasoning of the app server reasonable? I'm just concerned about sharing resources - the separate processes need to share the same memory space - is this a feature of an app server? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Sep 07, 2009 8:36 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Was my reasoning of the app server reasonable? I'm just concerned about sharing resources - the separate processes need to share the same memory space - is this a feature of an app server? |
First: Is the app server sufficiently provisioned? That is, does the app server have enough cpus? Enough memory? Enough disk? Can the box be tuned?
Your app developers will know better if their app can tolerate multiple instances of the same process, and where the app cannot share resources.
I'm going to guess that there is a database behind all this that keeps track (?) of where the trains are. Can the db tolerate multiple processes?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jshailes |
| Posted: Mon Sep 07, 2009 8:50 am Post subject: |
|
|
Apprentice
Joined: 18 May 2009
Posts: 31
|
Thanks for your reply.
As you've probably guessed I have very limited knowledge of j2ee but unfortunately am responsible for the design of the app - we have begun implementation but are no means a long way through. The server we currently have is limited - core 2, 4gb, 500gb but we have no issues getting a proper system when required.
We are currently using hibernate to interface to the db - I am hoping this can be shared between processes. Also the objects that are not persisted need to be shared between processes. I think I should probably call it a day posting on here since I really need to get a deeper understanding.
Many thanks for everyones help, it really has saved us hours of surmising. |
|
| Back to top |
|
|
|
|
