| Author |
Message
|
| eputsato |
 Posted: Tue Apr 14, 2009 5:32 am Post subject: Slow Subscribers |
 |
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
Is there any way I can achieve the following:
I have a number of non durable subscribers for a certain topic. This is some sort of "state topic", which means, that the subscriber is only required to receive the latest message from the the topic subscription. If he misses several previous updates it's not a problem.
When the publisher's rate is much more than the subscriber's it leads to growing of the subscriber's topic queue.
Ideally i would like not to have this queue for topic subscription at all...
I tried setting the model non-durable queue's size limit to 1, but in this case I may receive not necessarily the last message as the last one may simply not be pushed to the queue as it was full at the moment.
Could someone help me? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Apr 14, 2009 5:43 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Why are you not using retained publications and having the subscribers request them?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| eputsato |
| Posted: Tue Apr 14, 2009 6:51 am Post subject: |
|
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
I'm already using the retained publications....
I wrote a small app publishing 1,2,3...n as message text in an infinite loop and the other application (XMS) doing receives with a delay of 1sec, even when the publisher is already shutdown, the subscriber continues to receive his messages (up to the maximum length of his model queue)
while (!kbhit())
{
xms::Message *msg = NULL;
msg = consumer.receive();
printf("%s\t\t%s\r\n",msg->getJMSDestination().getName().c_str(), ((xms::TextMessage *)msg)->getText().c_str());
Sleep(100);
}
So finally he receives all the messages the subscriber sent while their number is less than subscribers queue.
However i would like the client messages in this case to look like:
1,100,200,500,10000
if the sender has sent 1,2,3,...,99,100,101,...,199,200,201
And the crucial point is to get the value 10 000 at the end. Which in fact was the last publication made by publisher. So no matter what my speed is I always need to get the most recent message available.
PS>The subscriber is still not durable
Last edited by eputsato on Wed Apr 15, 2009 12:39 am; edited 1 time in total |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Apr 14, 2009 7:00 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| What happens if the subscriber closes and re-opens the queue during the wait period? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Apr 14, 2009 7:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I'm confused here. You have an application which publishes status type messages, retained because only the last is really relevant but no apparent mechanism for clearing out "old" publications. You have a non-durable subscriber(s) which subscribe to this publication, who receive all of the publications, including the old ones which are not cleared off.
Have I sumarised your situation?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Apr 14, 2009 9:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You only need retained publications if you need to have a newly registered subscriber receive the last publication made, rather than having it wait for the next publication made.
All MQ based publications are delivered to queues, at least prior to MQ v7.
There's no automatic mechanism to remove a prior publication from a queue and replace it with a new publication to the queue.
There is an automatic mechanism to expire messages. |
|
| Back to top |
|
|
| eputsato |
| Posted: Tue Apr 14, 2009 9:09 pm Post subject: |
|
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
| Vitor wrote: |
I'm confused here. You have an application which publishes status type messages, retained because only the last is really relevant but no apparent mechanism for clearing out "old" publications. You have a non-durable subscriber(s) which subscribe to this publication, who receive all of the publications, including the old ones which are not cleared off.
Have I sumarised your situation? |
Yes, almost like this. In fact, I do not receive the "OLD-OLD" publications, made one year ago, instead i receive all the publications made while subscriber was connected.
But I need to get only the latest one even if there were another 10 during the 5 second period the subscriber was processing previous message.
This is called state. Each topic has it's state, i.e. the last publication n topic. I always need that last state. It's more important to get the last state and to lose all past state transitions, if they were not received because of the subscriber's being slow. This is not a messenger app. So I want to skip messages if they failed to be received in time and another updates happened since then on this topic. So when I do receive(), I need to get the very last topic message available. Instead I get the oldest one which i delayed to receive even though i was connected to topic. receive() takes data from the subscribers queue, but i need him to take data directly from the topic - to receive the last and only current retained message. Otherwise it breaks th whole idea of retained messages.
Should I clear the old publications manually? Who clears the subscribers queue?
And it's so weird and ineffective to reopen subscription every time I receive data. I don't think that it's correct way regarding system design.
Last edited by eputsato on Wed Apr 15, 2009 12:34 am; edited 4 times in total |
|
| Back to top |
|
|
| eputsato |
| Posted: Tue Apr 14, 2009 9:11 pm Post subject: |
|
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
| mqjeff wrote: |
You only need retained publications if you need to have a newly registered subscriber receive the last publication made, rather than having it wait for the next publication made.
All MQ based publications are delivered to queues, at least prior to MQ v7.
There's no automatic mechanism to remove a prior publication from a queue and replace it with a new publication to the queue.
There is an automatic mechanism to expire messages. |
I'm using MQ7. So is there a way for me to avoid using queues?
Or could you clarify - is there a mechanism for removing oldest messages from subscribers queue when size limit is reached/ As I've mentioned before, I can limit the queue size for subscribers to 1. But in this way I will get the oldest messages rather then the new.
Expiry by the way is based on time, but in my case it should base on the size of queue.
If there's a new publication on topic, then expire the previous one. Between them there could be a 5 hours delay (sometimes)
Last edited by eputsato on Wed Apr 15, 2009 12:42 am; edited 3 times in total |
|
| Back to top |
|
|
| eputsato |
| Posted: Tue Apr 14, 2009 9:16 pm Post subject: |
|
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
i need to publish stock prices. And the client doesn't need all the prices. There may be 10 per second(!) for each stock.
Instead he needs to see only the latest price to make a deal. And if he's ten times slower than the price feed rate I wouldn't like him finally after 10 minutes to continue receiving prices of the first minute, as his queue already has 10 000 old prices.
For example Sun JMS has two options: to remove oldest message from the queue or to remove newest.
MQ7 now works like "remove newest". It simply doesn't let the new message to come to subscriber's queue if its full. Instead it should remove the oldest one and gain space for the newest. |
|
| Back to top |
|
|
| eputsato |
| Posted: Wed Apr 15, 2009 7:07 am Post subject: |
|
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
Today I tried to limit the model queue size to 0 (no queue at all), however the system didn't let me do this.
So I still hope to receive any helpful suggestions from MQ guys. Thank you. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Apr 15, 2009 3:16 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You have been suggested to follow the path of message expiration. What have you done there? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Apr 15, 2009 3:29 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
To reiterate the scenario if you enable expiry, here's what will happen.
You publish message A, B, and C.
A slow subscriber reads A and starts to process.
Time passes.
You publish message D,E, F and G.
Time passes.
Message B, C, D and E expire. The queue now has only F and G on it.
The subscriber finishes processing A and goes to get the next message. It's F. |
|
| Back to top |
|
|
| eputsato |
| Posted: Wed Apr 15, 2009 9:25 pm Post subject: |
|
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
| fjb_saper wrote: |
| You have been suggested to follow the path of message expiration. What have you done there? |
And I wrote above to mqjeff
"
Expiry by the way is based on time, but in my case it should base on the size of queue.
If there's a new publication on topic, then expire the previous one. Between them there could be a 5 hours delay (sometimes)
" |
|
| Back to top |
|
|
| eputsato |
| Posted: Wed Apr 15, 2009 9:28 pm Post subject: |
|
|
Novice
Joined: 14 Apr 2009
Posts: 20
|
| mqjeff wrote: |
To reiterate the scenario if you enable expiry, here's what will happen.
You publish message A, B, and C.
A slow subscriber reads A and starts to process.
Time passes.
You publish message D,E, F and G.
Time passes.
Message B, C, D and E expire. The queue now has only F and G on it.
The subscriber finishes processing A and goes to get the next message. It's F. |
What happens if the time before the next price tick is 5 hours? It may happen on markets close. Will this lead to not being able to receive the last (and already expired) message?
Anyway this method is not accurate as in this case I won't be guaranteed to receive last and only last message. And how expire works for retained messages?
Imagine the following case:
the publisher posts 1,2,3,4
The subscriber receives 1 then he has a delay, then he manages to get 3, but by the time he finishes the 4 is already expired, so though he's ready for futher processing he won't get 4. So the last quote for him will be 3, but not 4. And it will stay like this unless he reconnects or the new quote arrives. So it's not hte way one may expect it to work |
|
| Back to top |
|
|
| sumit |
| Posted: Thu Apr 16, 2009 12:27 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Follow this scenario- you have 2 local queues ql1 and ql2 at subscriber's end. The end site application will interact with the ql2 only. ql1 is responsible to receive messages from publisher.
May be you can write a code for this or use standard utility (let's call it put-get utility). The way how it can work is:
1. A message comes to ql1 and triggeres put-get utility.
2. put-get checks if a message is there in ql2.
3. If NO, get the message from ql1 and put the same in ql2.
4. If YES, delete the message from ql2.
5. then, get message from ql1 and put the same in ql2.
In this way, you can always get the latest message for your end side users.
_________________
Regards
Sumit |
|
| Back to top |
|
|
|
|
