ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » Slow Subscribers

Post new topic  Reply to topic Goto page 1, 2, 3  Next
 Slow Subscribers « View previous topic :: View next topic » 
Author Message
eputsato
PostPosted: Tue Apr 14, 2009 5:32 am    Post subject: Slow Subscribers Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

Is there any way I can achieve the following:

I have a number of non durable subscribers for a certain topic. This is some sort of "state topic", which means, that the subscriber is only required to receive the latest message from the the topic subscription. If he misses several previous updates it's not a problem.
When the publisher's rate is much more than the subscriber's it leads to growing of the subscriber's topic queue.
Ideally i would like not to have this queue for topic subscription at all...

I tried setting the model non-durable queue's size limit to 1, but in this case I may receive not necessarily the last message as the last one may simply not be pushed to the queue as it was full at the moment.

Could someone help me?
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Apr 14, 2009 5:43 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Why are you not using retained publications and having the subscribers request them?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
eputsato
PostPosted: Tue Apr 14, 2009 6:51 am    Post subject: Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

I'm already using the retained publications....

I wrote a small app publishing 1,2,3...n as message text in an infinite loop and the other application (XMS) doing receives with a delay of 1sec, even when the publisher is already shutdown, the subscriber continues to receive his messages (up to the maximum length of his model queue)

while (!kbhit())
{
xms::Message *msg = NULL;
msg = consumer.receive();
printf("%s\t\t%s\r\n",msg->getJMSDestination().getName().c_str(), ((xms::TextMessage *)msg)->getText().c_str());
Sleep(100);
}

So finally he receives all the messages the subscriber sent while their number is less than subscribers queue.

However i would like the client messages in this case to look like:
1,100,200,500,10000
if the sender has sent 1,2,3,...,99,100,101,...,199,200,201

And the crucial point is to get the value 10 000 at the end. Which in fact was the last publication made by publisher. So no matter what my speed is I always need to get the most recent message available.

PS>The subscriber is still not durable


Last edited by eputsato on Wed Apr 15, 2009 12:39 am; edited 1 time in total
Back to top
View user's profile Send private message
zpat
PostPosted: Tue Apr 14, 2009 7:00 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5866
Location: UK

What happens if the subscriber closes and re-opens the queue during the wait period?
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Apr 14, 2009 7:01 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

I'm confused here. You have an application which publishes status type messages, retained because only the last is really relevant but no apparent mechanism for clearing out "old" publications. You have a non-durable subscriber(s) which subscribe to this publication, who receive all of the publications, including the old ones which are not cleared off.

Have I sumarised your situation?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Apr 14, 2009 9:09 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

You only need retained publications if you need to have a newly registered subscriber receive the last publication made, rather than having it wait for the next publication made.

All MQ based publications are delivered to queues, at least prior to MQ v7.

There's no automatic mechanism to remove a prior publication from a queue and replace it with a new publication to the queue.

There is an automatic mechanism to expire messages.
Back to top
View user's profile Send private message
eputsato
PostPosted: Tue Apr 14, 2009 9:09 pm    Post subject: Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

Vitor wrote:
I'm confused here. You have an application which publishes status type messages, retained because only the last is really relevant but no apparent mechanism for clearing out "old" publications. You have a non-durable subscriber(s) which subscribe to this publication, who receive all of the publications, including the old ones which are not cleared off.

Have I sumarised your situation?


Yes, almost like this. In fact, I do not receive the "OLD-OLD" publications, made one year ago, instead i receive all the publications made while subscriber was connected.

But I need to get only the latest one even if there were another 10 during the 5 second period the subscriber was processing previous message.

This is called state. Each topic has it's state, i.e. the last publication n topic. I always need that last state. It's more important to get the last state and to lose all past state transitions, if they were not received because of the subscriber's being slow. This is not a messenger app. So I want to skip messages if they failed to be received in time and another updates happened since then on this topic. So when I do receive(), I need to get the very last topic message available. Instead I get the oldest one which i delayed to receive even though i was connected to topic. receive() takes data from the subscribers queue, but i need him to take data directly from the topic - to receive the last and only current retained message. Otherwise it breaks th whole idea of retained messages.

Should I clear the old publications manually? Who clears the subscribers queue?

And it's so weird and ineffective to reopen subscription every time I receive data. I don't think that it's correct way regarding system design.


Last edited by eputsato on Wed Apr 15, 2009 12:34 am; edited 4 times in total
Back to top
View user's profile Send private message
eputsato
PostPosted: Tue Apr 14, 2009 9:11 pm    Post subject: Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

mqjeff wrote:
You only need retained publications if you need to have a newly registered subscriber receive the last publication made, rather than having it wait for the next publication made.

All MQ based publications are delivered to queues, at least prior to MQ v7.

There's no automatic mechanism to remove a prior publication from a queue and replace it with a new publication to the queue.

There is an automatic mechanism to expire messages.


I'm using MQ7. So is there a way for me to avoid using queues?

Or could you clarify - is there a mechanism for removing oldest messages from subscribers queue when size limit is reached/ As I've mentioned before, I can limit the queue size for subscribers to 1. But in this way I will get the oldest messages rather then the new.

Expiry by the way is based on time, but in my case it should base on the size of queue.
If there's a new publication on topic, then expire the previous one. Between them there could be a 5 hours delay (sometimes)


Last edited by eputsato on Wed Apr 15, 2009 12:42 am; edited 3 times in total
Back to top
View user's profile Send private message
eputsato
PostPosted: Tue Apr 14, 2009 9:16 pm    Post subject: Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

i need to publish stock prices. And the client doesn't need all the prices. There may be 10 per second(!) for each stock.
Instead he needs to see only the latest price to make a deal. And if he's ten times slower than the price feed rate I wouldn't like him finally after 10 minutes to continue receiving prices of the first minute, as his queue already has 10 000 old prices.
For example Sun JMS has two options: to remove oldest message from the queue or to remove newest.

MQ7 now works like "remove newest". It simply doesn't let the new message to come to subscriber's queue if its full. Instead it should remove the oldest one and gain space for the newest.
Back to top
View user's profile Send private message
eputsato
PostPosted: Wed Apr 15, 2009 7:07 am    Post subject: Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

Today I tried to limit the model queue size to 0 (no queue at all), however the system didn't let me do this.

So I still hope to receive any helpful suggestions from MQ guys. Thank you.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Apr 15, 2009 3:16 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

You have been suggested to follow the path of message expiration. What have you done there?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
mqjeff
PostPosted: Wed Apr 15, 2009 3:29 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

To reiterate the scenario if you enable expiry, here's what will happen.

You publish message A, B, and C.

A slow subscriber reads A and starts to process.

Time passes.

You publish message D,E, F and G.

Time passes.

Message B, C, D and E expire. The queue now has only F and G on it.

The subscriber finishes processing A and goes to get the next message. It's F.
Back to top
View user's profile Send private message
eputsato
PostPosted: Wed Apr 15, 2009 9:25 pm    Post subject: Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

fjb_saper wrote:
You have been suggested to follow the path of message expiration. What have you done there?


And I wrote above to mqjeff
"
Expiry by the way is based on time, but in my case it should base on the size of queue.
If there's a new publication on topic, then expire the previous one. Between them there could be a 5 hours delay (sometimes)
"
Back to top
View user's profile Send private message
eputsato
PostPosted: Wed Apr 15, 2009 9:28 pm    Post subject: Reply with quote

Novice

Joined: 14 Apr 2009
Posts: 20

mqjeff wrote:
To reiterate the scenario if you enable expiry, here's what will happen.

You publish message A, B, and C.

A slow subscriber reads A and starts to process.

Time passes.

You publish message D,E, F and G.

Time passes.

Message B, C, D and E expire. The queue now has only F and G on it.

The subscriber finishes processing A and goes to get the next message. It's F.

What happens if the time before the next price tick is 5 hours? It may happen on markets close. Will this lead to not being able to receive the last (and already expired) message?

Anyway this method is not accurate as in this case I won't be guaranteed to receive last and only last message. And how expire works for retained messages?

Imagine the following case:
the publisher posts 1,2,3,4
The subscriber receives 1 then he has a delay, then he manages to get 3, but by the time he finishes the 4 is already expired, so though he's ready for futher processing he won't get 4. So the last quote for him will be 3, but not 4. And it will stay like this unless he reconnects or the new quote arrives. So it's not hte way one may expect it to work
Back to top
View user's profile Send private message
sumit
PostPosted: Thu Apr 16, 2009 12:27 am    Post subject: Reply with quote

Partisan

Joined: 19 Jan 2006
Posts: 398

Follow this scenario- you have 2 local queues ql1 and ql2 at subscriber's end. The end site application will interact with the ql2 only. ql1 is responsible to receive messages from publisher.

May be you can write a code for this or use standard utility (let's call it put-get utility). The way how it can work is:

1. A message comes to ql1 and triggeres put-get utility.
2. put-get checks if a message is there in ql2.
3. If NO, get the message from ql1 and put the same in ql2.
4. If YES, delete the message from ql2.
5. then, get message from ql1 and put the same in ql2.

In this way, you can always get the latest message for your end side users.
_________________
Regards
Sumit
Back to top
View user's profile Send private message Yahoo Messenger
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2, 3  Next Page 1 of 3

MQSeries.net Forum Index » General IBM MQ Support » Slow Subscribers
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.