ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ API Support » Efficiency of processing messages in two phases

Post new topic  Reply to topic
 Efficiency of processing messages in two phases « View previous topic :: View next topic » 
Author Message
mqdogsbody
PostPosted: Tue May 26, 2015 2:40 am    Post subject: Efficiency of processing messages in two phases Reply with quote

Acolyte

Joined: 01 Jun 2010
Posts: 71

Current we use MQ in a simple way – getting and consuming each message in separate work-unit in a FIFO manner.

A colleague is seeking input on the feasibility of changing to a scheme whereby messages are browsed but not consumed from the queue, and then sometime later removed/consumed from the queue based on their MessageId (or other properties read from the message when it is originally browsed). The application would be responsible for retaining the MessageIds of browsed and processed messages until they have been consumed/deleted from the queue and it might be a considerable time later that the message is eventually consumed/deleted, even up to the end-of-day.

From research and manual pages the basic mechanisms appear to exist in terms of being able to MQGET’ing messages that match a MessageId. However given that the general thrust of the documentation seems to be based on obtaining the ‘First message that matches MsgId’, My colleague is concerned about the scalability of application issuing MQGET calls for MessageIds that are a long way from the front of the queue. Can anyone advise of the relative performance of MQGET’ing messages that are far from the front of the queue?

Similarly, in a recovery scenario, my colleague is concerned about the time it might take to issuing ‘Browse-next’ operations to wind-forward past possibly 10’s of thousands of previously browsed and processed messages to place in the queue where the next un-processed message can be found if this is not generally the way MQ is supposed to be used. Can anyone advise on the sensibility of such an approach?
_________________
-- mqDB --


Last edited by mqdogsbody on Tue May 26, 2015 3:32 am; edited 1 time in total
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue May 26, 2015 3:02 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

You've proposed a solution - browsing messages, storing message IDs, coming back later to get them by message ID.

But you haven't explained why you would want to do this. What is the problem you are trying to solve?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
pmb999
PostPosted: Tue May 26, 2015 4:04 am    Post subject: Re: Efficiency of processing messages in two phases Reply with quote

Newbie

Joined: 16 Jul 2009
Posts: 5

The time that takes to fully process a message is relatively long, where that processing entails writing to a complex SQL database. So the problem is to decouple the (partially) processing of a message and sending a response from the eventual completion of persisting data to the SQL database, and then removing the originating MQ message. In effect, we'd like to be able to keep the input message on MQ until much later. Also, to further aid the performance and response-time of the application, we'd like to minimise the MQ overhead incurred before sending a response, hence the preference for simply leaving it on the received queue, rather than redirecting it to another for later removal.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue May 26, 2015 4:45 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

What's wrong with moving the message to a different queue for later removal? It keeps your main process running smooth, fast (in memory hand off) and efficiently.

Leaving the message on the queue will on the contrary maximize the impact on the MQ side. Reading constantly the queue, having to load the catalog, forcing incoming messages to be written to disk, etc...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
pmb999
PostPosted: Fri Jun 05, 2015 8:21 am    Post subject: Re-queueing message to different queue...? Reply with quote

Newbie

Joined: 16 Jul 2009
Posts: 5

If, as the last posting suggests, we re-queue the received message to a different queue, is there any particular support for doing that, or is it a case of issueing an MQ PUT to a different queue to on-forward the incoming message?

I was wondering whether MQ had any 'fast-path' like feature to re-queue a message on a different queue without needing to transfer the message into and out of the application?
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri Jun 05, 2015 9:14 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Is it clear that you don't want to use syncpoint?

I know of customers that are using a browse followed by a later get. They get reasonable performance.

But depending on your timing, you should probably just look at holding the MQ message in a syncpoint.
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Fri Jun 05, 2015 10:58 am    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

Hi,

Well, here's a solution but it involves a commercial product (since you do not want to handle the movement of a message from queue A to queue B).

Let's say your original input queue is 'Q.A'. Create a new queue called 'Q.B'.

- Purchase and install MQ Message Replication (MQMR)
- Configure MQMR to replicate messages from 'Q.A' to 'Q.B'
- Have your application continue to perform destructive MQGets (not browse) against 'Q.A' (the original queue) and save the MessageID as you mentioned above.
- Have your other slow backend system process messages from 'Q.B' using the saved MessageID.

Problem solved. Your original queue ('Q.A') will always be at or near zero for queue depth, your application does not need to copy/forward messages to another queue and your slow backend system can process the messages whenever it gets to it.

Personally, I don't think you need to save the MessageID because the slow backend application will simply get the next message off 'Q.B' and process it. Both queue 'Q.A' (for your speedy frontend application) and queue 'Q.B' (for the slow backend application) get the EXACT same message and in the EXACT same order. So, just have each application do their thing.

Note: When MQMR replicates a message both the message data and the message's MQMD structure will be cloned. (i.e. PutTime, MessageId, CorrelId, UserId, etc..)

Food for thought.

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
pmb999
PostPosted: Sat Jun 06, 2015 5:10 am    Post subject: Thanks Reply with quote

Newbie

Joined: 16 Jul 2009
Posts: 5

Thank you for the further replies.

I don't think retaining messages within a sync-point can address a sustained discrepancy between the required (initial) processing speed and the down-stream processing speed. You may question how such a discrepancy will ever be resolved, but it can be addressed by periodically leveraging 'bulk' operations on (the results of) batches of MQ requests. I don't think it would be practical to retain sync-points while those batches are assembled, as the accumulation period might be several minutes or more in the event of site failure and disaster recovery.

The MQ replicator product could be a useful get out of jail solution if we take the suggested approach of replicating the input to another queue at the application level but run into performance barriers (though unsure how open my client is to introducing new products into its MQ infrastructure.)
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Sat Jun 06, 2015 7:43 am    Post subject: Re: Re-queueing message to different queue...? Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

pmb999 wrote:
If, as the last posting suggests, we re-queue the received message to a different queue, is there any particular support for doing that, or is it a case of issueing an MQ PUT to a different queue to on-forward the incoming message?

If you are able to change the application, doing a put to a different queue to on-forward the message is ideal.
You can then handle the next queue with syncpoint and batch size as desired without impacting the fast response of the first app.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ API Support » Efficiency of processing messages in two phases
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.