ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere DataPower » Datapower-MQ- design performance question

Post new topic  Reply to topic Goto page 1, 2  Next
 Datapower-MQ- design performance question « View previous topic :: View next topic » 
Author Message
anveshita
PostPosted: Tue May 24, 2011 7:55 am    Post subject: Datapower-MQ- design performance question Reply with quote

Master

Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam

I have two separate datapower flows say DP1 and Dp2.
DP1 listens to queue Q1 on QM1. Takes the message and hands it over to an application APP.
DP2 listens to queue Q2 on QM2.Takes the message and hands it over to the same APP1.

APP1 writes the responses to Q3 on QM3. Dp1 and DP2 read the responses from Q3. The way DP1 knows its messages is by tracking the correlation IDs. Same is the case with DP2. DP1 and DP2 has timeout values set on the flows, so if the response message from APP1 is not found on Q3, the flows time-out on the and the session(not sure right word) of the DP flow ends.

Now my questions:
Is it not a bad practice to read the messages based on correlation IDs? We are talking about huge ( nearly 1 M/day)? Does it not affect the performance of DP as it continuously checks MQ, "did I get a response"?

What is the best practice in the industry?

I think, by decoupling the message affinity ( looking for specific correlation id), by having a separate flow DP3 that looks for messages on Q3. Since Dp1 and DP2 know the origin of the message, they could put a string token on the header. Dp3 reads the messages from Q3 till the depth is zero and routes the messages based on token-sting.

What do you think? Please share your thoughts.
Back to top
View user's profile Send private message
harish_td
PostPosted: Tue May 24, 2011 9:04 pm    Post subject: Re: Datapower-MQ- design performance question Reply with quote

Master

Joined: 13 Feb 2006
Posts: 236

anveshita wrote:
Is it not a bad practice to read the messages based on correlation IDs? We are talking about huge ( nearly 1 M/day)?


If you have a request-response scenario, then this is by far the easiest option to implement in the middleware tier. DataPower has been designed to handle these scenarios very efficiently.

anveshita wrote:
Does it not affect the performance of DP as it continuously checks MQ, "did I get a response"?

You can set a timeout value, on the response queue to prevent this occurrence, after which point your response messages are not consumed by DataPower.

Simulate your expected load on a DataPower device and figure out for yourself. Performance metrics are dependent on the network configuration, message size and the data types as with any other solution (WMB implied)

In my opinion, If you want to handle messages in an async fashion, you are better off developing 3 separate flows (Multi Protocol Gateways for the pedantic). In your code, you can still inquire all the MQ Headers

My humble 2 cents
Back to top
View user's profile Send private message Yahoo Messenger
anveshita
PostPosted: Wed May 25, 2011 5:52 am    Post subject: Reply with quote

Master

Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam

Thanks.
Quote:

If you have a request-response scenario, then this is by far the easiest option to implement in the middleware tier. DataPower has been designed to handle these scenarios very efficiently.


I agree that is easy. But is this the best practice for high-volume transactions?

Quote:

You can set a timeout value, on the response queue to prevent this occurrence, after which point your response messages are not consumed by DataPower.


Let us I set the timeout value to say 20 sec. The fact remains that DP keeps polling the queue for the 20 sec, before letting it go. If I have say 10 such connections then all these connections will be polling too.


Quote:
In my opinion, If you want to handle messages in an async fashion, you are better off developing 3 separate flows (Multi Protocol Gateways for the pedantic). In your code, you can still inquire all the MQ Headers


I agree that decouples the request-response dependency.

We could simulate, however I wanted to know if anyone experience this issue.
Thanks once again.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed May 25, 2011 8:37 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Getting by CorrelationID is the best practice for request/response regardless of the kind of MQ application - MQ is optimized to handle this scenario using correlID. Adding custom data to the MQ header is either troublesome or slower.

An MQGET with WAIT is not actually polling - it is very likely that the timeout value you set is being passed into the MQGET statement. So the DP will not be polling, it will be sitting in a wait state until the MQGET returns. If you have 10 such, they will all be paused until the timeout expires.

This is also best practice across all MQ apps regardless of type.

The only reason to decouple a request/response into an asynch process is if the response really needs to be handled asynchronously. If, for example, the request/response is essentially a user request for data, the end user is still waiting for the response and so you should keep things synchronous.

But if the request/response is an update type of operation, where the user is only really waiting for a confirmation, you could decouple this. But then it's not really request/response anyway.
Back to top
View user's profile Send private message
anveshita
PostPosted: Wed May 25, 2011 9:12 am    Post subject: Reply with quote

Master

Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam

mqjeff wrote:

An MQGET with WAIT is not actually polling - it is very likely that the timeout value you set is being passed into the MQGET statement. So the DP will not be polling, it will be sitting in a wait state until the MQGET returns. .


Thanks. But this is contrary to what I am finding in Integration Patterns Redbook on Datapower :

DataPower Architectural Design Patterns Integrating and Securing Services Across Domains, Page No: 153


Quote:
The DataPower device polls the reply-to queue to find a correlated response message. The gateway examines the correlation ID value in the MQ header of messages on the reply-to queue. When this ID is the same as the message ID assigned to the request, the...


Am I missing anything?
Back to top
View user's profile Send private message
anveshita
PostPosted: Fri May 27, 2011 9:16 am    Post subject: Reply with quote

Master

Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam

Any help?
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri May 27, 2011 2:17 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20695
Location: LI,NY

I'd say that the difference between a request / reply pattern and an aggregation pattern really depends on the volume and the average response time.

If you have an average response time of 60 seconds but receive 1 msg per second, you need 60 threads to handle the load with a request response pattern. You will potentially be able to handle the load with two threads with an aggregation pattern with decoupled fan-out, fan-in flows. (one thread for the fan-out, one for the fan-in).

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
mqjeff
PostPosted: Sat May 28, 2011 1:43 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

anveshita wrote:
mqjeff wrote:

An MQGET with WAIT is not actually polling - it is very likely that the timeout value you set is being passed into the MQGET statement. So the DP will not be polling, it will be sitting in a wait state until the MQGET returns. .


Thanks. But this is contrary to what I am finding in Integration Patterns Redbook on Datapower :

DataPower Architectural Design Patterns Integrating and Securing Services Across Domains, Page No: 153


Quote:
The DataPower device polls the reply-to queue to find a correlated response message. The gateway examines the correlation ID value in the MQ header of messages on the reply-to queue. When this ID is the same as the message ID assigned to the request, the...


Am I missing anything?


I suspect that the Datapower documentation is not being technical in it's use of the word "polling".

The easiest thing to do, however, is see what is actually going on. You can examine the MQ queue in question, and the connections to it and the status of those connections and determine what it seems likely Datapower is doing.

Or you could issue an mq trace and determine EXACTLY what Datapower is doing.

Or you could open a PMR with the datapower team and find out that way, too. And perhaps suggest they clarify the documentation as well.
Back to top
View user's profile Send private message
anveshita
PostPosted: Tue May 31, 2011 11:19 am    Post subject: Reply with quote

Master

Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam

mqjeff
Quote:
Or you could issue an mq trace and determine EXACTLY what Datapower is doing.


Can you please let me know how it can be done? Haven't done so far.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue May 31, 2011 11:39 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

You would run the tracing on the qmgr side.

I don't know how to get DP to do it.

there's documentation in MQ's docs on how to do this on the qmgr.
Back to top
View user's profile Send private message
anveshita
PostPosted: Wed Mar 28, 2012 9:14 am    Post subject: DataPower with MQ Cluster Reply with quote

Master

Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam

I have two datapower box DP1 and DP2 using a MQ Cluster. The flows are designed to use "request-response" pattern.
basic setup:
REQ1/QM1,REQ1/QM2 and REQ1/QM3. All request queues are in cluster.
RSP1/QM1,RSP1/QM2 and RSP3/QM3. All response queues are in cluster.

DP1 puts message on REQ1/QM1.
DP2 puts message on REQ1/QM2.
APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2. Since we are using a request-response patter if DP1 puts a message on REQ1/QM1 it expects the response on RSP1/QM1. However sometimes the response would be sent by the cluster to RSP1/QM2. In which case DP2 will be unable to find the response as it is listening on RSP1/QM1.
Is it possible to tell DP to look at other queues in case of using request-response for responses? Should we not use "request/response" and use separate flows for request and response ( this is a costly proposal for us). Any ideas?
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Mar 28, 2012 9:23 am    Post subject: Re: DataPower with MQ Cluster Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

anveshita wrote:
APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2. Since we are using a request-response patter if DP1 puts a message on REQ1/QM1 it expects the response on RSP1/QM1. However sometimes the response would be sent by the cluster to RSP1/QM2. In which case DP2 will be unable to find the response as it is listening on RSP1/QM1.


Why is APP3 just throwing the response into the cluster? The typical request/response model is for the response to be addressed to the requestor for exactly the reasons you give, and using a cluster doesn't change that.


anveshita wrote:
Is it possible to tell DP to look at other queues in case of using request-response for responses?


I doubt it.

anveshita wrote:
Should we not use "request/response" and use separate flows for request and response ( this is a costly proposal for us).


Based on your earlier discussion in this thread request/response seems an obvious choice

anveshita wrote:
Any ideas?


Fix the app.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Mar 28, 2012 9:31 am    Post subject: Re: DataPower with MQ Cluster Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

anveshita wrote:
APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2.


This is the wrong thing for APP3 to do.
Back to top
View user's profile Send private message
anveshita
PostPosted: Wed Mar 28, 2012 9:50 am    Post subject: Re: DataPower with MQ Cluster Reply with quote

Master

Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam

mqjeff wrote:
anveshita wrote:
APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2.


This is the wrong thing for APP3 to do.


Correction:
APP3 responds to the cluster RSP1 so the response may end up either at RSP1/QM1 or RSP1/QM2.

Vitor:
The idea is to add RSP1 in the cluster so you can take advantage of the clustering. Reason why APP3 is sending the response tot he cluster. Are you saying we should not use the response queues in cluster?
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Mar 28, 2012 10:00 am    Post subject: Re: DataPower with MQ Cluster Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

anveshita wrote:
APP3 responds to the cluster RSP1 so the response may end up either at RSP1/QM1 or RSP1/QM2.


Yes, we got that.

anveshita wrote:
The idea is to add RSP1 in the cluster so you can take advantage of the clustering. Reason why APP3 is sending the response tot he cluster. Are you saying we should not use the response queues in cluster?


No, I'm saying you shouldn't mix and match concepts. You should either use a sync request reply model, where requests from 1-n requestors are distributed across 1-n responders by the cluster and the requestor waits for a response which (obviously) is addressed to the waiting responder or you use an async model where requests from 1-n requestors are distributed across 1-n responders by the cluster and responses are likewise distributed by the cluster. This latter clearly requires some kind of state store or similar so that any reponse can be processed anywhere.

A cluster supports both models and I can think of no reason why you wouldn't have response queues in the cluster to assist with future scaling.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » WebSphere DataPower » Datapower-MQ- design performance question
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.