| Author |
Message
|
| anveshita |
 Posted: Tue May 24, 2011 7:55 am Post subject: Datapower-MQ- design performance question |
 |
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
I have two separate datapower flows say DP1 and Dp2.
DP1 listens to queue Q1 on QM1. Takes the message and hands it over to an application APP.
DP2 listens to queue Q2 on QM2.Takes the message and hands it over to the same APP1.
APP1 writes the responses to Q3 on QM3. Dp1 and DP2 read the responses from Q3. The way DP1 knows its messages is by tracking the correlation IDs. Same is the case with DP2. DP1 and DP2 has timeout values set on the flows, so if the response message from APP1 is not found on Q3, the flows time-out on the and the session(not sure right word) of the DP flow ends.
Now my questions:
Is it not a bad practice to read the messages based on correlation IDs? We are talking about huge ( nearly 1 M/day)? Does it not affect the performance of DP as it continuously checks MQ, "did I get a response"?
What is the best practice in the industry?
I think, by decoupling the message affinity ( looking for specific correlation id), by having a separate flow DP3 that looks for messages on Q3. Since Dp1 and DP2 know the origin of the message, they could put a string token on the header. Dp3 reads the messages from Q3 till the depth is zero and routes the messages based on token-sting.
What do you think? Please share your thoughts. |
|
| Back to top |
|
 |
| harish_td |
| Posted: Tue May 24, 2011 9:04 pm Post subject: Re: Datapower-MQ- design performance question |
|
|
Master
Joined: 13 Feb 2006
Posts: 236
|
| anveshita wrote: |
| Is it not a bad practice to read the messages based on correlation IDs? We are talking about huge ( nearly 1 M/day)? |
If you have a request-response scenario, then this is by far the easiest option to implement in the middleware tier. DataPower has been designed to handle these scenarios very efficiently.
| anveshita wrote: |
| Does it not affect the performance of DP as it continuously checks MQ, "did I get a response"? |
You can set a timeout value, on the response queue to prevent this occurrence, after which point your response messages are not consumed by DataPower.
Simulate your expected load on a DataPower device and figure out for yourself. Performance metrics are dependent on the network configuration, message size and the data types as with any other solution (WMB implied)
In my opinion, If you want to handle messages in an async fashion, you are better off developing 3 separate flows (Multi Protocol Gateways for the pedantic). In your code, you can still inquire all the MQ Headers
My humble 2 cents |
|
| Back to top |
|
|
| anveshita |
| Posted: Wed May 25, 2011 5:52 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
Thanks.
| Quote: |
If you have a request-response scenario, then this is by far the easiest option to implement in the middleware tier. DataPower has been designed to handle these scenarios very efficiently. |
I agree that is easy. But is this the best practice for high-volume transactions?
| Quote: |
You can set a timeout value, on the response queue to prevent this occurrence, after which point your response messages are not consumed by DataPower. |
Let us I set the timeout value to say 20 sec. The fact remains that DP keeps polling the queue for the 20 sec, before letting it go. If I have say 10 such connections then all these connections will be polling too.
| Quote: |
| In my opinion, If you want to handle messages in an async fashion, you are better off developing 3 separate flows (Multi Protocol Gateways for the pedantic). In your code, you can still inquire all the MQ Headers |
I agree that decouples the request-response dependency.
We could simulate, however I wanted to know if anyone experience this issue.
Thanks once again. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed May 25, 2011 8:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Getting by CorrelationID is the best practice for request/response regardless of the kind of MQ application - MQ is optimized to handle this scenario using correlID. Adding custom data to the MQ header is either troublesome or slower.
An MQGET with WAIT is not actually polling - it is very likely that the timeout value you set is being passed into the MQGET statement. So the DP will not be polling, it will be sitting in a wait state until the MQGET returns. If you have 10 such, they will all be paused until the timeout expires.
This is also best practice across all MQ apps regardless of type.
The only reason to decouple a request/response into an asynch process is if the response really needs to be handled asynchronously. If, for example, the request/response is essentially a user request for data, the end user is still waiting for the response and so you should keep things synchronous.
But if the request/response is an update type of operation, where the user is only really waiting for a confirmation, you could decouple this. But then it's not really request/response anyway. |
|
| Back to top |
|
|
| anveshita |
| Posted: Wed May 25, 2011 9:12 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| mqjeff wrote: |
An MQGET with WAIT is not actually polling - it is very likely that the timeout value you set is being passed into the MQGET statement. So the DP will not be polling, it will be sitting in a wait state until the MQGET returns. . |
Thanks. But this is contrary to what I am finding in Integration Patterns Redbook on Datapower :
DataPower Architectural Design Patterns Integrating and Securing Services Across Domains, Page No: 153
| Quote: |
| The DataPower device polls the reply-to queue to find a correlated response message. The gateway examines the correlation ID value in the MQ header of messages on the reply-to queue. When this ID is the same as the message ID assigned to the request, the... |
Am I missing anything? |
|
| Back to top |
|
|
| anveshita |
| Posted: Fri May 27, 2011 9:16 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 27, 2011 2:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I'd say that the difference between a request / reply pattern and an aggregation pattern really depends on the volume and the average response time.
If you have an average response time of 60 seconds but receive 1 msg per second, you need 60 threads to handle the load with a request response pattern. You will potentially be able to handle the load with two threads with an aggregation pattern with decoupled fan-out, fan-in flows. (one thread for the fan-out, one for the fan-in).
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat May 28, 2011 1:43 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| anveshita wrote: |
| mqjeff wrote: |
An MQGET with WAIT is not actually polling - it is very likely that the timeout value you set is being passed into the MQGET statement. So the DP will not be polling, it will be sitting in a wait state until the MQGET returns. . |
Thanks. But this is contrary to what I am finding in Integration Patterns Redbook on Datapower :
DataPower Architectural Design Patterns Integrating and Securing Services Across Domains, Page No: 153
| Quote: |
| The DataPower device polls the reply-to queue to find a correlated response message. The gateway examines the correlation ID value in the MQ header of messages on the reply-to queue. When this ID is the same as the message ID assigned to the request, the... |
Am I missing anything? |
I suspect that the Datapower documentation is not being technical in it's use of the word "polling".
The easiest thing to do, however, is see what is actually going on. You can examine the MQ queue in question, and the connections to it and the status of those connections and determine what it seems likely Datapower is doing.
Or you could issue an mq trace and determine EXACTLY what Datapower is doing.
Or you could open a PMR with the datapower team and find out that way, too. And perhaps suggest they clarify the documentation as well. |
|
| Back to top |
|
|
| anveshita |
| Posted: Tue May 31, 2011 11:19 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
mqjeff
| Quote: |
| Or you could issue an mq trace and determine EXACTLY what Datapower is doing. |
Can you please let me know how it can be done? Haven't done so far. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue May 31, 2011 11:39 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You would run the tracing on the qmgr side.
I don't know how to get DP to do it.
there's documentation in MQ's docs on how to do this on the qmgr. |
|
| Back to top |
|
|
| anveshita |
| Posted: Wed Mar 28, 2012 9:14 am Post subject: DataPower with MQ Cluster |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
I have two datapower box DP1 and DP2 using a MQ Cluster. The flows are designed to use "request-response" pattern.
basic setup:
REQ1/QM1,REQ1/QM2 and REQ1/QM3. All request queues are in cluster.
RSP1/QM1,RSP1/QM2 and RSP3/QM3. All response queues are in cluster.
DP1 puts message on REQ1/QM1.
DP2 puts message on REQ1/QM2.
APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2. Since we are using a request-response patter if DP1 puts a message on REQ1/QM1 it expects the response on RSP1/QM1. However sometimes the response would be sent by the cluster to RSP1/QM2. In which case DP2 will be unable to find the response as it is listening on RSP1/QM1.
Is it possible to tell DP to look at other queues in case of using request-response for responses? Should we not use "request/response" and use separate flows for request and response ( this is a costly proposal for us). Any ideas? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 28, 2012 9:23 am Post subject: Re: DataPower with MQ Cluster |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anveshita wrote: |
| APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2. Since we are using a request-response patter if DP1 puts a message on REQ1/QM1 it expects the response on RSP1/QM1. However sometimes the response would be sent by the cluster to RSP1/QM2. In which case DP2 will be unable to find the response as it is listening on RSP1/QM1. |
Why is APP3 just throwing the response into the cluster? The typical request/response model is for the response to be addressed to the requestor for exactly the reasons you give, and using a cluster doesn't change that.
| anveshita wrote: |
| Is it possible to tell DP to look at other queues in case of using request-response for responses? |
I doubt it.
| anveshita wrote: |
| Should we not use "request/response" and use separate flows for request and response ( this is a costly proposal for us). |
Based on your earlier discussion in this thread request/response seems an obvious choice
| anveshita wrote: |
| Any ideas? |
Fix the app.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 28, 2012 9:31 am Post subject: Re: DataPower with MQ Cluster |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| anveshita wrote: |
| APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2. |
This is the wrong thing for APP3 to do. |
|
| Back to top |
|
|
| anveshita |
| Posted: Wed Mar 28, 2012 9:50 am Post subject: Re: DataPower with MQ Cluster |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| mqjeff wrote: |
| anveshita wrote: |
| APP3 consumes messages on REQ1/QM3 and responds to the cluster so the response may end up either at REQ1/QM1 or REQ1/QM2. |
This is the wrong thing for APP3 to do. |
Correction:
APP3 responds to the cluster RSP1 so the response may end up either at RSP1/QM1 or RSP1/QM2.
Vitor:
The idea is to add RSP1 in the cluster so you can take advantage of the clustering. Reason why APP3 is sending the response tot he cluster. Are you saying we should not use the response queues in cluster? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 28, 2012 10:00 am Post subject: Re: DataPower with MQ Cluster |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anveshita wrote: |
| APP3 responds to the cluster RSP1 so the response may end up either at RSP1/QM1 or RSP1/QM2. |
Yes, we got that.
| anveshita wrote: |
| The idea is to add RSP1 in the cluster so you can take advantage of the clustering. Reason why APP3 is sending the response tot he cluster. Are you saying we should not use the response queues in cluster? |
No, I'm saying you shouldn't mix and match concepts. You should either use a sync request reply model, where requests from 1-n requestors are distributed across 1-n responders by the cluster and the requestor waits for a response which (obviously) is addressed to the waiting responder or you use an async model where requests from 1-n requestors are distributed across 1-n responders by the cluster and responses are likewise distributed by the cluster. This latter clearly requires some kind of state store or similar so that any reponse can be processed anywhere.
A cluster supports both models and I can think of no reason why you wouldn't have response queues in the cluster to assist with future scaling.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
