| Author |
Message
|
| akil |
 Posted: Sat Apr 04, 2015 10:46 am Post subject: MQ vs HTTP or vice versa |
 |
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
Split from http://www.mqseries.net/phpBB2/viewtopic.php?t=69866&sid=d9a31c7a8ecc1aa020851448a3c358e1
In situations where the backends and the consumers are all http based, with no queue & XA support, what is the use of MQ?
You get a http request, put it on a queue, call backend 1 ( http ), then call backend 2 ( http ), then get an error, and then do what ? Put it into a repair queue for someone to look at it, so can a database with a UI .
MQ has its uses, but it's not the magic pill.
_________________
Regards |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Sat Apr 04, 2015 11:06 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| akil wrote: |
MQ has its uses, but it's not the magic pill. |
On the other-hand, WebServices are not the answer to life, the universe and everything.
You will find that Billions if not Trillions of $$$$$, Euros, Pounds, Yen etc are moved everyday around the world using MQ with not a sign of HTTP or SOAP anywhere to be seen. Without a proper Queueing system this just could not happen.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| nelson |
| Posted: Sat Apr 04, 2015 11:10 am Post subject: |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
| akil wrote: |
In situations where the backends and the consumers are all http based, with no queue & XA support, what is the use of MQ?
You get a http request, put it on a queue, call backend 1 ( http ), then call backend 2 ( http ), then get an error, and then do what ? Put it into a repair queue for someone to look at it, so can a database with a UI .
MQ has its uses, but it's not the magic pill. |
Exactly, in situations 
My claim is that IIB-MQ are still very coupled. You require MQ installed on the same integration node server for this features:
| Quote: |
Record and replay
Global transactionality
Event-driven processing nodes (aggregate, collector, sequence, resequence, and timeout nodes)
FTEInput and FTEOutput nodes
CDInput and CDOutput nodes
SCA nodes (with MQ bindings)
SAP nodes (with transactional processing)
Integration nodes with HTTP listeners
HTTP proxy servlet
High availability configurations |
So, by now, you can't assume that if you do not actually use MQ (as your transport protocol), you don't require it. |
|
| Back to top |
|
|
| zpat |
| Posted: Sun Apr 05, 2015 4:44 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
If you do not use MQ as your transport - then you are happy to lose or duplicate data. That is the fact of the matter.
The fact that people are choosing transactionally unsafe transports is deeply worrying.
HTTP can also simply fail under high load, because the internal tomcat queue is limited (and if there was more, then requests would still time out).
Inexperienced IT staff imagine that queuing (or asynchronous) transports are slow - when in fact non-persitent messages have barely measureable latency (over the underlying network latency).
Whatever the question (aside from massive bulk files) MQ is the answer and I always tell projects that if they follow my advice - the design will work perfectly, be manageable, scaleable, reliable and all the rest.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Apr 05, 2015 6:20 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| zpat wrote: |
If you do not use MQ as your transport - then you are happy to lose or duplicate data. That is the fact of the matter.
The fact that people are choosing transactionally unsafe transports is deeply worrying.
HTTP can also simply fail under high load, because the internal tomcat queue is limited (and if there was more, then requests would still time out).
Inexperienced IT staff imagine that queuing (or asynchronous) transports are slow - when in fact non-persitent messages have barely measureable latency (over the underlying network latency).
Whatever the question (aside from massive bulk files) MQ is the answer and I always tell projects that if they follow my advice - the design will work perfectly, be manageable, scaleable, reliable and all the rest. |
Strange that all the WebService bibles neglect all these very relevant points.
There again, I'm biased and not just because of 16years of using MQ. It just makes more sense in the majority of cases to use this stuff.
What does get me annoyed is people confusing MQ with MSMQ (a toy).
all those HTTP related thingies do have a role to play but when it comes down to it and you want to reliably move lots of messages then MQ is the way to go.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| akil |
| Posted: Sun Apr 05, 2015 6:30 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
In a scenario where the clients are http consumers , and ther servers are http / soap servers, what role does an MQ based integration layer play?
Servers already guarantee idempotent operations, clients already expect a synchronous protocol. On a timeout they retry fearlessly ( because of idempotent operations ). Storing http request into a queue is of no use to either the server or the client.
Whether the servers and clients should be MQ based or should embrace web services can be debated, my remark was that I am happy that the bus realises that not all environments are equal.
_________________
Regards |
|
| Back to top |
|
|
| zpat |
| Posted: Sun Apr 05, 2015 8:31 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Simple - you don't use http clients. You don't use http servers.
Use MQ instead for both client and server.
You wouldn't believe the trouble that http has caused us in broker (resulting in several APARs being raised for example) that could all have been avoided by using MQ at all times and in all places.
Http is simply inferior is every way and there is no need to use it.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Apr 06, 2015 4:39 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| akil wrote: |
Whether the servers and clients should be MQ based or should embrace web services can be debated, my remark was that I am happy that the bus realises that not all environments are equal. |
You do realize, of course, that web services also come with the JMS transport flavor... right? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| akil |
| Posted: Mon Apr 06, 2015 9:07 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
Yes, I am aware.
All I am saying is that the choice of transport isn't a decision of the bus alone, clients and servers should support it first.
In situations where there is no MQ , and there are such situations, even in banks, then having a MQ pre-req didn't serve any purpose.
I have seen recommendations that force an MQ within the bus, with the clients and servers over http, and said to be zero message loss, which is not true at all. Those reco's seem to come from a through and through MQ environment and are mis-placed in a http environment .
_________________
Regards |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Apr 09, 2015 5:08 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
So I failed miserably trying to reference this thread in conversation today, during an HTTP versus MQ debate.
What exactly is the problem with HTTP compared to an MQ Client call?
Why is this:
App 1 --> MQ Client --> MQ Server --> MQ Client - App 2
Better than this:
App 1 -->HTTP---> App 2
I could not make the case. "HTTP does not fail silently" I was told. If the HTTP call fails, its recorded in a log just the same as the MQ Client API call failing does.
MQ Conversion is not required in this scenario - you don't need to explain to me the benefits of that.
And if they are prepared to use HTTP, they are expecting near real time synchronous responses, so MQ's ability to queue doesn't impress in this scenario.
I was trying to make the case for if it matters use MQ (channeling my inner zpat), yet failed. HTTP between 2 servers in the same data center will be just fine.
Educate me please. Not on why to use MQ when you need queuing, or data conversion, or one partner is already MQ enabled. 2 new apps want to communicate, both are ins the same data center, both know HTTP. Why add MQ. Why the "overhead"?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 09, 2015 8:14 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
How about App2 dies after the commit but before sending the return code to App1 ? Think about time tied up in waiting when App2 gets overloaded and the wait time increases for each call from App1. What happens when App2 receives the call but App1 times out before App2 is done processing it? Are you going to retry? How do you check for duplicates?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Apr 10, 2015 3:26 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| fjb_saper wrote: |
| How about App2 dies after the commit but before sending the return code to App1 ? |
Same problem with MQ, like when an MQ Client issues a MQCMIT call and gets a non successful MQRC?
| fjb_saper wrote: |
| Think about time tied up in waiting when App2 gets overloaded and the wait time increases for each call from App1. |
Same problem with MQ, you are just waiting on your MQGET with Wait a little longer for the reply.
| fjb_saper wrote: |
| What happens when App2 receives the call but App1 times out before App2 is done processing it? Are you going to retry? How do you check for duplicates? |
OK, so App1 can check the reply queue at a later time to see if the reply is there, and decide to retry if its not there. But how do you know the original reply won't land a split second after that decision is made? I think you have the same problem with either HTTP or MQ - you didn't get a reply in the expected time frame, and now you have a decision to make on whether to resubmit or not.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| akil |
| Posted: Sat Apr 11, 2015 9:32 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
The server has to give assurance to the clients that they can retry safely.
When the server cant, the ESB steps in to give this guarantee.
The ESB flow would be
Http -- audit ( reject duplicates ) -- http request
The above allows clients to retry safely,
The server has to give a way to query the result of an operation ( this is used in case of a timeout ).
In the unlikely case that the server does not provide a way to get the status, then http isn't going to cut it, the server has to use a persistent queue, if the server does not do that, then on every timeout a manual reconciliation will need to be done.
_________________
Regards |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Apr 11, 2015 7:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PeterPotkay wrote: |
| fjb_saper wrote: |
| How about App2 dies after the commit but before sending the return code to App1 ? |
Same problem with MQ, like when an MQ Client issues a MQCMIT call and gets a non successful MQRC? |
O.K. I'll grant you that one. Although the side queue pattern is a nice one where you can interrogate the side queue about the status (The status was saved within the same UOW) without touching the original app...
| PeterPotkay wrote: |
| fjb_saper wrote: |
| Think about time tied up in waiting when App2 gets overloaded and the wait time increases for each call from App1. |
Same problem with MQ, you are just waiting on your MQGET with Wait a little longer for the reply. |
Not quite. If you are dealing with a one way message, you still need to wait for your 204 status code... MQ put and forget... (+ for mq in my book). You will get a queue depth until App1 catches up...
| PeterPotkay wrote: |
| fjb_saper wrote: |
| What happens when App2 receives the call but App1 times out before App2 is done processing it? Are you going to retry? How do you check for duplicates? |
OK, so App1 can check the reply queue at a later time to see if the reply is there, and decide to retry if its not there. But how do you know the original reply won't land a split second after that decision is made? I think you have the same problem with either HTTP or MQ - you didn't get a reply in the expected time frame, and now you have a decision to make on whether to resubmit or not. |
Again you are assuming a 2 way communication. On a 2 way communication, you are right there is not so much to be gained... On a one way communication however... Especially if the pipe is not as big as the demand....
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sun Apr 12, 2015 11:57 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
fjb_saper,
Good thoughts on how for 1 way communications an MQ Queue Manager will likely have more capacity for concurrent incoming messages than an HTTP server.
zpat, in another thread again you mention if you value your data use MQ. Why? Is http going to fail in a way that would make the client think the data was safely sent when in fact it wasn't? I need to understand how to put up a good argument against the Use-WebServics-For-Everything contingent, but I still struggle trying to convince them to use MQ if they care about their data. There is no problem when legitimate queueing is needed, or if data conversion is needed. But why is MQ better for semi-synchronous request/replies than HTTP?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
