| Author |
Message
|
| smdavies99 |
 Posted: Sun Oct 13, 2013 8:09 am Post subject: Cluster Output Side Best Practices |
 |
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I'm looking for some best practices for the output side of a WMQ Cluster.
By this I mean this.
The input message stream is load balanced to the two nodes in the cluster where they are processed by the application and then the output is put on a queue for a client application to pick up.
The Client Apps would connect to one of the queue managers so would miss 50% of the output messages.
I know that the Clients can be configured to connect to multiple QM's so if the first one is down then it can connect to the other node in the cluster.
How do other sites setup their systems so that the single connect clients get all the available messages.
One possible solution is to have a 3rd Queue Manager that collected all the messages for output but that could have cost implications as AFAIK, this QM should run on a separate Active-Passive environment.
Thoughts and Brickbats most welcome.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Sun Oct 13, 2013 1:51 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| There's a lovely devworks article that goes into this in great detail. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Oct 13, 2013 9:52 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I've looked at at least half a dozen Developerworks articles and so far this 'lovely article' seems to have eluded me. Would you care to give me a pointer?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Oct 13, 2013 11:46 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Oct 14, 2013 12:04 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Ahh Thanks. I've read the first one but I'll dig into it a bit deeper now but I'd missed the second reference. Thanks.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Oct 14, 2013 4:48 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
If my observations are correct then it would be possible to use an Active-Active solution with Message Broker /IIB as the transformation engine in many cases with the following restriction
- The WMQ connected clients may need modification to handle the CCDT configuration needed to handle the duplicate paths to the Queue Managers. This system is to be used in a very multi-vendor solution (20+ different suppliers).
This may proves to be an impossible task (the vendors would much prefer to use MSMQ!)
The remaining stumbling block is the clients that connect using TCP/IP. As this is a single point connection we can't on the face of it use and Active-Active WMQ/Broker solution.
I'd prefer to use Active/Passive as we have implemented these in many places without issue.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Oct 14, 2013 4:57 am Post subject: Re: Cluster Output Side Best Practices |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| smdavies99 wrote: |
How do other sites setup their systems so that the single connect clients get all the available messages.
|
Can you have multiple instances of that client app where 1 or more instances of that same app are connected to QMA and 1 or more connected to QMB, so that both instances of the output queue for that app are being processed?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Oct 14, 2013 5:03 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Nice thought Peter. My guess is that in many cases this would be impossible. From my experience the level of expertise in a good number of vendors in this area is very limited. As I indicated for many of them their preferred transport is MSMQ (the programmers dream) and not WMQ (the nightmare).
Many would also try to charge us for the costs of making the changes!
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Oct 14, 2013 5:18 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| smdavies99 wrote: |
| ...As I indicated for many of them their preferred transport is MSMQ (the programmers dream) and not WMQ (the nightmare)... |
It pains me, greatly, to say this but - have you considered bridging MSMQ to WMQ?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Oct 14, 2013 5:51 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| exerk wrote: |
It pains me, greatly, to say this but - have you considered bridging MSMQ to WMQ? |
No pain needed. We have a bridge tool but that still has issues, well MSMQ has issues all of its own. One tip for anyone else reading this, the location of the first MSMQ queue you write to is important in a failover scenario.
As for this solution? Not really practical and AFAIK, MSMQ is not really up to working in this type of environment.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Oct 14, 2013 5:58 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| smdavies99 wrote: |
| ...We have a bridge tool but that still has issues... |
Purely out of curiosity, but is that the MSMQ-MQ Bridge or a roll-your-own solution?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Oct 14, 2013 6:24 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| smdavies99 wrote: |
| One tip for anyone else reading this, the location of the first MSMQ queue you write to is important in a failover scenario. |
Care to explain this?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Oct 14, 2013 6:32 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| exerk wrote: |
| smdavies99 wrote: |
| ...We have a bridge tool but that still has issues... |
Purely out of curiosity, but is that the MSMQ-MQ Bridge or a roll-your-own solution? |
It's our own solution (ducks to avoid incoming missiles). I'm trying to retire is but not having a lot of success.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Oct 14, 2013 6:38 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| bruce2359 wrote: |
| smdavies99 wrote: |
| One tip for anyone else reading this, the location of the first MSMQ queue you write to is important in a failover scenario. |
Care to explain this? |
If you store the messages that are going from WMQ to MSMQ in the wrong place and a failover happens then you could lose data.
We have seen this when we stored the data on the local server (ie we wrote the data to the queue on the remote server). Changing it to store the data on the remote server and we didn't lose any data. 90% of the time it worked flawlessly but every so often we lost a message which contained a sequence number. That's how we knew that we'd lost one.
This may well have been issues in how the MSMQ system was setup in the first place. I have tried but I can't get a definitive answer on the problem.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Oct 14, 2013 6:44 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| smdavies99 wrote: |
Nice thought Peter. My guess is that in many cases this would be impossible. From my experience the level of expertise in a good number of vendors in this area is very limited. As I indicated for many of them their preferred transport is MSMQ (the programmers dream) and not WMQ (the nightmare).
Many would also try to charge us for the costs of making the changes! |
In cases like that what we do is have only one instance of the output queue and we cluster it so other QMs can see it. The consuming app then has their one and only instance consume from that one output queue. And everyone has to be aware that if that QM becomes N/A, the output messages will queue up onthe System Cluster Transmit Queue of the other QMs int he cluster.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
