Author |
Message
|
murlee_99 |
Posted: Thu Aug 10, 2006 4:24 pm Post subject: Cluster over WAN ? - Masters Please Read |
|
|
 Novice
Joined: 28 May 2002 Posts: 10 Location: NZ
|
Hi,
We have the following scenario to accomplish.
The application is very simple, We have a J2EE App Server which acts as both publisher and receiver for messages in the queue manager. And for all obvious reasons we are using the Clustered QMs. With respect to transactional behaviour I'm not sure if my understanding of clusters is correct. Lets say there are two Queue Managers QM1, QM2 in a cluster and both are full repositories.
Message procuder - QM1
QM2 - Message consumer
In a normal scenario message producer can publish messages to QM1 and consumer can prickup the messages from QM2 using a clustered queue with all correct channels etc. But consider the following scenarios.
Scenario 1
------------
1. Producer sends message to the QM1 using the appropriate channel.
2. QM1 persists message to the disk
3. Producer receives an ack from the QM1.
4. Producer commits transaction.
5. --- Disaster -- QM1 died
Can the message consumer receive the message? Question really is does QM1 communicate to QM2 before sending an ack back to the Producer?
Scenario 2
------------
1. Producer sends message to the QM1 using the appropriate channel.
2. QM died before sending the ack back to the Producer
Is there a way to the client to automatically connect to QM2 on different listner? In other words is client side fail-over supported/possible?
Scenario 3 - WAN
---------------------
If we were to deploy this application on a geographically different location, (just in case if a plane crashes into our data center ) and be able to continue the operations. The two locations are connected by a frame-relay link.
The secondary site is passive and it will be truned on active when need be. Passive means there are no Producers/Consumers, it is allowed to run the WMQ QMs, if need be.There are two things we need to worry about Database data and the messages in the queues/topics. Oracle has lots of options with regards to this scenario to maintain a copy of the data continually to the other site.
With WMQ
option a) Is it possible to store messages in a database instead of using file system? I couldn't find any topic in this matter.
option b) Run a cluster over WAN. I know this is possible, but I'm not sure what is the best approach.
If we choose to join all the queue managers into a cluster from both sites and one full repository on each site, that is probably good enough. Any issues that I should be aware of?
Because of the fact that the secondary site is passive and it would be ideal not to copy messages that are produced and consumed immediately. I'm not sure how WMQ behaves if there is a consumer connected directly to the QM where the message is produced, would the QM still copy the message to all the other QMs? Please clarify.
Ideally any message that has been in a queue/topic for more than a minute should be made available on secondary site queue managers.
Is it possible?
Sorry for too many questions.
Any thoughts/comments please?
Regards,
Murali _________________ Regards,
Murali
IBM Certified MQSeries Specialist |
|
Back to top |
|
 |
Vitor |
Posted: Fri Aug 11, 2006 12:16 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Okay, like you said a lot of questions.
If you do a search in this section you'll find endless numbers of threads concerning the use of clustering to provide failover. I don't propose to repeat them here (happy reading!) but IMHO MQ clustering is unsuitable for high-availability, disaster recover clustering. That's better done with an HA product - MQ clustering is intended to provide workload balancing and simplified administration for large numbers of queue managers.
In your normal scenario, the producer connects to QM1 and writes to a clustered queue. The comsumer conneted to QM2 cannot see this message because QM1 is hosting it unless you're running MQv6 and have done some clever stuff with the parameters!! Just because a queue is shared through the cluster does not make the messages available through the cluster. The only way this would work is if you're on z/OS and using shared queues. You'll need to set up some point-to-point stuff to move the message onto QM2, which sort of negates clustering
In your failure scanario 1, if QM1 dies then QM2 will have got the message on commit, which is the same time QM1 would have got the ack if you're using MQ ack. This assumes message transport as intimated above.
In scenario 2, the answer is yes. Look up client connection tables in the Clients manual.
In scenario 3, please see my rant at the top of this post! There's no reason why you can't run a cluster over a WAN, I disagree with the wisdom of using it for HA
You can't store messages in a database.
I repeat, messages in a cluster are not copied to all the other QMs a la database replication. That's just not how it works. Check the clustering manual.
Topics don't work like that either. Check the Pub/Sub manual.
Hope that helps, if only to point you in the direction of some information sources.
Happy Reading!  _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
jefflowrey |
Posted: Fri Aug 11, 2006 1:49 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
Vitor wrote: |
You can't store messages in a database. |
To expand a little bit - you don't need to change MQ to use a database persistance mechanism instead of it's existing filestore mechanism.
If you configure MQ correctly and your applications, you can guarantee that you will never lose any message.
You can never guarantee when the message arrives, however! MQ is asychronous.
Designing applications that communicate over an asynchronous protocol - especially when it provides guaranteed once-only delivery - is different than designing apps that communicate over a synchronous protocol like HTTP.
For one thing, you can avoid using ACKS in a lot of use cases. All communications can be "fire-and-forget" unless there is business information that needs to be returned - like in a request/reply scenario or in an order/report kind of scenario. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
Vitor |
Posted: Fri Aug 11, 2006 1:58 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Good Morning jefflowrey - can't sleep? Or not in your usual timezone?
I agree that you shouldn't need to use a database to store messages. My (possibly flawed) interpretation of the question was "Can I replace the file system with an ODBC connection and punt the messages down that?", possibly to produce the kind of message replication murlee_99 is trying for - message put onto QM1 available on QM2 despite the queue being on QM1.
And even if you could use that kind of ODBC, replicating messages like that would open up a world of pain and misery, and would almost result in guaranteed once-only-but-occassionally-twice delivery!  _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
jefflowrey |
Posted: Fri Aug 11, 2006 2:11 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
Yeah, using databases to provide inter-application communication isn't the best of ideas.
I'm actually usually up this early during the week... It just feels earlier this morning...
In fact, I think murlee_99 is trying to decide between using the SIBus in WAS v6 and using MQ. And as to that, as long as the apps are all running inside WAS, then using the SIBus is a reasonable idea. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
murlee_99 |
Posted: Tue Aug 15, 2006 7:05 pm Post subject: |
|
|
 Novice
Joined: 28 May 2002 Posts: 10 Location: NZ
|
Thans Heaps. As Vitor said I read the docs again and it is very clear that clustering in MQ is not for HA. On z/OS you can probably use queue-sharing groups to publish to one QM and read from another QM.
I figured it is probably the client channel defnition table that provides the client side fail-over. But what is the point of being able to do client side fail-over when the QM itself is not highly available. I could be wrong here, but to me MQ clusteris definitely not the answer to make a QM highly available. MQ has been around for a long time, I'm sure there are ways to make a QM highly available, without having to do the OS cluster.
Vitor, would you be able to share those tricks to make QM highly available, as in the tricks that you have to do to publish to one QM and read from another QM. Just few pointers would be very helpful.
Jeff you are very close, I'm comparing with Sun SJSMQ not SIBus.
The reason I asked for replacement of DB for filesystem is to be able to quickly achive disaster recovery. I know the messages are not lost when you use file-system, but in the case of DR you might want to bring up your QM/MQ somewhere else with the same state as your production QM/MQ. There are a number of ways/tools to perform db replication but not for file replication. I guess for file-system you could use SAN replication techniques. Any better ideas?
I'm still unsure if we can use DB as message store rather than the file-system. Vitor said you can't, Jeff did you mean you dont need to but you can?
Also, what ever I'm asking here is possible with a number of messaging systems such as SJSMQ, SpritWave and probably SIBus?
Thanks for your time. _________________ Regards,
Murali
IBM Certified MQSeries Specialist |
|
Back to top |
|
 |
Vitor |
Posted: Tue Aug 15, 2006 11:55 pm Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
murlee_99 wrote: |
Vitor, would you be able to share those tricks to make QM highly available, as in the tricks that you have to do to publish to one QM and read from another QM. Just few pointers would be very helpful.
|
It's quite simple - you buy some HA software, cluster some machines with it and run queue managers on them!
Remember that in an HA scenario, the failure machine becomes the live machine to all intents and purposes (takes over the IP address for instance) so you're not publishing to one QM and reading from another - you're publishing to a QM that then fails and is reincarnated on the failure machine.
There's a support pack on the IBM website that describes running MQ under most (if not all) of the popular brands of HA software, giving helpful hints and guidance. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
jefflowrey |
Posted: Wed Aug 16, 2006 2:16 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
I meant that not only do you not need to replace the file store with a db store, but you also can't.
And yes, in an MQ HA configuration then "shared disk" of some kind is required. In the old days this required fancy expensive SCSI cards. Nowadays most people use contact admin of some kind or another (that's "network attached storage", usually on a "storage area network").
And yes, MQ Clustering is not an HA solution. Never intended to be, and hopefully never sold as such.
I have heard commentary that the kind of failover that you see with SIBus and other such is also not completely HA according to the standards, and this was from someone who probably should have read the standards. The gist was that there were some requirements that couldn't be met without shared disk and os level failover. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
Vitor |
Posted: Wed Aug 16, 2006 2:29 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
And to underline these comments, when I speak of HA software I'm assuming shared RAIDed disk. I've only experience of HACMP, Veritas and SecureGuard (disclaimer! ) and these certainly expect shared disk to be available for them to work them magic.
(You RAID it to remove a potential single point of failure) _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
murlee_99 |
Posted: Wed Aug 16, 2006 3:45 am Post subject: |
|
|
 Novice
Joined: 28 May 2002 Posts: 10 Location: NZ
|
Thank you very much Vitor and Jeff.
Given the fact that we don't have very high volume of messages, looks like we might stick with SIBus if we go down the WebSphere path. _________________ Regards,
Murali
IBM Certified MQSeries Specialist |
|
Back to top |
|
 |
jefflowrey |
Posted: Wed Aug 16, 2006 4:19 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
Using SIBus instead of MQ has nothing to do with the volume of your messages.
In fact, using HA has nothing to do with the volume of your messages. HA is mostly about, umm, high availability.
If you only send one message a day, but it's calculating bonuses for the CEO... then you never want MQ to be unavailable when that message gets sent.
The decision to use SIBus instead of MQ should factor in things like how long the product has been around, and more importantly the nature of the apps that have to use the transport. You may have a hard time talking to CICS from SIBus. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
murlee_99 |
Posted: Wed Aug 16, 2006 3:47 pm Post subject: |
|
|
 Novice
Joined: 28 May 2002 Posts: 10 Location: NZ
|
You are right. HA has nothing to do with the volume of messages. The main reason for going SIBus path is to meet the HA requirements. As you said SIBus is quite new and we all know how stable the early versions usually are Obvously WMQ is proven to perform under high volume where as SIBus is and unknown (may be not a/c to IBM). Our volumes are relatively low.
Offcourse if you are talking to MVS, z/OS or one of those legacy monsters then WMQ would serve better. Fortunately we dont have to.
Hence SIBus.
Thanks for sharing your thoughts. _________________ Regards,
Murali
IBM Certified MQSeries Specialist |
|
Back to top |
|
 |
jefflowrey |
Posted: Wed Aug 16, 2006 4:53 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
My general opinion is that SIBus is only good for cases where all of the applications are running under WAS v6.
IN practice, though, everything is a special case and so... it depends. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
|