| Author |
Message
|
| EKhalil |
 Posted: Fri Jul 11, 2008 9:56 am Post subject: Is it possible for the broker to lose persistent messages ? |
 |
|
Voyager
Joined: 29 Apr 2003
Posts: 99
Location: Boston, MA
|
The broker's database (not user database) which resides remotely got hosed and the node had to be rebooted. Highly active flow's data reconciliation showed two persistent messages missing - meaning could not locate them in any of the queues or the receiving app's logs. Is there any way that the broker could lose messages that were being processed while the WMBRK's database went down ??? What exactly happens to the messages that are under broker's jurisdiction if it loses the database...(There are no error entries in the logs with the exception of failed stop command issued to the broker. The broker was actually terminated by killing its PID)
(WMB 6.1 WMQ 6.0)
BTW:Are there any ramifications to having broker's database located on a different node ??? |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Fri Jul 11, 2008 11:55 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
There are lots of possible implications to broker operations when you have the broker database on a different server.
Personally, I think this is a very bad idea and should be avoided at all costs.
There will be performance implications in
- Pub/Sub operations
- Flows that use Aggregation
If the network goes down you are 'stuffed'
Your experience of having to reboot the remote system is also typical of the problems you may encounter with this scenario.
I'm not sure about the real reasons for the loss of messages but what do you really think if you pulled the rug from under your chair like you did.
Perhaps it is time for a rebuild with the DB locally. Remember, a WBI License included a license for a restricted version of DB2 for use with broker so it can't be a cost issue can it?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| EKhalil |
| Posted: Fri Jul 11, 2008 12:19 pm Post subject: |
|
|
Voyager
Joined: 29 Apr 2003
Posts: 99
Location: Boston, MA
|
I am new to the company im currently at and there are many configurations that could use a make over. Besides the performance improvement are there any other reasons ??? At this place db2 is out of questions cause DBA's will only support Oracle. The downgraded performance is not felt by the business so its going to be hard to justify purchase of new Oracle licenses and DBA's support efforts just by stating the obvious...
How about the issue I mentioned earlier in the post : what does happen to persistent messages when the broker loses its database. I know that MQ Everyplace and SCADA persistent messages are stored inside the Broker's tables...but there is no mention bout the mechanics used with persistent MQ messages. I picutured a persistent message in one of the queues it was destined to go in the event of failure...maybe even on input queue since broker doesn't really take it off, right ??? There is always a possiblity that a human error wiped it out...but honestly what happens step by step in the event such as the one I described ??? |
|
| Back to top |
|
|
| sridhsri |
| Posted: Fri Jul 11, 2008 12:28 pm Post subject: |
|
|
Master
Joined: 19 Jun 2008
Posts: 297
|
I respectfully disagree with smdavies. I believe the broker database should be on a remote machine. From V6, aggregation nodes no longer use the database. They use MQ Queue. I agree that while using Pub/Sub there could be a problem. That said, I still believe that the database and broker should not compete for the same resources (RAM, Processor etc). It could also cause problems while designing systems for HA. If the database went down, you would be bringing down the broker too and vice-versa.
As far as persistent messages being lost is broker, it is possible if the flows are badly designed. For Ex: If a node is not wired to any output, that message will be discarded. There could be other possible reasons too. |
|
| Back to top |
|
|
| EKhalil |
| Posted: Fri Jul 11, 2008 12:45 pm Post subject: |
|
|
Voyager
Joined: 29 Apr 2003
Posts: 99
Location: Boston, MA
|
yeah hear your point here...Any other inputs on the remote location of broker's database ???
K, so per IBM's response the broker should have continued processing blissfully unaware of its own database outage...
I am in the process of trying to recreate the issue...If someone has any know how bout this it would be greatly appreciated...I can not find any tech doc which would be detailed enough to describe what exactly happens or maybe Im just having bad luck finding one ??? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 11, 2008 12:45 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| sridhsri wrote: |
I respectfully disagree with smdavies. I believe the broker database should be on a remote machine. From V6, aggregation nodes no longer use the database. They use MQ Queue. I agree that while using Pub/Sub there could be a problem. That said, I still believe that the database and broker should not compete for the same resources (RAM, Processor etc). It could also cause problems while designing systems for HA. If the database went down, you would be bringing down the broker too and vice-versa.
As far as persistent messages being lost is broker, it is possible if the flows are badly designed. For Ex: If a node is not wired to any output, that message will be discarded. There could be other possible reasons too. |
  I disagree. The DB should be with the broker. Any problem in the network between the broker and its DB can cause unexpected results including a broker shutdown.
As for resource competion, the broker's DB uses only a minimum of resources and this is barely noticeable. Consider the Broker, its DB and QMGR a unit.
Pushing your logic to the extreme you should also access a remote qmgr as the local qmgr is competing for resources with the broker??
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| EKhalil |
| Posted: Fri Jul 11, 2008 1:13 pm Post subject: |
|
|
Voyager
Joined: 29 Apr 2003
Posts: 99
Location: Boston, MA
|
At my previous employer we had everything locally db2 and qmgr - this was initially set up by IBM and we stay configured like that. Here things are a bit different...I guess one can always do a cost benefit analysis 
Anyways, I'm still looking for a book or an expert who can tell me about broker's database outage impact on persistent messages it was processing while the crash occured...
Here is a snippet from IBM's response:
"Thanks for the update. If a broker's database goes down while a message flow deployed to this broker is processing a persistent message, the message should not get lost or fail. It should still continue to get processed. The broker would not detect its database failure until a command is run on the broker that require it to access its database. Regardless, considering its a persistent message it should not get lost anyways. Also, it should not matter if the broker database is local or remote. "
Unfortunately it is very broad and it will take me a while before I get to a higher level support person, so if anyone knows the precise mechanics please speak up |
|
| Back to top |
|
|
| sridhsri |
| Posted: Fri Jul 11, 2008 1:18 pm Post subject: |
|
|
Master
Joined: 19 Jun 2008
Posts: 297
|
| EKhalil, I want to understand why you think that the database was the reason your persistent messages were lost ? I agree with IBM's response. It doesn't make sense that broker db being unavailable should cause anything to persistent messages. I think its coincidental. There must be some logic in the flow (I suspect) that is causing this. |
|
| Back to top |
|
|
| EKhalil |
| Posted: Fri Jul 11, 2008 1:25 pm Post subject: |
|
|
Voyager
Joined: 29 Apr 2003
Posts: 99
Location: Boston, MA
|
| cause the flow has been in production for years and not once have they lost a message and MQ does not lose persistent messages and I have never lost a message ever . To top this I was lucky enough to never have to recover from MQ logs before either...The broker stopped responding yet did not produce errors or any entries for that matter... |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jul 11, 2008 2:19 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
What is the transaction mode of the flow? If its set to yes, if the message is persistent, if you have fail queue and/or catch queues and /or dead letter queues the message won't be lost.
But if you pick up a persistent message outside of syncpoint and the flow errors and you don't have your nodes wired correctly, then sure, the message is gone. Its not MQ's fault, its not the Broker's fault, its the application developer's fault. The message was in memory and you did not handle it. The fact that it ran for years OK may only mean you've ben lucky and haven't hit the particular error condition you failed to consider originally.
Regarding the DB placement issue - if you are talking about the DB the Broker uses I think its follhardy to run it on any server other than the Broker's server! The Broker's DB is actually used a lot less than people think as the broker processes messages all day long. It will not compete for resources.
If its an existing application DB that your flow is connecting to, well you got no choise, the DB is where it is.
If you are creating a new application DB and have a choice where to put it, I would not put it on the same server as the broker. Why choose to have it run on the same CPUs. memory, I/O as the broker?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jul 11, 2008 3:43 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
In an HA configuration, running the Broker's database on a separate logical partition from the Broker itself vastly simplifies the failover configuration, and vastly shortens the failover time.
Especially if I have multiple broker instances sharing a database, then it makes a lot more sense to put the database separate.
I would not necessarily advocate putting the database logical partition on a separate physical machine, however. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Jul 11, 2008 10:01 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Also bear in mind that remote connection to Application databases can give you problems all of their own.
In one of the Customers I support there are two remote SQL Server DB's. There are on different servers.
I still have a Problem Report open from nearly 18months ago where a network glitch caused a flow to error out and put the message on its error queue.
The customer refuses to accept that it was a network outage for a few milliseconds that caused the problem even though the System Log messages clearly states that a network connection was lost.
There are plusses & minuses about where Application DB should be located. This clearly depends upon the overall system design & constraints.
Broker DB's as several people have suggested should really be on the same system as the Broker itself.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| SAFraser |
| Posted: Sun Jul 13, 2008 7:40 am Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
We have been running about 50 brokers for five years with their databases on separate Oracle database clusters. Our message flows are simple and do not require the DB in runtime.
I have many times allowed the DBAs to take down the brokers' DB for maintenance - never a problem in five years. Also, the DB has crashed a couple of times during production hours - never a problem.
One time, I had a DB error in the WMQI error log when there was a network glitch. Just the once.
DB2 is not a supported product at our agency. If I could win the fight to get the brokers' DBs as local DB2 databases, I would have to maintain them myself. I assure you, that would be a bad business decision!
Some may say, "well, you have just been lucky." Maybe so. Just thought I'd share the experience.
Shirley |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jul 13, 2008 8:40 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| SAFraser wrote: |
We have been running about 50 brokers for five years with their databases on separate Oracle database clusters. Our message flows are simple and do not require the DB in runtime.
I have many times allowed the DBAs to take down the brokers' DB for maintenance - never a problem in five years. Also, the DB has crashed a couple of times during production hours - never a problem.
One time, I had a DB error in the WMQI error log when there was a network glitch. Just the once.
DB2 is not a supported product at our agency. If I could win the fight to get the brokers' DBs as local DB2 databases, I would have to maintain them myself. I assure you, that would be a bad business decision!
Some may say, "well, you have just been lucky." Maybe so. Just thought I'd share the experience.
Shirley |
Shirley,
Just a guess here but maybe you can confirm:
a) you do not use pub/sub heavily
b) your flows are running all the time and you do not run into the scenario where the broker swaps out flow instances from an eg and needs to reload them.
Thanks
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
