| Author |
Message
|
| jmq |
 Posted: Mon Oct 25, 2010 3:24 pm Post subject: WMB Design query |
 |
|
Newbie
Joined: 25 Oct 2010
Posts: 4
|
Hello Friends,
I am facing a design dilema, kindly help if possible.
I have file input node giving multiple xml messages out(approx 5-10) at frequency of every 2 minutes. All msgs needs to be persistent. What could be a good design solution with a reason.
1)WMB directly updating the database using either compute, database node etc. But good design suggests low i/o
2)or have a stored procedure at database end but 2 phase commit will have much tighter control in option 1.
3)or any better option.
Thanks in advance. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Oct 26, 2010 4:07 am Post subject: Re: WMB Design query |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jmq wrote: |
| I have file input node giving multiple xml messages out(approx 5-10) at frequency of every 2 minutes. All msgs needs to be persistent. |
Pedantically, files are not persistent. They need to be managed carefuly for all the reasons previous discussed in this forum.
| jmq wrote: |
| What could be a good design solution with a reason. |
A good design is one that meets your requirements.
| jmq wrote: |
| 1)WMB directly updating the database using either compute, database node etc. But good design suggests low i/o |
How is this higher i/o than any other solution? What's your thinking here?
| jmq wrote: |
| 2)or have a stored procedure at database end but 2 phase commit will have much tighter control in option 1. |
2 phase commit with what? You've not described any transactional partner in this scenario. Certainly not with whatever's writing the files.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jmq |
| Posted: Tue Oct 26, 2010 5:08 am Post subject: |
|
|
Newbie
Joined: 25 Oct 2010
Posts: 4
|
Thanks for your reply Vitor.
In the option 1,High IO here means wmb is continuosly database logging every minute.
Is wmb directly updating databse a good design?
option 2 suggests stored procedure call triggered by wmb at remote user database
in the option 1 and 2, the 2 phase commit with WMQ as a transaction co-ordinator and wmb and database are transaction partners.
Or any better option? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 26, 2010 5:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jmq wrote: |
In the option 1,High IO here means wmb is continuosly database logging every minute.
Is wmb directly updating databse a good design? |
I don't see how this is higher i/o than options 2. Either WMB issues an insert or update for each message, or it's issuing a stored procedure call for each message that then issues an insert or update. In either event (presumably) the same database tables are affected to the same degree.
| jmq wrote: |
| in the option 1 and 2, the 2 phase commit with WMQ as a transaction co-ordinator and wmb and database are transaction partners. |
Since when has a queue manager acted as a transaction co-ordinator? And how is the broker updating a database a 2 phase commit? It's not async any more than any other application is.
I don't understand a) what you're driving at b) where you see more i/o in one solution than another or c) why this is a dilema.
If the load of this solution peaks at 10 databse actions every 2 minutes I'm also uncertain why you're so hung up on this. 10 actions every 2 seconds might be a cause for tuning, but not at those low volumes unless the XML is huge (50Mb+). Is it?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jmq |
| Posted: Tue Oct 26, 2010 5:50 am Post subject: |
|
|
Newbie
Joined: 25 Oct 2010
Posts: 4
|
|
| Back to top |
|
|
| rajmq |
| Posted: Tue Oct 26, 2010 6:16 am Post subject: |
|
|
Partisan
Joined: 29 Sep 2002
Posts: 331
Location: USA
|
Yes. Generally it is acceptable but again while doing a design there are so many other factors to be considered.
Some more information for your opt2
If you are using oracle 10g.There is a feature called DML, invoke the store procedure from WMB and SP's will insert the data to db.
If any issue in the data or insert, you can store failure message automatically in the failure tables using DML functions.
WMB Flows ---> Database (Store Procedures) --> Main table
DML Functions -----> Error table
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Certified System Administrator - WebSphere Business Integration Message Broker V6.0 |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 26, 2010 6:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jmq wrote: |
| Sorry for wrong terminology. I mean Global co-ordinated transaction with WMQ act as transaction manager |
And I still don't see why you'd use 2-phase commit (with all the overhead that involves) in this simple, local, scenario. There's nothing in what you've said that's "global" to be co-ordinated.
Which is not to say you couldn't. It's just a lot of setup work & cpu for no apparent benefit.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 26, 2010 6:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Vitor's point is that you don't get any XA at all when starting a flow with FileInput.
FileInput is entirely non-transactional. There's otherwise nothing about the set of requirements you have stated that eliminates Broker as a design choice for this.
The use of a stored procedure to do the final insert has little to do with transactionality. If you want to keep the broker flow coded at a degree of separation from the logical structure of the tables, you could create a stored procedure that took in data in a form that made sense at the broker, and then did some transformation and rearrangement so that the structure of the tables could be changed without altering the Broker flow.
But that's a design choice and has little to do with performance, unless the transformation is sufficiently faster in the DB than in ESQL. But you won't know *that* until you test it. |
|
| Back to top |
|
|
| jmq |
| Posted: Tue Oct 26, 2010 8:51 am Post subject: |
|
|
Newbie
Joined: 25 Oct 2010
Posts: 4
|
Hello Vitor,mqjeff,
The reason I think I will need global transaction is that I need to update remote oracle database and would want to ensure that in any problem either at wmb or database the database is not committed to ensure data
integrity.
How will I ensure global transaction to ensure data integrity? if XA cant be enabled at fileinput? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 26, 2010 8:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jmq wrote: |
| The reason I think I will need global transaction is that I need to update remote oracle database and would want to ensure that in any problem either at wmb or database the database is not committed to ensure data integrity. |
Are you sure you mean think? You don't, unless your broker flow is bizarre beyond words. Or you mean something other than "not running on the same machine as broker" by "remote".
| jmq wrote: |
| How will I ensure global transaction to ensure data integrity? if XA cant be enabled at fileinput? |
You won't. It can't.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Oct 26, 2010 8:57 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Welcome to the wonderful world of compensating transactions.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 26, 2010 8:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| jmq wrote: |
| How will I ensure global transaction to ensure data integrity? if XA cant be enabled at fileinput? |
You won't. It can't. |
Just don't start your flow with a FileInput node.
 |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 26, 2010 8:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Vitor wrote: |
| jmq wrote: |
| How will I ensure global transaction to ensure data integrity? if XA cant be enabled at fileinput? |
You won't. It can't. |
Just don't start your flow with a FileInput node.
|
Now you've done it...... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
