| Author |
Message
|
| wmbwmq |
 Posted: Mon Jul 18, 2011 9:52 am Post subject: Filebased lookup |
 |
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Hiya,
We have a requirement to use a lookup table which contains mostly static data with a possibility of change once every week or so. Now with V7 we are trying to avoid using a database because lookup is all we need. So what are the other options we have?
I was thinking of using an IA91 with a dedicated flow to read from file and cache it with unlimited expiry. And whenever a change is made to lookup then restart the IA91 flow. Can you pls comment if this would be reliable approach or if you think there are other options.
Thanks |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Jul 18, 2011 10:17 am Post subject: Re: Filebased lookup |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| Can you pls comment if this would be reliable approach or if you think there are other options. |
I shudder to think how many times the question of static data lookup by flows has been discussed in this forum, though I find your use of the phrase "mostly static" interesting - exactly how static is this? How often the data changes and if it's written back to has an impact.
I'd advise you to review to forum where this has been discussed to death, with the pros & cons of each method expounded. I'll settle for asking another common question in this context - where does this "change once a week or so" come from? The forum is littered with people who were told this by their end users, designed a solution that allowed for it and discovered the data hasn't actually been changed in years.
This is relevent as if the data really does change every week or so that favours some designs over another. If the users just think it might change once a week, or don't really know when they'd change it but are playing it safe, a different and simpler solution might be better.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jul 18, 2011 10:19 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
This is quite a common question.
I expect one of my esteemed colleagues will be along shortly to suggest using a Java Singleton. That may be ideal if you are into that new fangled Java language (only joking..)
Whe I'd recommend is a database table with one row per flow. The primary key would be the flow name which is available in ESQL.
A timernode would be set for a refresh interval. I use 24hours.
Then a shared bit of ESQL (between the normal flow input path and the timer) synchronised with an ATOMIC structure.
It is not that difficult to implement. We use this for all flows. All the things like endpoint URL's, db usernames & passwords, DSN's, timeout values etc can be configured into the row variables.
Then when you make a change to a row all you need to do is stop and start the flow then they new variables will be in use.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 18, 2011 10:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| I expect one of my esteemed colleagues will be along shortly to suggest using a Java Singleton. |
I held back! I held back! I didn't say it.....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Mon Jul 18, 2011 10:38 am Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Thanks Vitor, smdavies the inputs.
Actually the lookup table is completely static and contains only environment related information(like in Production say we might have 32 instances of destination queues for loadbalancing and where as in QA only 8 instances). The table itself is very small. So using a database would be simply waste of license money?
And the lookup table is changed by the broker admin (say instead of 32 instances just send to 16 while maintainance is being done in other 16). This usually happens once a week or several weeks/months.
So you are suggesting not to use file based lookup at all? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jul 18, 2011 11:15 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Quote: |
The table itself is very small. So using a database would be simply waste of license money?
|
Are you saying that you don't have any supported databases that you can use?
Are you saying that your broker setup does not use a database anywhere?
If this is the case then IMHO, your site is in the minority.
Have you looked at the additional Databases that Broker 7.0.0.1 can support?
Some of those don't cost $$$$$
No I'm not suggesting using a file for this data.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 18, 2011 11:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| we might have 32 instances of destination queues for loadbalancing and where as in QA only 8 instances |
Why are not using clustering for this? Even if you have other static data you need to legitimately look up.
| wmbwmq wrote: |
| So using a database would be simply waste of license money? |
Hey, you're wasting WMQ's load balancing features. Why stop there? 
As my worthy associate points out, your site is mostly likely paying for a database anyway. There may well be one convieniently located for your use (but of course there may not be, especially if the database license is per-CPU or similar rather than site).
| wmbwmq wrote: |
| So you are suggesting not to use file based lookup at all? |
IMHO the general opinion is that file based lookup is the least favoured solution to your kind of problem, but I again urge you to review the debate so you can make an informed decision. Just because the majority here don't like it doesn't mean it's not the best solution for your unique set of circumstances.
Unless all it does it control loadbalancing in which case use clustering! 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Mon Jul 18, 2011 12:01 pm Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Yes this is for a small adhoc project now migrating to V7. From Broker point of view we don't have a database (We have other brokers using databases but we don't want to use them).
Can you pls explain why you would not suggest file based lookup? . I was with the impression that file based lookup would be faster than d/b based. Also, I am planning to read from file once and cache it using IA91.
Please comment. |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Mon Jul 18, 2011 12:05 pm Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Vitor,
We can't go with clustering. We need to preserve the message sequencing. (I know you would say it is not a good design etc, but our infrastructure demands it, sorry i can't explain any more on that) |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 18, 2011 12:23 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| Can you pls explain why you would not suggest file based lookup? |
It's cumbersome, hard to administer and somewhat clunky.
| wmbwmq wrote: |
| I was with the impression that file based lookup would be faster than d/b based. Also, I am planning to read from file once and cache it using IA91. |
a) Most modern database systems are quite fast enough to give a straight file read a run for their money, especially on a table as small as you're describing
b) If you're going to cache the result in memory what does the speed of the access method matter anyway? So long as it doesn't take 10 seconds to retrieve the data to put in the cache (and if it does, sack your DBA) you're fine.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 18, 2011 12:28 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| We need to preserve the message sequencing. |
You're load balancing while maintaining message sequencing? 
Correct me if I'm wrong, but the only way that I can see it work is if all the messages for a unit of work (business transaction, individual user, etc) have to arrive at the same target in sequence; it's the next unit of work can be balanced off onto another target. Otherwise the first time you diverted work off to a new queue manager it would complain that there were messages missing (because other queue managers had processed them).
You can use a WMQ cluster for that. Especially if you've got WMB as well.
I look forward to being corrected (because if that's not how you're load balancing your design is interestingly odd!)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Mon Jul 18, 2011 12:44 pm Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
message sequencing of independent messages (so it is to preserve message affinity and not MQ multiplart message segments).
The destination instance is decided by complex message broker code based on message payload. That is why i said i can't explain  but i can assure you we have business justification for that. (Here is an example, say you are sending hundreds of online banking transaction on different accounts in batches. Now how will you make sure that the transactions on an individual account happens in sequence?. So all transactions belonging to same account should all be sent to any one instance of the queue and read by the app with single thread in FIFO order. Hope I made it clear.
Now back to filebased lookup, can you pls elaborate. Like if we follow a precedure where we always stop the flow, change the file permission writable, change the lookup table, change permission to readonly and then start the flow. Do you still see any other issues that I am missing?
Thanks for your inputs btw. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 18, 2011 12:58 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| message sequencing of independent messages (so it is to preserve message affinity and not MQ multiplart message segments). |
Yes, I did think you meant individual messages that have to be processed in sequence. If you were talking about segmented message it really would have made no sense.
| wmbwmq wrote: |
| The destination instance is decided by complex message broker code based on message payload. |
That's potentially a worse decision that message affinity. You're using WMB as a workflow engine.
| wmbwmq wrote: |
| Now how will you make sure that the transactions on an individual account happens in sequence?. So all transactions belonging to same account should all be sent to any instance of the queue and read by the app with single thread in FIFO order. |
By making sure they're grouped correctly. That's what I did last time I load balanced hundreds of banking transactions where some needed to be processed together over a WMQ cluster.
| wmbwmq wrote: |
| Hope I made it clear. |
You did and if your load balancing solution makes you happy then go in peace.
| wmbwmq wrote: |
| Now back to filebased lookup, can you pls elaborate. Like if we follow a precedure where we always stop the flow, change the file permission writable, change the lookup table, change permission to readonly and then start the flow. Do you still see any other issues that I am missing? |
Look through the forum. Seriously. This has been done to death. And if you're happy with file based look up supported by that procedure go for it. You're the only one who can make the decision; we can't possibly know all the nuances of your site.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Mon Jul 18, 2011 1:13 pm Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Vitor
We don't have the liberty of grouping the incoming messages. The fronend is tiny and 3rd party software. And there are many functionally different types of frondends 
And thanks for the help anyway. Basically we can get a license for database(not my money but i thought it would be a overkill to use a database just to maintain a small table. Besides database needs it's own periodical maintainance that has direct downtime impact on broker. On top of that we have VCS clustering so it would be a big pain.
I have decided to go with file based lookup. Will be doing elaborate performance and reliability testing before we go live with V7. Will update the forum on my findings. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 19, 2011 4:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| I have decided to go with file based lookup. |
Good for you.
| wmbwmq wrote: |
| Will update the forum on my findings. |
I hope it works out for you & it will useful information in the "static data" debate. When it starts again.
(Can't believe no-one's suggested a Singleton.....)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
