| Author |
Message
|
| jnoubi |
 Posted: Sat Feb 11, 2012 12:48 am Post subject: Sharing variables between brokers |
 |
|
Novice
Joined: 26 Sep 2006
Posts: 24
|
We have a requirement to replace a global id generator in our environment. Currently, we use a database to generate a sequence number for each new request and thus each new transaction has a unique and incremental id. The first problem with this design is that database access is slow. The second is that it is a single point of failure.
I am wondering if someone is able to use brokers to do the same job of generating ids, where two brokers (active/passive) can be hosting an id generator?
I know shared variables are only within one EG but I am wondering if we can use a semaphore+shared variables to solve this problem? any ideas?
Appreciate the feedback.
regards,
Jnoubi |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Sat Feb 11, 2012 1:44 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
How slow is slow (for the DB) access? Have you measured it?
How many messages/sec do you expect to process?
Have you looked at how the DB is tuned?
Have you looked a how the network is configured?
You would be surprised how much a simple thing like turning off AutoNegociate and configuring Full Duplex can do to the network throughput.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Feb 11, 2012 2:40 am Post subject: Re: Sharing variables between brokers |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| jnoubi wrote: |
We have a requirement to replace a global id generator in our environment. Currently, we use a database to generate a sequence number for each new request and thus each new transaction has a unique and incremental id. The first problem with this design is that database access is slow. The second is that it is a single point of failure.
I am wondering if someone is able to use brokers to do the same job of generating ids, where two brokers (active/passive) can be hosting an id generator?
I know shared variables are only within one EG but I am wondering if we can use a semaphore+shared variables to solve this problem? any ideas?
Appreciate the feedback.
regards,
Jnoubi |
If all you use it for is generating a unique Id, why don't you use the UUID generator of the broker? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jnoubi |
| Posted: Sat Feb 11, 2012 3:50 am Post subject: |
|
|
Novice
Joined: 26 Sep 2006
Posts: 24
|
Thanks for your replies.
Answering fjb_saper: there are two issues with using the UUID generator of the broker. First, the the unique is is to be sequential, i.e. each time a new id is created it has to be an increment over the previous one. Second, this broker will become a single point of failure, so if the broker is down, the sequence is lost and this will break the requirement.
Answering smdavies9: the network and database database tuning are not my area of expertise but there were some vendor specialists who looked into this and optimized it. The expected workload is expected to cause dramatic slowdown of the response and one of the recommendations was to move this sequencing to a different solution off the databae. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Feb 11, 2012 6:20 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Each EG is a process.
You can do any kind of interprocess communication that you want to design and create your own global cache that will hold this value, and provide any requestor with 'the next' unique id.
This is a usecase where you might consider something like solidDB as it will likely be a lot faster than a standard database.
You could look at the cache nodes in the relevant supportPac.
You could look at using an MQ Queue to hold a message, and use browse or syncpoint to ensure that you know who's got the button.
Or you could do the smart thing and remove the requirement that unique id's need to be sequential. Or at least change the definition of "sequential" to mean "strictly larger than" and then just use seconds since epoch. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sat Feb 11, 2012 8:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jnoubi wrote: |
| First, the the unique is is to be sequential, i.e. each time a new id is created it has to be an increment over the previous one. Second, this broker will become a single point of failure, so if the broker is down, the sequence is lost and this will break the requirement. |
Why does it need to be sequential? What's the technical requirement for this? If there's no technical requirement for this, what's the business requirement apart from "the business have indicated that this should be sequential"? A large number of business requirements seem to be things the business think are important (like confirmation & audit reports) but add no value to the system & are in some cases unusable.
If there is some valid reason for this, does it justify the extra cost in design and infrastructure that it will require? That's not a question for you but a question for whoever's setting the requirement.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Feb 11, 2012 10:20 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
If there is some valid reason for this, does it justify the extra cost in design and infrastructure that it will require? That's not a question for you but a question for whoever's setting the requirement. |
Especially if you want to avoid a single point of failure.
This WILL COST SOME MONEY.
I'd still like to know the estimated message rate. Quite a few of us here do have experience of running broker in a high throughput operation. Many of these instances involve using a Database.
I can personally testify that with proper design and end to end system tuning, you can move from quadruple the throughput.
This was in an old 2 socket Pentium P4 system. Modern servers run a whole lot faster.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| jnoubi |
| Posted: Sun Feb 12, 2012 4:25 am Post subject: |
|
|
Novice
Joined: 26 Sep 2006
Posts: 24
|
Thanks guys for your input and helpful discussion.
I have come across the Sequence/Resequence nodes which seem to be promising. The nice thing about them is that they store the sequence number on an MQ queue, which means that it is still accessible if the broker is down and it will be recovered if the whole system is down.
As for the rate of processing and architecture, this system is in a governmental environment so I cant disclose much here, but I appreciate your feedback regarding the scalability of message broker. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Feb 12, 2012 7:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| jnoubi wrote: |
| I have come across the Sequence/Resequence nodes which seem to be promising. |
You need to make sure you thoroughly understand the scaling impact of using these nodes.
I am not saying that they will impede performance.
I'm just saying you need to look very closely at the documentation and completely understand what it says about deploying particularly the Sequence node to more than one EG and to more than one Broker. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Feb 12, 2012 11:42 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Can you have the source process (requester) assign it the unique Id?
Barring that, can you have an app as pre-processor assign it?
Do you have message affinity or do you just need to have a sequential number range assigned to the process?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Feb 13, 2012 5:43 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
solidDB provides a seamless transition to an in-memory DB2 database that is ten times faster than disk-based DB2 database. Direct drop-in replacement.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Mon Feb 13, 2012 6:21 am Post subject: Re: Sharing variables between brokers |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
A uniqueId (provided by Broker) is nice, but a lot of times you need sequence-numbers (e.g. that actually roll over "9998", "9999", "0000", "0001, ...) to have message-traceability.
We have implemented that through DB-accesses (each time the flow gets executed) "SELECT from DUAL", which is performing surprisingly well. Databases have special support for Sequences, in Oracle it is called "from DUAL"! I must add, that Broker and Oracle sit quite near to each other with good hardware around (and probably we do not have your throughput-rates  ).
I am just adding here that keeping counters reliably/persistent with the help of the DB is a good choice. Anything else: DB-Performance, -reliablity is then DB-stuff (as discussed already in this thread).
MQ for keeping (very small portions of) data persistent (and still fast) should also be an option.
Else use a shared row, and if broker stops/crashes (does this happen?) put in automatic/manual instructions to pick up from the sequence again (e.g. through loading the "manually" set current sequence counter from the DB, or by checking the last message-sequence-number from an actual message). As I understand you only have a single sequence number.... o.k. that only works when there is only one Broker.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 13, 2012 7:21 am Post subject: Re: Sharing variables between brokers |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqsiuser wrote: |
A uniqueId (provided by Broker) is nice, but a lot of times you need sequence-numbers (e.g. that actually roll over "9998", "9999", "0000", "0001, ...) to have message-traceability.
We have implemented that through DB-accesses (each time the flow gets executed) "SELECT from DUAL", which is performing surprisingly well. Databases have special support for Sequences, in Oracle it is called "from DUAL"! I must add, that Broker and Oracle sit quite near to each other with good hardware around (and probably we do not have your throughput-rates ).
I am just adding here that keeping counters reliably/persistent with the help of the DB is a good choice. Anything else: DB-Performance, -reliablity is then DB-stuff (as discussed already in this thread).
MQ for keeping (very small portions of) data persistent (and still fast) should also be an option.
Else use a shared row, and if broker stops/crashes (does this happen?) put in automatic/manual instructions to pick up from the sequence again (e.g. through loading the "manually" set current sequence counter from the DB, or by checking the last message-sequence-number from an actual message). As I understand you only have a single sequence number.... o.k. that only works when there is only one Broker. |
all well and good. We don't know the throughput rates they want. If the OP were to share that with us, I am sure that a suitable solution could be proposed. Without that sort of critical bit of data we are all standing on the poop deck of a ship going round the horn facing into the wind and doing No 1's.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2012 7:24 am Post subject: Re: Sharing variables between brokers |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| we are all standing on the poop deck of a ship going round the horn facing into the wind and doing No 1's. |
I don't do No. 1's on a poop deck. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
