| Author |
Message
|
| deepak_paul |
 Posted: Wed Aug 10, 2011 5:35 am Post subject: In-memory cache using Row Shared variables |
 |
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
I have a requirement and couple of questions with regard to DB caching which are as below.
Requirement:
1. Get all table values from DB and construct an XML message per record with unique and cache that XML message in memory across EG and Broker and keep them in sync always.
2. Request Cache retrieval with the key to obtain the respective XML message in cache.
3. This database is DB2 and the amount of DB data to be cached must be around 250 - 500KB( less than 5000 records)
4. Database change is done as and when required and expected to not be often. (maybe once in a week or month or year)
Questions:
1. How we can make other message flow in the same execution group access the shared row variable value set in a message flow? Interfacing caching message flow with other Message flow via MQ/JMS( or any other possible means) could be an approach to go with? Any other possible approach?
2. How we can make other Execution Group in Broker access the shared row variable value set in a message flow in the EG? Interfacing the Caching message flow with other EG via MQ/JMS( or any other possible means) could be an approach to go with? Any other possible approach?
3. What are the advantages of using Row Shared variable over Cache nodes(IA91 Support Pack) other than a fact that ROW SHARED variables can hold complex tree structure [The disadvantage in using Cache nodes for this scenario is that we would not be able to store the value as XML structure as the fact is that Cache nodes use Key-Value pair always.[Value is most likely a char]]
Any suggestions.
_________________
Regards
Paul |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Wed Aug 10, 2011 5:40 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Hi Paul,
The best DB cache option IMHO is solidDB. You use it just like a DB2 instance, only all DB2 operations hit memory not disk.
You can share data through solidDB. If solidDB is not an option for you, you can use a Singleton to share information.
If you don't want to code the Singleton yourself, you can use the IA91 IBM support pack which implements a Singleton, although I personally dont like that one.
So my first preference is solidDB and my next very close preference is custom coded Singleton. Very last way down on the list is the support pack.
Lance
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 10, 2011 5:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You can't access shared variables from outside the same message flow they are declared in. You would have to set up some mechanism to request the cache and return it (as you say, possibly using MQ or JMS).
The same thing is true of the EG.
The advantages of the SHARED ROW is that it is provided with the product. Some will also claim that it performs better than IA91 - I've not run tests.
You could look at something like ExtremeScale or SolidDB instead of building this kind of cache yourself.
You could look at using a java singleton to share data across each EG (one EG == one JVM). |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Fri Aug 12, 2011 7:11 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
Thank you Lancelotlinc and mqjeff for your valuable suggestions.
We are trying push hard on bringing SolidDB in our Enterprise.
While exploring ROW SHARED variable, We were advised by IBM SME that "ROW SHARED variable loops thru all the records even if it sees the matching value in the first row itself." This may sometimes degrade the performance too. Though this sounds correct, it is obvious that accessing ROW SHARED variables is much more faster than accessing Database directly.
With that fact, we would like to go with ROW SHARED variables approach in order to gain some performance for our scenerio as much as we can. Now, the question is,
1. While interfacing/accessing with ROW SHARED variable that is in another Message flow/Execution Group, we would like to see which transport protocol is faster between MQ or TCPIP. We did a quick load test to compare the performance between these transports. What we observed is that MQ takes very less time to process/access RS than TCP IP. However, our assumption is that MQ indeed uses TCP IP transport to communicate. So we are wondering why accessing RS variables with TCPIP transport is not fast but lot slower than with MQ.
Please correct us if our assumption is wrong and suggest any fastest way to access RS variable.
_________________
Regards
Paul |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Mon Aug 15, 2011 7:27 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
If we look at the Websphere MQ v6 Fundamentals redbook(http://www.redbooks.ibm.com/redbooks/pdfs/sg247128.pdf), we will see below piece of information.
| Quote: |
5.3.8 Providing network access to a queue manager
Queue managers and clients within the infrastructure need to establish
communication with a queue manager over a network. To do this, they use an
underlying communications protocol. The following protocols can be used for
WebSphere MQ communication:
_ Transmission Control Protocol/Internet Protocol (TCP/IP)
_ SNA LU 6.2 (Windows and z/OS only)
_ NetBIOS (Windows only)
_ SPX (Windows only)
This book only discusses the TCP/IP protocol. For details of the other protocols,
see WebSphere MQ System Administration Guide, SC34-6584.
In order for a communication link to be established over TCP/IP, the queue
manager must be listening for that connection to arrive on a particular port.
|
Any thoughts on what transport protocol is being used to communicate and transfer message within Queue Manager or Broker to Queue Manager?
_________________
Regards
Paul |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 15, 2011 7:32 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
MQ uses any and all of those transports to implement MQ and MQI channel traffic, each of which is a proprietary MQ protocol.
You're asking the wrong question. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Aug 15, 2011 11:12 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| deepak_paul wrote: |
Any thoughts on what transport protocol is being used to communicate and transfer message within Queue Manager or Broker to Queue Manager? |
Why?
Why do you want to know or even care what goes on between Broker and its Queue Manager?
All you need to know is that Broker connects directly to its Queue Manager. It does NOT use a Client Connection.
Because this is all happening inside a Queue Manager it is all internal. How WMQ Does that is a different problem.
The answer to that one is probably, 'it depends'.
A clue as to how things move aroun on for example, Linux/Unix is the kernel configuration you have to do before you install and cofigure WMQ. This is all based upon semaphores and shared memory. Then Messages are (if persistent) stored in files in directories that have the names of the queue that are queued under. That is not rocket science but just a result of doing some observations AND reading the documentation.
I am sure if you were to dig deep enough into the Linux Kernel and activare the kernel debugger and spent a god month of sundays pouring over kernel code you could decipher the mechanisms that WMQ uses to move stuff around. But really, why? Why? and thrice Why?
Why bother.
Is the sky going to fall in if you don't know?
Id the earth suddenly going to be revealed as being flat?
It works. Isn't that enough?
[/quote]
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kathir |
| Posted: Mon Sep 26, 2011 4:57 am Post subject: shared row and property file |
|
|
Apprentice
Joined: 12 Sep 2011
Posts: 30
|
| How to read from property file and implement shared row( memory cache) |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Sep 26, 2011 5:00 am Post subject: Re: shared row and property file |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| kathir wrote: |
| How to read from property file and implement shared row( memory cache) |
You've been told this, several times. You can repeat asking this question, in several different ways, and you will not get a significantly different answer, or will you likely get the free training you are hoping for.
The answer I will give here is "Hire a programmer".
You are asking a question that is akin to asking "how do I turn left in a motor vehicle". If you don't know how to drive, it won't help you to be told "you turn the wheel". And, if, having been told that you turn left by turning the wheel, you then get into the driver's seat of an automobile and get on the highway and STILL DO NOT KNOW HOW TO DRIVE, then you will likely crash. And kill someone. |
|
| Back to top |
|
|
| MrSmith |
| Posted: Mon Sep 26, 2011 6:03 am Post subject: |
|
|
Master
Joined: 20 Mar 2008
Posts: 215
|
nice analogy  |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Mon Sep 26, 2011 7:06 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi,
| Quote: |
| We were advised by IBM SME that "ROW SHARED variable loops thru all the records even if it sees the matching value in the first row itself." |
I think this is true if you are using SELECT -clauses against actually of any internal data or sub tree (Environment, XMLNSC....). But If you use references and build your own loops you can do it more efficiently.
--
Marko |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Sep 26, 2011 7:18 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
For few number of cache rows, some solution like Shared Row may be a good choice. As the amount of data you are caching increases, solidDb looks more and more attractive.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Mon Sep 26, 2011 7:23 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
| lancelotlinc wrote: |
| For few number of cache rows, some solution like Shared Row may be a good choice. As the amount of data you are caching increases, solidDb looks more and more attractive. |
I agree.
--
Marko |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Thu Sep 29, 2011 8:53 pm Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
| lancelotlinc wrote: |
| For few number of cache rows, some solution like Shared Row may be a good choice. As the amount of data you are caching increases, solidDb looks more and more attractive. |
I agree too.
1. Just wanted to see if we have some cache performance statistics of Shared Row, Cache support pac node and solidDb for different message size.
2. Curious to know if next Message Broker release will have this solidDb embeded in broker itself for caching purpose?
_________________
Regards
Paul |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Sep 30, 2011 1:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| deepak_paul wrote: |
| 1. Just wanted to see if we have some cache performance statistics of Shared Row, Cache support pac node and solidDb for different message size. |
There is no "we" here. There is only you and I and a bunch of other random strangers on the internet with no common organization nor under any common SLA>
| deepak_paul wrote: |
| 2. Curious to know if next Message Broker release will have this solidDb embeded in broker itself for caching purpose? |
Yes, it's a question. Nobody here is able to tell you anything resembling an answer here. Any such information about potential future capabilities of IBM message broker is not public information and would not be released except under NonDisclosure agreements. And since this is a random public internet forum, anyone who had signed those agreements would not be able to discuss such things here. |
|
| Back to top |
|
|
|
|
