| Author |
Message
|
| EricCox |
 Posted: Wed Apr 02, 2014 7:18 am Post subject: How to load static data into an execution group's JVM heap |
 |
|
Master
Joined: 08 Apr 2011
Posts: 292
|
To all,
In this article it states we can load static data into an execution group's JVM heap. How do I do this?
Are they referring to using a FileRead Node to read static data from a file and load it to a shared variable in the Message Flow?
In my particular case we have a table with four columns being called 2 million times per hour consistently. We need to move that static data, 480 rows, into an in memory structure in Broker so we do not incur the cost of making the remote call to the database, Oracle in this case.
We need to be able to search via two out of the four columns and return the other two description fields that are found in the lookup. This is all string data to search on and strings being returned.
What is the best, most simple way to move this data into an in memory structure that can be searched by two out of the four columns?
Thanks as always,
EMC |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Apr 02, 2014 7:36 am Post subject: Re: How to load static data into an execution group's JVM he |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| EricCox wrote: |
| In this article it states we can load static data into an execution group's JVM heap. How do I do this? |
What article?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| EricCox |
| Posted: Wed Apr 02, 2014 7:38 am Post subject: Article |
|
|
Master
Joined: 08 Apr 2011
Posts: 292
|
|
| Back to top |
|
|
| ghoshly |
| Posted: Wed Apr 02, 2014 8:56 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
| If WMB 8 is the one being used, why you are not using inbuilt extreme scale? Use hash map to capture the complete data into Global map. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Apr 02, 2014 9:52 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| ghoshly wrote: |
| If WMB 8 is the one being used, why you are not using inbuilt extreme scale? Use hash map to capture the complete data into Global map. |
Using the global cache is not as easy as you make out. Given the requirements given to us, I'd really not recommend the global cache.
If another reader of this thread has solved the searching for data in the cache problem (making it efficient) then I am sure that sharing the details with us would be well received.
AFAIK, the V8 Broker cache is a stripped down version of 'eXtreme Scale'. The V9 one is a little better. Until you actually get down to it and do a POC you can't really understand the limitations of the cache.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| EricCox |
| Posted: Wed Apr 02, 2014 10:00 am Post subject: SHARED Variables |
|
|
Master
Joined: 08 Apr 2011
Posts: 292
|
I'm looking at using something like this:
DECLARE CODES SHARED ROW;
SET CODES = SELECT * FROM Database.CODES;
But I am at a loss to find any documentation on how to use a WHERE clause in a SELECT on the ROW datatype. Does anyone have an example of how to use a SELECT and WHERE clause on a ROW Datatype.
CODES
Key1 | Key2 | Desc1 | Desc2
1 ABC Short Long
2 DEF Short Long
3 GHI Short Long
I need to then be able to do something like:
SELECT Desc1, Desc2 FROM CODES
WHERE Key1 = 1 AND Key2 = 'ABC';
Of course I need to pass in the values 1 and 'ABC' dynamically as variables but you get it.
Thanks |
|
| Back to top |
|
|
| EricCox |
| Posted: Wed Apr 02, 2014 11:42 am Post subject: Following the Example Throws Exception |
|
|
Master
Joined: 08 Apr 2011
Posts: 292
|
I've followed the example in the docs for DECLARATION but am encountering an error.
Here is the code from the example:
CREATE SCHEMA testSchema
DECLARE mySharedRow SHARED ROW;
DECLARE initialized SHARED BOOLEAN myINIT();
CREATE COMPUTE MODULE testModule
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
SET OutputRoot.XMLNSC.Top.TEST.Result1 VALUE = initialized;
SET OutputRoot.XMLNSC.Top.TEST.Result2 = mySharedRow;
END;
END MODULE;
CREATE FUNCTION myINIT( ) RETURNS BOOLEAN
BEGIN
LOG EVENT VALUES('myINIT CALLED');
SET mySharedRow.Top[] = SELECT A.MyCol1, A.MyCol2 from Database.Test AS A;
RETURN TRUE;
END;
Here is the error I'm getting:
2014-04-02 15:32:01.553569 5157 UserTrace BIP2574W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '4.48') : Finding next SELECT result.
2014-04-02 15:32:01.553580 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''0134''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-02 15:32:01.553602 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''CHK''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-02 15:32:01.553612 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''Sweep To Line Paydown''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-02 15:32:01.553621 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''Y''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-02 15:32:01.553629 5157 UserTrace BIP2574W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '4.48') : Finding next SELECT result.
2014-04-02 15:32:01.553641 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''0135''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-02 15:32:01.553650 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''CHK''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-02 15:32:01.553659 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''POS Debit''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-02 15:32:01.553668 5157 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''Y''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-02 15:32:01.553676 5157 UserTrace BIP2574W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '4.48') : Finding next SELECT result.
2014-04-02 15:32:01.553699 5157 UserTrace BIP2571W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '4.48') : Failed to find any more items in the FROM clause satisfying the WHERE clause.
2014-04-02 15:32:01.554765 5157 Error BIP2628E: Exception condition detected on input node 'Test_Flow.FRAUD.ACCT.INFO.RQ'.
The input node 'Test_Flow.FRAUD.ACCT.INFO.RQ' detected an error whilst processing a message. The message flow has been rolled-back and, if the message was being processed in a unit of work, it will remain on the input queue to be processed again. Following messages will indicate the cause of this exception.
Check the error messages which follow to determine why the exception was generated, and take action as described by those messages.
2014-04-02 15:32:01.554775 5157 RecoverableException BIP2230E: Error detected whilst processing a message in node 'Test_Flow.Load TRANCODES'.
The message broker detected an error whilst processing a message in node 'Test_Flow.Load TRANCODES'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2014-04-02 15:32:01.554783 5157 RecoverableException BIP2934E: Error detected whilst executing the function or procedure ''myINIT''.
The message broker detected an error whilst executing the function or procedure ''myINIT''. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2014-04-02 15:32:01.554788 5157 RecoverableException BIP2488E: ('.myINIT', '4.10') Error detected whilst executing the SQL statement ''SET trancodesSharedRow.Top[ ] = (SELECT T.TRANCODE AS TRANCODE, T.ACCTTYPE AS ACCTTYPE, T.TRANDESC AS TRANDESC, T.MEMO_DESC AS MEMO_DESC FROM Database.giabkr.TRANCODES AS T);''.
The message broker detected an error whilst executing the given statement. An exception has been thrown to cut short the SQL program.
See the following messages for details of the error.
2014-04-02 15:32:01.554793 5157 RecoverableException BIP2112E: Message broker internal error: diagnostic information '', '{1}'.
An internal software error has occurred in the message broker. Further messages will indicate the effect of this error on the broker's transactions.
Shutdown and restart the message broker. If the problem continues to occur, then restart the system. If the problem still continues to occur contact your IBM support center.
I have followed the example with the most minor changes possible. Does anyone have idea why I'm encountering an exception after assigning the values from the db to the Shared Row? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Apr 02, 2014 4:12 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you tried enclosing the assignment of values to the shared row in a
BEGIN ATOMIC -- END ATOMIC block?
You did not display enough of the error text to give us an idea... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| EricCox |
| Posted: Thu Apr 03, 2014 5:07 am Post subject: Give it a Try |
|
|
Master
Joined: 08 Apr 2011
Posts: 292
|
Thanks FJ
Let me try wrapping it in the BEGIN/END ATOMIC and see.
I hope all is well in 2014.
EMC |
|
| Back to top |
|
|
| EricCox |
| Posted: Thu Apr 03, 2014 6:14 am Post subject: BEGIN ATOMIC did not solve it |
|
|
Master
Joined: 08 Apr 2011
Posts: 292
|
Here is the entire text of the error message. You can see the succesfull assignment of the last two rows and then the error.
2014-04-03 10:04:47.364808 5161 UserTrace BIP2574W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '5.48') : Finding next SELECT result.
2014-04-03 10:04:47.364828 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''0134''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-03 10:04:47.364860 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''CHK''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-03 10:04:47.364876 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''Sweep To Line Paydown''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-03 10:04:47.364892 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''Y''' to field / variable ''trancodesSharedRow.Top[483]''.
2014-04-03 10:04:47.364904 5161 UserTrace BIP2574W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '5.48') : Finding next SELECT result.
2014-04-03 10:04:47.364920 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''0135''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-03 10:04:47.364936 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''CHK''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-03 10:04:47.364952 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''POS Debit''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-03 10:04:47.364968 5161 UserTrace BIP2566I: Node 'Test_Flow.Load TRANCODES': Assigning value '''Y''' to field / variable ''trancodesSharedRow.Top[484]''.
2014-04-03 10:04:47.364984 5161 UserTrace BIP2574W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '5.48') : Finding next SELECT result.
2014-04-03 10:04:47.365016 5161 UserTrace BIP2571W: Node 'Test_Flow.Load TRANCODES': ('.myINIT', '5.48') : Failed to find any more items in the FROM clause satisfying the WHERE clause.
2014-04-03 10:04:47.366296 5161 Error BIP2628E: Exception condition detected on input node 'Test_Flow.FRAUD.ACCT.INFO.RQ'.
The input node 'Test_Flow.FRAUD.ACCT.INFO.RQ' detected an error whilst processing a message. The message flow has been rolled-back and, if the message was being processed in a unit of work, it will remain on the input queue to be processed again. Following messages will indicate the cause of this exception.
Check the error messages which follow to determine why the exception was generated, and take action as described by those messages.
2014-04-03 10:04:47.366312 5161 RecoverableException BIP2230E: Error detected whilst processing a message in node 'Test_Flow.Load TRANCODES'.
The message broker detected an error whilst processing a message in node 'Test_Flow.Load TRANCODES'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2014-04-03 10:04:47.366320 5161 RecoverableException BIP2934E: Error detected whilst executing the function or procedure ''myINIT''.
The message broker detected an error whilst executing the function or procedure ''myINIT''. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2014-04-03 10:04:47.366328 5161 RecoverableException BIP2488E: ('.myINIT', '5.10') Error detected whilst executing the SQL statement ''SET trancodesSharedRow.Top[ ] = (SELECT T.TRANCODE AS :TRANCODE, T.ACCTTYPE AS :ACCTTYPE, T.TRANDESC AS :TRANDESC, T.MEMO_DESC AS :MEMO_DESC FROM Database.db.TRANCODES AS T);''.
The message broker detected an error whilst executing the given statement. An exception has been thrown to cut short the SQL program.
See the following messages for details of the error.
2014-04-03 10:04:47.366340 5161 RecoverableException BIP2112E: Message broker internal error: diagnostic information '', '{1}'.
An internal software error has occurred in the message broker. Further messages will indicate the effect of this error on the broker's transactions.
Shutdown and restart the message broker. If the problem continues to occur, then restart the system. If the problem still continues to occur contact your IBM support center.
2014-04-03 10:04:47.366360 5161 UserTrace BIP11507W: Rolled back a local transaction.
A local transaction has been rolled back for work done on the message flow thread.
2014-04-03 10:04:48.368260 5161 UserTrace BIP11501I: Received data from input node ''FRAUD.ACCT.INFO.RQ''.
The input node ''FRAUD.ACCT.INFO.RQ'' has received data and has propagated it to the message flow ''Test_Flow''.
2014-04-03 10:04:48.368672 5161 Error BIP2648E: Message backed out to a queue; node 'Test_Flow.FRAUD.ACCT.INFO.RQ'.
Node 'Test_Flow.FRAUD.ACCT.INFO.RQ' has received a message which has previously been backed out one or more times because of a processing error in the main path of the message flow. The failure terminal is not attached, so the message broker is putting the message directly to the requeue or dead letter backout queue associated with this node. The MQMD 'backoutCount' of the message now equals the 'backoutThreshold' defined for the WebSphere MQ input queue.
Examine previous messages and the message flow to determine why the message is being backed out. Correct this situation if possible. Perform any local error recovery processing required.
2014-04-03 10:04:48.369988 5161 UserTrace BIP2638I: The MQ output node 'Test_Flow.FRAUD.ACCT.INFO.RQ' attempted to write a message to queue ''SYSTEM.DEAD.LETTER.QUEUE'' connected to queue manager ''QMBKRD12''. The MQCC was '0' and the MQRC was '0'.
2014-04-03 10:04:48.370008 5161 UserTrace BIP2615I: The WebSphere MQ input node 'Test_Flow.FRAUD.ACCT.INFO.RQ' has backed out the message to the backout requeue or the dead letter queue.
Message backout processing has been invoked, and the message has either been backed out by being written to the backout requeue or dead letter queue, as determined by the WebSphere MQ queue manager and queue configuration.
No user action required.
2014-04-03 10:04:48.370160 5161 UserTrace BIP11506I: Committed a local transaction.
A local transaction has been committed for work done on the message flow thread.
2014-04-03 10:04:53.370400 5161 UserTrace BIP11504I: Waiting for data from input node ''FRAUD.ACCT.INFO.RQ''.
A thread is waiting for data from input node ''FRAUD.ACCT.INFO.RQ'' in flow ''Test_Flow''.
2014-04-03 10:07:01.898294 4885 UserTrace BIP2632I: Message received and propagated to 'out' terminal of MQ input node '.InputNode'.
2014-04-03 10:07:01.898611 4885 UserTrace BIP6060I: Node '.InputNode' used parser type ''Properties'' to process a portion of the incoming message of length '0' bytes beginning at offset '0'.
2014-04-03 10:07:01.898666 4885 UserTrace BIP6061I: Node '.InputNode' used parser type ''MQMD'' to process a portion of the incoming message of length '364' bytes beginning at offset '0'. The parser type was selected based on value ''MQHMD'' from the previous parser.
2014-04-03 10:07:01.898748 4885 UserTrace BIP6061I: Node '.InputNode' used parser type ''XMLS'' to process a portion of the incoming message of length '347' bytes beginning at offset '364'. The parser type was selected based on value ''XMLS'' from the previous parser.
2014-04-03 10:07:01.899868 4885 Information BIP2152I: Configuration message received from broker.
An execution group received a command from the Broker.
No user action required.
2014-04-03 10:07:01.899960 4885 Information BIP2153I: About to ''Change'' an execution group.
An execution group is about to perform an action.
No user action required. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 03, 2014 6:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Just to dot the i & cross the t, which 8.0.0.n is this? if n !=2, why not?
Smells like a PMR to me. Especially if there's a file in the broker's error directory.......
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| EricCox |
| Posted: Thu Apr 03, 2014 6:27 am Post subject: n = 2 |
|
|
Master
Joined: 08 Apr 2011
Posts: 292
|
Yes, we did get to 8.0.0.2. We couldn't quite get to 8.0.0.3 to start.
Yes, I agree. I'm fiddling a little more and then open a PMR.
Thanks Vitor |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 03, 2014 6:39 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Are you using treat info / warning as error from the DB?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| EricCox |
| Posted: Thu Apr 03, 2014 6:46 am Post subject: |
|
|
Master
Joined: 08 Apr 2011
Posts: 292
|
You said:
| Quote: |
Are you using treat info / warning as error from the DB?
|
I'm not sure. How do I use that or tell?
The odd thing is that at the beginning of the flow it executes the SQL statement and fills the trancodesSharedRow variable, but then when done loading the ROW...at the end seems to make a different call/SQL statement that shows as failing at the end of the trace. I don't have any second statement such as this in the error at the end of the trace.
I am trying to do the call to fill the shared row exactly as it is shown in the documentation and shown above. I just have a different query. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 03, 2014 7:01 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Create an errorhandler and trap the DB Error. If SQL 100 ignore...
I believe there is a checkbox somewhere on the node / properties that says treat warnings/info messages as error...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
