| Author |
Message
|
| Raj kumar |
 Posted: Fri Jun 27, 2008 3:21 am Post subject: Problem in storing message in Environment Variables |
 |
|
Novice
Joined: 27 Jun 2008
Posts: 14
Location: India
|
Hello Friends,
Can anyone please help me in this issue or give any suitable solution for the below scenario?
I have a single broker and a single execution group on WMB V6.0.0.1.
My requirement is to pick all the messages from a queue and create a batch message and do a sort on the batched message and give it as a single message to output queue.
I tried to do the same, by using MQget node looped to a compute node and store all individual messages to Environment.Variable Array. From there I perform a sort and get a single message by copying it to Output Root to get a single batched message.
The problem I am facing is... This logic works fine to batch up to 120 messages, when the count increases beyond that, it throws a health indicator exception and memory exception.
My expectation is to store about 1K messages in the variable array.
Can anyone please help me, how to increase this heap memory for execution group?
Below or some logs for your investigation.
( WBRK6_DEFAULT_BROKER ) The broker has detected that the Execution Group 'default', process ID 492, has shutdown.
An Execution Group has shutdown. A new instance of the execution group will be started in approximately 0 seconds.
Investigate any prior system log messages for possible problems. Contact your IBM support center if you are unable to determine the cause of the shutdown.
2008-06-25-16.09.53.569000 Instance:DB2 Node:000
PID:1580(db2fmp.exe) TID:1524 Appid:none
Health Monitor HealthIndicator::update Probe:500
ADM10501W Health indicator "Monitor Heap Utilization" ("db2.mon_heap_util")
breached the "upper" warning threshold of "85 %" with value "88 %" on
"instance" "DB2". Calculation:
"((db2.mon_heap_cur_size/db2.mon_heap_max_size)*100);" = "((245760 / 278528) *
100)" = "88 %". History (Timestamp, Value, Formula): " (06/25/2008
16:04:36.123000, 59, ((163840 / 278528) * 100)), (06/25/2008 15:59:35.843000,
59, ((163840 / 278528) * 100)), (06/25/2008 15:54:35.172000, 29, ((81920 /
278528) * 100)), (06/25/2008 15:49:35.908000, 29, ((81920 / 278528) * 100)),
(06/25/2008 15:44:35.643000, 29, ((81920 / 278528) * 100)), (06/25/2008
15:39:35.378000, 29, ((81920 / 278528) * 100)), (06/25/2008 15:34:36.161000, 29,
((81920 / 278528) * 100)), (06/25/2008 15:29:35.943000, 29, ((81920 / 278528) *
100)), (06/25/2008 15:24:35.679000, 29, ((81920 / 278528) * 100))"
2008-06-25-16.05.17.553000 Instance:DB2 Node:000
PID:1580(db2fmp.exe) TID:4108 Appid:none
Health Monitor HealthIndicator::update Probe:500
ADM10502W Health indicator "Database Backup Required" ("db.db_backup_req") is
in state "1" on "database" "DB2.DEFBKDB6".
2008-06-25-16.04.36.295000 Instance:DB2 Node:000
PID:1580(db2fmp.exe) TID:1524 Appid:none
Health Monitor HealthIndicator::update Probe:500
ADM10500E Health indicator "Log Filesystem Utilization" ("db.log_fs_util")
breached the "upper" alarm threshold of "85 %" with value "88 %" on "database"
"DB2.DEFBKDB6". Calculation: "((os.fs_used/os.fs_total)*100);" =
"((27696951296 / 31461662720) * 100)" = "88 %". History (Timestamp, Value,
Formula): "()"
2008-06-25-16.04.36.279002 Instance:DB2 Node:000
PID:1580(db2fmp.exe) TID:1524 Appid:none
Health Monitor HealthIndicator::update Probe:500
Thanks in advance!!
Raj. |
|
| Back to top |
|
 |
| francoisvdm |
| Posted: Fri Jun 27, 2008 4:41 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
Do not put the MQGet node in a loop, you usually get a stack overflow error after a while. Rather use a compute node in front of MQGet to control the "propagation" to the MQGet node. I usually put TRUE in environment, then change it to FALSE when MQGet follow "No message" terminal and then first compute node knowwhen to stops.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| Raj kumar |
| Posted: Sat Jun 28, 2008 1:43 am Post subject: |
|
|
Novice
Joined: 27 Jun 2008
Posts: 14
Location: India
|
| Thanks for your idea will try this and see whether this help to store 1K messages. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Jun 28, 2008 2:06 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
If you try to output more than say a certain number of messages in a single unit of work, you could get an execution group abend due to memory allocation problems. In my experience 500 messages work fine.
I would not store the messages in the environment anyway. I'd put them into a database Then you can retrieve them nicely.
I described how I used a DB for this purpose a few months ago in this forum.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon May 11, 2009 7:51 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I've just had the stack overflow issue with MQGET node in a loop. (WMB 6.1 FP 03, Win XP)
I only processed about 200 messages (sending them to FileOutput, not storing in environment variable).
Surely this is not acceptable? What exactly is wrong with looping MQGET in a message flow?
Is it possible to control the commit interval and choose when to commit during a message flow loop without ending the message flow? |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Mon May 11, 2009 9:12 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
See my reply earlier in this thread... it works great.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| zpat |
| Posted: Mon May 11, 2009 9:17 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I think IBM should make their product operate under reasonable expectations in the most logical way.
As it happens I want to limit the number of messages per file so it may not matter that much, but I don't like the idea that a severe error occurs when doing something so basic as looping in a message flow a few hundred times!
IBM market WMB as a visual wiring programming tool so I am doing nothing other than using it normally.
Is there any other way to stop the stack overflow condition occuring?
Is the use of PROPAGATE enough in itself - I thought ending a compute node with RETURN TRUE was the same as PROPAGATE and RETURN FALSE? But is there a difference here? |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Mon May 11, 2009 10:12 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
The way I understand it is that the node "never" ends as long as you sit in the loop, so "something" needs to keep track of which nodes are active. That is why its runs out of stack space....any application in the world will do that eventually.
When you use propagate you force that "line" of execution to end and the stack tracking does not overflow.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| zpat |
| Posted: Mon May 11, 2009 10:20 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| So I can just change RETURN TRUE to PROPAGATE and RETURN FALSE then in my compute node which is in this loop? |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Mon May 11, 2009 10:48 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
The way I have used it is as follows:
Controlling compute node with the following code:
| Code: |
SET Environment.TurnaroundCount = 1;
WHILE Environment.TurnaroundCount > 0 AND Environment.TurnaroundCount <= 50 DO
SET Environment.TurnaroundCount = Environment.TurnaroundCount + 1;
PROPAGATE TO TERMINAL 'out';
END WHILE;
RETURN FALSE; |
Following this node is the MQGet node.
After the MQGet node when it is successful is your normal execution path...just, not in a loop now.
Following the "no message" or "error" terminals I have a compute node with the following code:
| Code: |
SET Environment.TurnaroundCount = 0; -- stop the propagate loop
RETURN TRUE; |
Hope this helps
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| zpat |
| Posted: Mon May 11, 2009 11:56 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Can someone clarify this?
When PROPAGATE is issued - the issuing compute node processing is suspended until the message flow logic started by the propagate terminates?
Then the compute node resumes processing and it can issue another propagate or terminate itself as needed?
In other words - is this a single threaded process?
If not it would be hard to control the looping as I need to process the message data from the propagated MQGET in the original compute node. |
|
| Back to top |
|
|
| mgk |
| Posted: Tue May 12, 2009 12:35 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| When PROPAGATE is issued - the issuing compute node processing is suspended until the message flow logic started by the propagate terminates? |
Yes - think of it like a fuction call to the next node
| Quote: |
| Then the compute node resumes processing and it can issue another propagate or terminate itself as needed? |
Yes
| Quote: |
| In other words - is this a single threaded process? |
Yes If you want extra threads against a flow, use additional instances. Each instance processes a message through the entire flow.
Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue May 12, 2009 12:39 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
|
| Back to top |
|
|
|
|
