| Author |
Message
|
| pamulapatianil |
 Posted: Thu Jul 18, 2013 2:58 am Post subject: Piling of the messages in the Input Queue - Strange Issue |
 |
|
Newbie
Joined: 17 Jul 2013
Posts: 5
|
Hello Guys,
I am facing a strange issue. please help me in understanding the root cause.
We deployed our flows in Prod broker some 15 days back and until yesteday everything is working fine and all of sudden today the messages are piling up in the input Queue. The flow is not picking up the messages.
[b][u]The below is the error we are seeing from the event log:[/u][/b]
Jul 16 02:41:46 wmqqx2013 WebSphere Broker v7003[7744]: (WMQSQM06_BK.WMQSQM06_PLM_ABC)[5]
BIP2951I: Event generated by user code. Additional information :
'ABC_PLM_Mapping.WMB_ExceptionHandling.ThrowError' '{1}' '{2}' '{3}' '{4}' '{5}' '{6}' '{7}' '{8}' '{9}' :
WMQSQM06_BK.79b0c35f-3b01-0000-0080-b1e0aaf40822: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 229:
SqlThrowExceptionStatement::execute: ComIbmComputeNode: ABC_PLM_Mapping#FCMComposite_1_7.WMB_ExceptionHandling#FCMComposite_1_1
Jul 16 02:41:48 wmqqx2013 WebSphere Broker v7003[7744]: (WMQSQM06_BK.WMQSQM06_PLM_ABC)[5]
BIP2232E:Error detected whilst handling a previous error in node 'ABC_PLM_Mapping.WMB_ExceptionHandling.ThrowError'. :
WMQSQM06_BK.79b0c35f-3b01-0000-0080-b1e0aaf40822: /build/S700_P/src/DataFlowEngine/ImbDataFlowNode.cpp: 885:
ImbDataFlowNode::logExceptionList: ComIbmComputeNode: ABC_PLM_Mapping#FCMComposite_1_7.WMB_ExceptionHandling#FCMComposite_1_1
Jul 16 02:41:48 wmqqx2013 WebSphere Broker v7003[7744]: (WMQSQM06_BK.WMQSQM06_PLM_ABC)[5]
BIP2230E: Error detected whilst processing a message in node 'ABC_PLM_Mapping.UpdateStatus'. :
WMQSQM06_BK.79b0c35f-3b01-0000-0080-b1e0aaf40822: /build/S700_P/src/DataFlowEngine/ImbComputeNode.cpp: 489: ImbComputeNode::evaluate:
ComIbmComputeNode: ABC_PLM_Mapping#FCMComposite_1_5
Jul 16 02:41:48 wmqqx2013 WebSphere Broker v7003[7744]:(WMQSQM06_BK.WMQSQM06_PLM_ABC)[5]
BIP2488E: (.ABC_PLM_Mapping_Compute.Main, 21.4) Error detected whilst executing the SQL statement
'THROW EXCEPTION CATALOG 'Please refer Logs for Error' MESSAGE 10005 VALUES( 'Please refer Logs for Validation Error');'. :
WMQSQM06_BK.79b0c35f-3b01-0000-0080-b1e0aaf40822:/build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlStatementGroup.cpp: 641: SqlStatementGroup::execute: ComIbmComputeNode: ABC_PLM_Mapping#FCMComposite_1_5
Jul 16 02:41:48 wmqqx2013 WebSphere Broker v7003[7744]: (WMQSQM06_BK.WMQSQM06_PLM_ABC)[5]
BIP2176S: Could not find message 10005 in catalogue Please refer Flex Logs for Error.cat,
inserts are Please refer Flex Logs for Validation Error ABC_PLM_Mapping.UpdateStatus {2} {3} {4} {5} {6}:
WMQSQM06_BK.79b0c35f-3b01-0000-0080-b1e0aaf40822: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 229:
SqlThrowExceptionStatement::execute: ComIbmComputeNode: ABC_PLM_Mapping#FCMComposite_1_5
In the flow in one of the compute node we are using this statement to throw an exception. is this the one causing the issue? Then how it was working till now?
THROW USER EXCEPTION CATALOG 'Please refer Logs for Error' MESSAGE 10005 VALUES ('Please refer Logs for Validation Error');
Please help in understanding the root cause of the issue. |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Thu Jul 18, 2013 3:49 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Its only strange if you don't know whats going on.
One of your messages has become a poison message due to a validation error (the payload is incorrect or the code is not written to deal with it).
You need to make use of backout queues and retry counts in order to avoid this pitfall of a poison message.
To correct this for the short term, stop the message flow, empty the queue, and restart the message flow. Inspect the messages one-by-one until you find the offending message. Don't put the offending message back in the queue.
To correct this for the long term, properly handle this condition in your error processing subflow and add one backout queue for each input queue you have, and set the retry count to something like three.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| pamulapatianil |
| Posted: Thu Jul 18, 2013 6:08 am Post subject: |
|
|
Newbie
Joined: 17 Jul 2013
Posts: 5
|
Thanks for the quick reply lancelotlinc.
Actually in our flow we are already handling this.We attached a Exception Handling flow and set a backout Queue for the Input Queue and given backout count 3.
So If some message which are not in-aligned with the Schema is comming it'll try 6 times and kept into backout Queue.
For other exception also it is doing the same thing.
And when this issue happened,
We found out that flow got stopped forcefully by the broker because of this issue and once we stop/start the flow all the messages got drained.
So It seems there is no problem in the message. But I am not sure again why this is happening frequently.[piling of messages] |
|
| Back to top |
|
|
| McueMart |
| Posted: Thu Jul 18, 2013 6:37 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
To me, the way you are using this line of code doesn't look right:
| Code: |
| THROW USER EXCEPTION CATALOG 'Please refer Logs for Error' MESSAGE 10005 VALUES ('Please refer Logs for Validation Error'); |
Look at the correct way you should be throwing USER exceptions as you shouldnt be getting:
"Could not find message 10005 in catalogue Please refer Flex Logs for Error.cat,"
Im not 100% sure this is causing your issue but it might be! |
|
| Back to top |
|
|
| pamulapatianil |
| Posted: Fri Jul 19, 2013 2:44 am Post subject: |
|
|
Newbie
Joined: 17 Jul 2013
Posts: 5
|
Hi McueMart
I checked the syntax for the USER EXCEPTION against the ESQL documentation it looks fine.
Even I was thinking this could be root cause of the issue.
"Could not find message 10005 in catalogue Please refer Flex Logs for Error.cat,"
Is it because of some Environment variables issue? Why it is unable to find the message in catalogue.. if the message exists? |
|
| Back to top |
|
|
| McueMart |
| Posted: Fri Jul 19, 2013 3:08 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
|
| Back to top |
|
|
| pamulapatianil |
| Posted: Fri Jul 19, 2013 5:36 am Post subject: |
|
|
Newbie
Joined: 17 Jul 2013
Posts: 5
|
McueMart,
I tried by removing the CATALOG(i.e using default CATALOG) and giving message number between this(2951 - 2999). Still the same issue.
So it is happening because of some other reason :( :( |
|
| Back to top |
|
|
| dogorsy |
| Posted: Sun Jul 21, 2013 10:37 pm Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| If you removed the CATALOG and changed the message number, you cannot possibly have the same issue. You may have a "similar" one, but the error message would be different, unless, of course, you changed the throw statement and did not re-deploy the flow. |
|
| Back to top |
|
|
| pamulapatianil |
| Posted: Tue Jul 23, 2013 2:06 am Post subject: |
|
|
Newbie
Joined: 17 Jul 2013
Posts: 5
|
Thanks dogorsy.....Sure I'll try to regenerate the bar file and redeploy once and check whether it is the same issue or not..
Please Can anyone explain about this CATALOG? I tried seraching in the documentation but even in the examples also they are using there own CATALOG. If we are doing this should we create anything in the server?
But in my case we didnt create any CATALOGUES in the server then how it worked until now???? :no: |
|
| Back to top |
|
|
| dogorsy |
| Posted: Tue Jul 23, 2013 2:09 am Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
if you did not create a catalog, then let it default. It worked until now because that bit of code was probably never exercised before ( nor tested ! ).
The infocenter explains how to use the THROW statement, how to specify the CATALOG and gives a few good examples. |
|
| Back to top |
|
|
|
|
