| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Execution group crashes-messages found in Dead letter queue |
« View previous topic :: View next topic » |
| Author |
Message
|
| doll |
 Posted: Sat May 07, 2011 9:36 pm Post subject: Execution group crashes-messages found in Dead letter queue |
 |
|
Novice
Joined: 04 Dec 2010
Posts: 19
|
Hi All,
An execution group crashes in production(results in abend file) and restarts at a particular time eventhough no data is placed in database at that time .
Due to this messages are found in DLQ with header reason 65536.
In that execution group,there are two flows.
First flow:
Timer node(starts every 5 min)->compute->MQOutput(queue name is Input queue of second flow)
Second flow:
MQInput->compute (database opertion is performed)->MQOutput
When i checked the log,the failure location is input node of the second flow.I checked the abend file and found the below info:
abend
| Code: |
----------------------------- Stack dump for current thread ( 5912)
(0xd69e5e88+0x000007a0) ttcdrv [/pa/apps/oracle/product/9.2.0/lib32/libclntsh.a(shr.o)]
(0xd68e447c+0x00000050) nioqwa [/pa/apps/oracle/product/9.2.0/lib32/libclntsh.a(shr.o)]
(0xd65cad5c+0x00000640) upirtrc [/pa/apps/oracle/product/9.2.0/lib32/libclntsh.a(shr.o)]
(0xd671b528+0x0000006c) kpurcsc [/pa/apps/oracle/product/9.2.0/lib32/libclntsh.a(shr.o)]
(0xd6736098+0x0000053c) kpuexecv8 [/pa/apps/oracle/product/9.2.0/lib32/libclntsh.a(shr.o)]
(0xd67371bc+0x00000ff8) kpuexec [/pa/apps/oracle/product/9.2.0/lib32/libclntsh.a(shr.o)]
(0xd67638b8+0x0000001c) OCIStmtExecute [/pa/apps/oracle/product/9.2.0/lib32/libclntsh.a(shr.o)]
(0xd397d714+0x00000000) <no name available> [/opt/IBM/mqsi/6.0/merant/lib/UKor820.so]
(0xd390e088+0x00000000) <no name available> [/opt/IBM/mqsi/6.0/merant/lib/UKor820.so]
(0xd390cd10+0x00000000) <no name available> [/opt/IBM/mqsi/6.0/merant/lib/UKor820.so]
(0xd3948988+0x00000000) <no name available> [/opt/IBM/mqsi/6.0/merant/lib/UKor820.so]
(0xd2e0c710+0x00000000) <no name available> [/opt/IBM/mqsi/6.0/lib/libodbc.a(odbc.so)]
(0xd2dc8530+0x00000000) <no name available> [/opt/IBM/mqsi/6.0/lib/libodbc.a(odbc.so)]
(0xd459d60c+0x0000010c) execute__16ImbOdbcStatementFv [/opt/IBM/mqsi/6.0/lib/libMessageServices.a(libMessageServices.a.so)]
(0xd7fd2620+0x00000750) executeStatement__17SqlExternalDbStmtCFR14SqlEvalEnvironP17Im
bOdbcConnectionRC10ImbWstringQ2_3std6_PtritXTP17Sql
ExpressionNodeTlTPCP17SqlExpressionNodeTRCP17SqlExpression
NodeTPP17SqlExpressionNodeTRP17SqlExpressionNode_T4RC18
SqlGeneralLocationi [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd7fd0268+0x00001e5c) executeQuery__21SqlDatabaseSelectBaseCFR14SqlEvalEnviron [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd7fe5490+0x0000006c) evaluate__26SqlDatabaseAggregateSelectCFR14SqlEvalEnvironR8
ImbValue [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd81bf274+0x00001e28) execute__19SqlDeclareStatementCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd81747fc+0x000000d8) execute__17SqlStatementGroupCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd82065e4+0x00000210) execute__15SqlCompoundStmtCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd80907c0+0x000001c8) execute__10SqlRoutineCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd824abcc+0x00000144) execute__9SqlModuleCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd82428bc+0x00000024) execute__9SqlSchemaCFR18SqlStatementResult_148_23 [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd82bbbc4+0x0000054c) evaluate__19SqlComputeInterfaceFRC18ImbMessageAssembly
R18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd966eb20+0x00000388) evaluate__14ImbComputeNodeFRC18ImbMessageAssemblyPC
19ImbDataFlowTerminal [/opt/IBM/mqsi/6.0/lil/imbdfsql.lil]
(0xd4737170+0x000001ec) evaluate__19ImbDataFlowTerminalFRC18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lib/libMessageServices.a(libMessageServices.a.so)]
(0xd47369cc+0x000000cc) propagateInner__19ImbDataFlowTerminalFRC18Imb
MessageAssemblyP19ImbDataFlowTerminal [/opt/IBM/mqsi/6.0/lib/libMessageServices.a(libMessageServices.a.so)]
(0xd49937e4+0x000004a4) propagate__19ImbDataFlowTerminalFRC18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lib/libMessageServices.a(libMessageServices.a.so)]
(0xd96a4ce4+0x00000284) propagateToTerminal__14ImbComputeNodeFRC8Imb
ValueiRC18ImbMessageAssemblyRC18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lil/imbdfsql.lil]
(0xd82bb988+0x0000006c) propagateToTerminal__19SqlComputeInterfaceCFRC8Imb
ValueiRC18ImbMessageAssemblyR18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd823276c+0x00000db0) execute__21SqlPropagateStatementCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd81747fc+0x000000d8) execute__17SqlStatementGroupCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd82065e4+0x00000210) execute__15SqlCompoundStmtCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd8086dbc+0x000008a4) invoke__10SqlRoutineCFR18SqlStatementResultRC15
auto_ptr_vectorXT17SqlExpressionNode_RCQ2_3std8auto_ptrXT17Sql
ExpressionNode_RCQ2_3std8auto_ptrXT17SqlExpressionNode_b [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd80869f0+0x000002c4) evaluate__25SqlCallableFunctionFnCallCFR9SqlResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd7ee8338+0x00000058) evaluate__17SqlExpressionNodeCFR14SqlEvalEnvironR8ImbValue [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd8234ce0+0x00000dfc) execute__13SqlAssignmentCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd81747fc+0x000000d8) execute__17SqlStatementGroupCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd82065e4+0x00000210) execute__15SqlCompoundStmtCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd80907c0+0x000001c8) execute__10SqlRoutineCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd824abcc+0x00000144) execute__9SqlModuleCFR18SqlStatementResult [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd82428bc+0x00000024) execute__9SqlSchemaCFR18SqlStatementResult_148_23 [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd82bbbc4+0x0000054c) evaluate__19SqlComputeInterfaceFRC18ImbMessageAssembly
R18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lib/libImbRdl.a(libImbRdl.a.so)]
(0xd966eb20+0x00000388) evaluate__14ImbComputeNodeFRC18ImbMessageAssemblyPC19Imb
DataFlowTerminal [/opt/IBM/mqsi/6.0/lil/imbdfsql.lil]
(0xd4737170+0x000001ec) evaluate__19ImbDataFlowTerminalFRC18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lib/libMessageServices.a(libMessageServices.a.so)]
(0xd47369cc+0x000000cc) propagateInner__19ImbDataFlowTerminalFRC18
ImbMessageAssemblyP19ImbDataFlowTerminal [/opt/IBM/mqsi/6.0/lib/libMessageServices.a(libMessageServices.a.so)]
(0xd49937e4+0x000004a4) propagate__19ImbDataFlowTerminalFRC18ImbMessageAssembly [/opt/IBM/mqsi/6.0/lib/libMessageServices.a(libMessageServices.a.so)]
(0xd7cba720+0x000027cc) run__18ImbCommonInputNodeFP11ImbOsThread [/opt/IBM/mqsi/6.0/lib/libMQLibrary.a(libMQLibrary.a.so)]
(0xd7cc702c+0x00000048) run__Q2_18ImbCommonInputNode10ParametersFP11ImbOsThread [/opt/IBM/mqsi/6.0/lib/libMQLibrary.a(libMQLibrary.a.so)]
(0xd33966d8+0x00000070) run__27ImbThreadPoolThreadFunctionFP11ImbOsThread [/opt/IBM/mqsi/6.0/lib/libCommonServices.a(libCommonServices.a.so)]
(0xd338fb70+0x0000005c) threadRun__11ImbOsThreadFv [/opt/IBM/mqsi/6.0/lib/libCommonServices.a(libCommonServices.a.so)]
(0xd3362d38+0x000000a4) threadBootStrap__11ImbOsThreadFPv [/opt/IBM/mqsi/6.0/lib/libCommonServices.a(libCommonServices.a.so)]
(0xd010d668+0x0000011c) _pthread_body [/usr/lib/libpthreads.a(shr_xpg5.o)]
(0x00000000) <invalid code address>
---------------------------------------------------------------------- |
Can someone please provide me your view on this.
Thanks in Advance!
Last edited by doll on Sun May 08, 2011 10:04 pm; edited 2 times in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Sat May 07, 2011 10:09 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Please enclose your stack trace with [ code ] tags.
Also make sure the lines do not get too wide...
Looks like your thread is blowing up attempting the DB operation... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun May 08, 2011 1:00 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
To add to what FJB has said,
You seem to be using WMB 6.0. This is now out of support so you won't get much help from IBM if you raise a PMR.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| doll |
| Posted: Sun May 08, 2011 10:03 pm Post subject: Execution group crashes-messages found in Dead letter queue |
|
|
Novice
Joined: 04 Dec 2010
Posts: 19
|
Yes I agree,IBM support is not provided for WMB 6.0 and I will make sure the lines do not get too wide.
As for the thread blowing up attempting db operation confuses me a little.
Since at that time no database operation is performed as no data is placed to process as per the business team.
Database operation is done in the day time but no error message found at that time.Through that database operation in compute node, we are calling a db function to fetch data from one table and archive in another table.
I will update if I get any further details on this.
Thanks for the support!
Last edited by doll on Sun May 22, 2011 10:50 pm; edited 1 time in total |
|
| Back to top |
|
|
| doll |
| Posted: Tue May 10, 2011 11:06 pm Post subject: |
|
|
Novice
Joined: 04 Dec 2010
Posts: 19
|
Hi,
I checked the execution group log file and found error on memory space. Error detail such as "Allocation failure.need 528 bytes".The error may be due to this so I will check increasing the jvm heap size.
I have got the error details from system log.Here is the info.
| Code: |
Apr 22 13:08:29 brk user:err|error WebSphere Broker v60010[1388796]: (brk_broker.28834126-1401-0000-0080-dd1f0e1e0b28)[5912]BIP2228E: Severe error:
/build/S600_P/src/CommonServices/Unix/ImbAbend.cpp 923 signal received Abend file: /var/mqsi/common/errors/brk_broker.28834126-1401-0000-0080-dd1f0e1e0
b28.1388796.5912.abend action: abort
Apr 22 13:08:34 brk user:warn|warning WebSphere Broker v60010[1343634]: (brk_broker)[2571]BIP2060W: The broker has detected that the Execution Grou
p MC_1_Publish, process ID 1388796, has shutdown. : brk_broker.agent: /build/S600_P/src/AdminAgent/ImbAdminAgent.cpp: 4263: ImbAdminAgent::startAndMoni
torADataFlowEngine: :
Apr 22 13:08:34 brk user:info WebSphere Broker v60010[1388798]: (brk_broker.MC_1_Publish)[1]BIP2201I: Execution Group started: process '1388798';
thread '1'; additional information: brokerName 'brk_broker'; executionGroupUUID '28834126-1401-0000-0080-dd1f0e1e0b28'; executionGroupLabel 'MC_1_Publi
sh'; defaultExecutionGroup 'false'; queueManagerName 'brk'; trusted 'false'; dataSourceName 'DSN'; userId 'brk1'; migrationNeeded 'false'; brokerUU
ID 'b4a9987a-0e01-0000-0080-a8674a587326'; filePath '/opt/IBM/mqsi/6.0'; workPath '/var/mqsi'; ICU Converter Path ''. : brk_broker.28834126-1401-0000-00
80-dd1f0e1e0b28: /build/S600_P/src/DataFlowEngine/ImbMain.cpp: 354: main: :
Apr 22 13:08:41 brk user:info WebSphere Broker v60010[1388798]: (brk_broker.MC_1_Publish)[1]BIP7407I: Publish/Subscribe subscriptions recovery sta
rted. Found '185' subscriptions. : brk_broker.28834126-1401-0000-0080-dd1f0e1e0b28: /build/S600_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 820
: SubscriptionService.readSubscriptions: :
Apr 22 13:08:42 brk user:info WebSphere Broker v60010[1388798]: (brk_broker.MC_1_Publish)[1]BIP7410I: Publish/Subscribe subscriptions recovery. Re
covered '184' subscriptions. : brk_broker.28834126-1401-0000-0080-dd1f0e1e0b28: /build/S600_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 820: Su
bscriptionService.readSubscriptions: :
Apr 22 13:08:42 brk user:info WebSphere Broker v60010[1388798]: (brk_broker.MC_1_Publish)[1]BIP7033I: Publish/Subscribe state recovery completed.
: brk_broker.28834126-1401-0000-0080-dd1f0e1e0b28: /build/S600_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 695: SubscriptionService.readSubscri
ptions: :
Apr 22 13:08:47 brk user:err|error WebSphere Broker v60010[1388798]: (brk_broker.MC_1_Publish)[5912]BIP2648E: Message backed out to a queue; node
'MC_1_Delivery.MC.TRIGGERMSG.DEL_1.F1.BROKER.BROKER'. : brk_broker.28834126-1401-0000-0080-dd1f0e1e0b28: /build/S600_P/src/DataFlowEngine/ImbMqInputNo
de.cpp: 1953: ImbCommonInputNode::eligibleForBackout: ComIbmMQInputNode: MC_1_Delivery#FCMComposite_1_1 |
If anyone faced similar issue,please share your ideas.
Thank you |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
