| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Messages getting slowly processed |
« View previous topic :: View next topic » |
| Author |
Message
|
| chetu777 |
 Posted: Sun Jan 10, 2010 10:02 am Post subject: Messages getting slowly processed |
 |
|
Acolyte
Joined: 07 Sep 2009
Posts: 59
|
Hi All,
My clients are using WMB 5.3 on Unix box.
The problem what I am facing right now is that from past two days the messages are processing slowly. This happens once in 4-5 hours where messages gets slowly processed for around 10-15min (like 2-3 min one message or so ) and all of a sudden the messages gets processed in a fraction of a second without any delay. Again the problem appears after some hours.
The flow has many subflows in it, bascially Insertion into DB operation takes place in these subflow and in one subflow we establish a connection to one of the legacy system and msgs are passed to that.
These flows were working absolutly fine for past 4 years from 2005 and all of a sudden has thrown this problem.
Can anyone let me know what might be the reason and how to go by solving this.
I am waiting for this error to reoccur so that I can apply readlog and formatlog commands. But before that can anyone suggest something. Its a bit urgent for me to solve this problem
Thanks in advance |
|
| Back to top |
|
 |
| chetu777 |
| Posted: Sun Jan 10, 2010 10:10 am Post subject: |
|
|
Acolyte
Joined: 07 Sep 2009
Posts: 59
|
| I have one input queue for the pick operation where huge amount of data comes in at the pick operations and this i/p queue is rooted to the legacy message broker flow which intern has many subflows to divide different kind of messages and do different operations |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Jan 10, 2010 10:30 am Post subject: Re: Messages getting slowly processed |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| chetu777 wrote: |
These flows were working absolutly fine for past 4 years from 2005 and all of a sudden has thrown this problem.
Can anyone let me know what might be the reason and how to go by solving this.
|
Find what in the evironment changed all of a sudden.
As you've already indicated, you'll know more once the problem reoccurs but as a first glance theory it sounds like an external job soaking or locking resources. Certainly if the WMB flows & configuration haven't changed for 4 years it's not that. Unless someone's applied maintenance and not mentioned it to you, which I would consider envirnomental.
I'd start by asking the DBAs when the databases and backed up or reorganised & if that schedule (or method) has changed all of a sudden.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| chetu777 |
| Posted: Mon Jan 11, 2010 12:28 am Post subject: |
|
|
Acolyte
Joined: 07 Sep 2009
Posts: 59
|
Hi Vitor and all,
Thanks for the reply. The issue seems to be resolved now:)
Fristly one thing has taken place on Sunday, the whole server has been shutdown and restarted as part of weekly schedule by admins.
And secondly the problem have not resurfaced again from morning.
So to analyse I went through the messages in exception and failure queue of past two days. What I could find was messages of one particular type were more, these messages were to be inserted into a same table. (I also remember that day I had seen these messages at the top of the queue when slow procesing of messages were taking place).
In the exception message, I have found out the below mentioned error.
[IBM][CLI Driver][DB2/6000] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code "68". SQLSTATE=40001
From this I think I can come to the conclusion that the broker was trying to insert the message into the DB but due to the fact that there was some deadlock condition in accessing the DB table, the messages were rolled back after timeout. Now that server has been restarted the process which was causing the deadlock has been released and working fine now.
I hope my inference is correct. Can anyone kindly let me know if I am wrong.
I could not use Mqsiread and mqsiformat log commands as the errors have not occured again.
Also pasting the exception message for furhtur analysis for anyone.
- <Error>
<TimeOfError>2010-01-09 14:22:25.081094</TimeOfError>
<MessageID>414d5120502e434f4e2e544449532e304b441cd52004c301</MessageID>
<ProcessID>PKWACL</ProcessID>
<RequestID>2623563</RequestID>
- <ExceptionList>
<File>/build/S500_P/src/DataFlowEngine/ImbComputeNode.cpp</File>
<Line>472</Line>
<Function>ImbComputeNode::evaluate</Function>
<Type>ComIbmComputeNode</Type>
<Name>TDMS_TDIS#FCMComposite_1_19.SUBFLOW_TDMS_WAR_CLM_TDIS#FCMComposite_1_3</Name>
<Label>TDMS_TDIS.SUBFLOW_TDMS_WAR_CLM_TDIS</Label>
<Text>Caught exception and rethrowing</Text>
<Catalog>BIPv500</Catalog>
<Severity>3</Severity>
<Number>2230</Number>
- <RecoverableException>
<File>/build/S500_P/src/DataFlowEngine/ImbRdl/ImbRdlStatementGroup.cpp</File>
<Line>158</Line>
<Function>SqlStatementGroup::execute</Function>
<Type>ComIbmComputeNode</Type>
<Name>TDMS_TDIS#FCMComposite_1_19.SUBFLOW_TDMS_WAR_CLM_TDIS#FCMComposite_1_3</Name>
<Label>TDMS_TDIS.SUBFLOW_TDMS_WAR_CLM_TDIS</Label>
<Text>Error detected, rethrowing</Text>
<Catalog>BIPv500</Catalog>
<Severity>3</Severity>
<Number>2488</Number>
- <Insert>
<Type>5</Type>
<Text>.SUBFLOW_TDMS_WAR_CLM_TDIS_Validate_and_Insert_Data_into_Database.Main</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>487.4</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>THROW USER EXCEPTION MESSAGE 2951 VALUES( 'SQL insert error on TWRNTY_CLAIM.', 'SQLState1=', SQLSTATE, 'SQLCode1=', SQLCODE, 'SQLErrorText1=', SQLERRORTEXT, 'SQLNativeError1=', SQLNATIVEERROR);</Text>
</Insert>
- <UserException>
<File>/build/S500_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp</File>
<Line>173</Line>
<Function>SqlThrowExceptionStatement::execute</Function>
<Type>ComIbmComputeNode</Type>
<Name>TDMS_TDIS#FCMComposite_1_19.SUBFLOW_TDMS_WAR_CLM_TDIS#FCMComposite_1_3</Name>
<Label>TDMS_TDIS.SUBFLOW_TDMS_WAR_CLM_TDIS</Label>
<Text>User Generated SQL 'USER' exception</Text>
<Catalog>BIPv500</Catalog>
<Severity>1</Severity>
<Number>2951</Number>
- <Insert>
<Type>5</Type>
<Text>SQL insert error on TWRNTY_CLAIM.</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>SQLState1=</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>40001</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>SQLCode1=</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>-1</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>SQLErrorText1=</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>[IBM][CLI Driver][DB2/6000] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code "68". SQLSTATE=40001</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>SQLNativeError1=</Text>
</Insert>
- <Insert>
<Type>5</Type>
<Text>-911</Text>
</Insert>
</UserException>
</RecoverableException>
</ExceptionList>
</Error> |
|
| Back to top |
|
|
| nathanw |
| Posted: Mon Jan 11, 2010 1:25 am Post subject: |
|
|
Knight
Joined: 14 Jul 2004
Posts: 550
|
I know it sounds like eagle eyed hindsight but as contact admin I read the problem i thought DB access maybe re cacheing or locking
This may well occur again and unless you can find out what is causing the deadlocking there may be problems again.
Check the logs on the db for any more details and also check here
http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.messages.sql.doc/doc/msql00911n.html
_________________
Who is General Failure and why is he reading my hard drive?
Artificial Intelligence stands no chance against Natural Stupidity.
Only the User Trace Speaks The Truth  |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jan 11, 2010 6:05 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| chetu777 wrote: |
I hope my inference is correct. Can anyone kindly let me know if I am wrong.
|
If there wasn't a backup job, deadlock would have been my next guess. You still need to speak to your DBAs about this.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
