ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » System.Broker.Timeout.queue messages processing slow in IIB

Post new topic  Reply to topic
 System.Broker.Timeout.queue messages processing slow in IIB « View previous topic :: View next topic » 
Author Message
srina_a
PostPosted: Thu Sep 07, 2017 12:05 pm    Post subject: System.Broker.Timeout.queue messages processing slow in IIB Reply with quote

Newbie

Joined: 07 Sep 2017
Posts: 5

Hi Team,

We migrated recently from V7 to IIB9 , we are experiencing slowness processing messages from SYSTEM.BROKER.TIMEOUT.Queue, It's taking 4 hours reprocess 4000 messages . We Opened a PMR for that.
Mean time we want to take a back up of System.broker.timeout.queue and reprocess using dmpmqmsg utility can we do that? my understanding SBTQ messages has timeout control properties in the header that will strip out when you take back up, Is that true? Can any one suggest on this?


Thank you,
Srina.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Sep 07, 2017 12:23 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

I've no idea if dmpmqmsg strips header properties or not, or if the behavior is configurable. I imagine IBM wrote some documentation for dspmqmsg that's someplace on their web site.

If you're worried, use something else (like RFHUtil) which preserves such things.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
srina_a
PostPosted: Thu Sep 07, 2017 1:11 pm    Post subject: Reply with quote

Newbie

Joined: 07 Sep 2017
Posts: 5

Thank you, Vitor. I never reprocessed messages from System.broker.timeout.queue , we want make sure taking a backup using RFH util will not be any issues, i mean no header properties will be changed?
Parallelly, i will give a try in lower environment.

Thank you,

Regards
Srina.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Sep 07, 2017 11:44 pm    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

Try dmpmqmsg, unload to file, reload to queue, look at the messages - are they 100% identical?

I would trust dmpmqmsg (since it is based on qload) more than rfhutil to restore the header information.

But it is a matter of minutes to try it yourself.

Other things to consider, use configurable services to have separate queues instead of all flows sharing the system timeout queue.

Set an expiry if you don't want to process old ones (e.g. use the capped expiry feature in MQ).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
srina_a
PostPosted: Fri Sep 08, 2017 8:52 am    Post subject: Reply with quote

Newbie

Joined: 07 Sep 2017
Posts: 5

Thank you Zpat for suggestions, using configurable services to have separate queues this requires a code change, we planned for next release. We don't want to set an expiry of this messages bcs this is critical for our business, we will try to change the logic instead.
We had a production issue yesterday, one of the service back end failed those time out messages landed in S.B.T.Queue and processing very slow, Due business criticality we decided to take back up of those messages from System.Broker.Timeout.Queue using dmpmqmsg utility , but we are not able to reprocess those messages, i guess time out notification properties in the message's header changed, that's why those messages not able reprocess from the queue( We Opend a PMR for this) and we observe S.B.T.Q processing the messages very slow even after back service available. Earlier in v7 we don't see this problem no code changes from v7 to IIB9. In v7 we used to have 70 k messages in S.B.T.Q never had issue when ever back end service available it processed in no time but, in IIBv9, Broker took 4 hours to reprocess 4000 messages in S.B.T.Q.
My question here is, Is it safe to take back up of S.B.T.Q messages and reprocess ?or we have to wait till it processed by the broker. Will the broker have any impact if messages stuck in S.B.T.Q? If any one have encountered the same situation, have recommendations please share.

Thank you,
Srina.
Back to top
View user's profile Send private message
srina_a
PostPosted: Thu Sep 14, 2017 7:23 am    Post subject: This post can move to your challenge fourm Reply with quote

Newbie

Joined: 07 Sep 2017
Posts: 5

We had a major issue again 15k business critical messages struck in System.Broker.Timeout .queue we opend a Sev1 PMR with IBM , IBM has no utility to back up those messages to reprocess, almost 12 hrs IIBv9 broker took to process this messages. observation from PMR is Timeout notification node is single threaded where as the main flow running in 100 threads. Apart from changing code we would like see this fourm can help in defining a tool to help us move messages off the timeout queue in the event that the problem happens again so that they can be re-entered into the flow manually.

Couple of members from this group suggested to use RFH Util to reprocess the messages from S.B.T.Q , I would like to confirm in this forum there is no utility as such to reprocess from S.B.T.Q.

Thank you,
Srina.
Back to top
View user's profile Send private message
rekarm01
PostPosted: Fri Sep 15, 2017 10:01 am    Post subject: Re: SYSTEM.BROKER.TIMEOUT.QUEUE messages processing slow Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1415

srina_a wrote:
We migrated recently from V7 to IIB9 , we are experiencing slowness processing messages from SYSTEM.BROKER.TIMEOUT.QUEUE ...

What fixpacks have been applied to IIBv9? There are fixes that address performance related issues for Timeout nodes, such as IC94086.

srina_a wrote:
My question here is, Is it safe to take back up of S.B.T.Q messages and reprocess?

If you have a Sev 1 PMR open with IBM, you can get an official answer to this question.

But one unofficial answer, is NO, it is not safe to manually add or delete messages from any broker timeout queue, especially if the broker is busy scanning messages on the queue at the same time, trying to store information in local memory about what messages to retrieve later, and when.

Even if the broker is shut down, it might be ok to manually delete messages from the queue, but probably still tricky to manually put them back. It would be better to re-run the upstream transactions to recreate the deleted timeout request messages instead.
Back to top
View user's profile Send private message
srina_a
PostPosted: Mon Sep 18, 2017 9:08 am    Post subject: Reply with quote

Newbie

Joined: 07 Sep 2017
Posts: 5

Thank you for updates rekarm01, We are at IIB Version: 9007.

IBM PMR team provided utility to reprocess S.B.T.Q messages , We are testing this in our lower environment , keep you update.

Thank You,

Srina.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » System.Broker.Timeout.queue messages processing slow in IIB
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.