|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
getting persistent messages from the queue is very slow |
« View previous topic :: View next topic » |
Author |
Message
|
hughson |
Posted: Thu Feb 17, 2022 2:26 am Post subject: |
|
|
 Padawan
Joined: 09 May 2013 Posts: 1959 Location: Bay of Plenty, New Zealand
|
an4ous wrote: |
Where can I find commit/rollback statistic for a specific queue? |
There aren't commit/rollback statistics on a per queue basis. A transaction could include more than one queue.
You should look at MQI accounting data in order to discover which applications have large numbers of rollbacks.
Cheers,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
 |
an4ous |
Posted: Thu Feb 17, 2022 3:02 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022 Posts: 38
|
hughson wrote: |
an4ous wrote: |
Where can I find commit/rollback statistic for a specific queue? |
There aren't commit/rollback statistics on a per queue basis. A transaction could include more than one queue.
You should look at MQI accounting data in order to discover which applications have large numbers of rollbacks.
Cheers,
Morag |
CommitCount/CommitFailCount from MQIAccounting that's what I need?
or BackCount? |
|
Back to top |
|
 |
hughson |
Posted: Thu Feb 17, 2022 3:45 am Post subject: |
|
|
 Padawan
Joined: 09 May 2013 Posts: 1959 Location: Bay of Plenty, New Zealand
|
an4ous wrote: |
hughson wrote: |
an4ous wrote: |
Where can I find commit/rollback statistic for a specific queue? |
There aren't commit/rollback statistics on a per queue basis. A transaction could include more than one queue.
You should look at MQI accounting data in order to discover which applications have large numbers of rollbacks.
Cheers,
Morag |
CommitCount/CommitFailCount from MQIAccounting that's what I need?
or BackCount? |
I suspect all of those would be interesting, but you were specifically interested in why you have so many rollbacks. _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
 |
markt |
Posted: Thu Feb 17, 2022 4:30 am Post subject: |
|
|
 Knight
Joined: 14 May 2002 Posts: 508
|
Remember that svrconn channels will often issue a speculative MQBACK as they end to ensure that commits don't happen even if the app has abended. So counting calls to that verb are not necessarily useful. (An MQBACK when there's no actual work to really be rolled back is not expensive.) |
|
Back to top |
|
 |
bruce2359 |
Posted: Thu Feb 17, 2022 5:04 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
markt wrote: |
Remember that svrconn channels will often issue a speculative MQBACK as they end to ensure that commits don't happen even if the app has abended. So counting calls to that verb are not necessarily useful. (An MQBACK when there's no actual work to really be rolled back is not expensive.) |
gbaddeley wrote: |
bruce2359 wrote: |
an4ous wrote: |
around 1400 commits and 500 rollbacks per second |
Seems to me to be a high percentage of rollbacks.
I'd suggest an application trace to identify what is driving so many rollbacks. |
There should not be ANY rollbacks. This indicates an issue that needs to be found and fixed. Rollbacks will be impacting overall performance. |
I agree. Both of these statements can be true. There can be badly written applications that issue rollbacks; and svrcon channels can issue rollbacks.
This long-running thread has been a wild ride presuming (as I read it) that some MQ defect is responsible for this performance issue. This, too, can also be true.
We've not yet heard from the app developer to be able to exclude the app as the problem source. Does the app issue rollbakcs? I so, why?
I’ve enabled channel stats for SDR-RCVR channels, but not SVRCONNs.
I recall working with a client whose app put persistent "transaction state messages" to a "transaction state queue" in a multi-legged transaction. From time to time one or another of the legs would fail (bad data, whatever), and the transaction would be rolled back. From time to time, but nowhere near the percentage of rollbacks in the OPs app. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
an4ous |
Posted: Thu Feb 17, 2022 6:25 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022 Posts: 38
|
How can I to distinguish real rollbacks and "speculative MQBACK of svrconn channels"? |
|
Back to top |
|
 |
bruce2359 |
Posted: Thu Feb 17, 2022 8:49 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
bruce2359 wrote: |
We've not yet heard from the app developer to be able to exclude the app as the problem source. Does the app issue rollbakcs? I so, why? |
_________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
bruce2359 |
Posted: Thu Feb 17, 2022 9:19 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
What is splunk's involvement? I found a few hits searching google for splunk+performance. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
an4ous |
Posted: Thu Feb 17, 2022 10:25 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022 Posts: 38
|
splunk is not a bottleneck. Splunk is ready to accept many more requests |
|
Back to top |
|
 |
bruce2359 |
Posted: Thu Feb 17, 2022 10:37 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
an4ous wrote: |
splunk is not a bottleneck. Splunk is ready to accept many more requests |
Are you a Splunk guru? What is your evidence that Splunk is not a bottleneck?
We have yet to hear from the developers of the app. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
an4ous |
Posted: Thu Feb 17, 2022 10:56 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022 Posts: 38
|
I am not splunk guru, but in our company to one splunk server write many other buesness systems from other projects (not via ibm wmq), they don't have the same problems as we do with perfomance.
Also our splunk administrators says that splunk server has resources for more activity.
Our developers remain silent |
|
Back to top |
|
 |
bruce2359 |
Posted: Thu Feb 17, 2022 12:06 pm Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
an4ous wrote: |
Our developers remain silent |
You appear target-locked on finding MQ at fault. It is quite common for developers to lay blame on MQ until MQ folks prove that MQ is not the problem.
Problem source identification demands full participation from all involved: developers, hardware, network, o/s, database, MQ, splunk, … clearly not happening here.
I’d suggest that you push back - point out to management that you (and the users of the poorly performing app) are not being fully supported in this effort.
Welcome to the MQ world. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
hughson |
Posted: Fri Feb 18, 2022 2:22 am Post subject: |
|
|
 Padawan
Joined: 09 May 2013 Posts: 1959 Location: Bay of Plenty, New Zealand
|
an4ous wrote: |
How can I to distinguish real rollbacks and "speculative MQBACK of svrconn channels"? |
I'm not sure you can, but we have yet to hear what your developer has to say about when they might themselves issue an MQBACK. That information will help. For example, they might have chosen to write to an application log whenever they issue an MQBACK, as that would be an unusual path for the application to take.
Another interesting factor that will help is whether the disconnects are explicit, or implicit. An implicit disconnect will cause a rollback. That statistic is also part of the MQI Accounting record you should capture.
Be very interesting to see these records. Keep us posted.
Cheers,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
 |
an4ous |
Posted: Fri Feb 18, 2022 2:40 am Post subject: |
|
|
Apprentice
Joined: 14 Jan 2022 Posts: 38
|
In MQI accounting, I see a lot of similar records
Code: |
...
MonitoringType: MQIAccounting
QueueManager: 'MGR_PROD'
IntervalStartDate: '2022-02-17'
IntervalStartTime: '14.15.54'
IntervalEndDate: '2022-02-17'
IntervalEndTime: '14.15.54'
CommandLevel: 910
ConnectionId: x'414d51435247534d47525f50524f442090970d6272cdf522'
SeqNumber: 0
ApplicationName: 'CamelApp'
ApplicationPid: 31473
ApplicationTid: 2596926
UserId: 'USER_NAME'
ConnDate: '2022-02-17'
ConnTime: '14.15.54'
ConnName: 'CLIENT_IP'
ChannelName: 'SPLUNK.SVRCONN'
RemoteProduct: 'MQJM'
RemoteVersion: '09020300'
DiscDate: '2022-02-17'
DiscTime: '14.15.54'
DiscType: Normal
OpenCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
OpenFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
CloseCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
CloseFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
PutCount: [0, 0]
PutFailCount: 0
Put1Count: [0, 0]
Put1FailCount: 0
PutBytes: [0, 0]
GetCount: [0, 0]
GetFailCount: 0
GetBytes: [0, 0]
BrowseCount: [0, 0]
BrowseFailCount: 0
BrowseBytes: [0, 0]
CommitCount: 2
CommitFailCount: 0
BackCount: 1
InqCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
InqFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
SetFailCount: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DurableSubscribeCount: [0, 0, 0]
NonDurableSubscribeCount: [0, 0, 0]
SubscribeFailCount: 0
DurableUnsubscribeCount: [0, 0]
NonDurableUnsubscribeCount: [0, 0]
UnsubscribeFailCount: 0
SubRqCount: 0
SubRqFailCount: 0
CbCount: [0, 0, 0, 0]
CbFailCount: 0
CtlCount: [0, 0, 0, 0]
CtlFailCount: 0
StatCount: 0
StatFailCount: 0
PutTopicCount: [0, 0]
PutTopicFailCount: 0
Put1TopicCount: [0, 0]
Put1TopicFailCount: 0
PutTopicBytes: [0, 0]
...
|
what should I pay attention to?
For every 2 CommitCount there is 1 BackCount it is normally? |
|
Back to top |
|
 |
hughson |
Posted: Fri Feb 18, 2022 3:11 am Post subject: |
|
|
 Padawan
Joined: 09 May 2013 Posts: 1959 Location: Bay of Plenty, New Zealand
|
This application doesn't appear to have done any work. It disconnected in the same second that it connected and has done no puts or gets.
Are all your application connections like this? Do you have any that actually did some "work"?
Cheers,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
 |
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|
|
|