| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| JMS selector - performance issue on deep queues |
« View previous topic :: View next topic » |
| Author |
Message
|
| bef1 |
 Posted: Thu Aug 06, 2020 4:59 am Post subject: JMS selector - performance issue on deep queues |
 |
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
- Our MQ server is running on version 7.5.0.7 (yeah, I know...)
- The JMS client being used by the application causing the issue is using an MQ v9 client.
A new JMS client application was running a few days ago.
The MQ number of requests sent by this new application to our MQ server was not very high, in the range of a few dozen of requests every minute.
However, while the new application was running, it was experiencing slow response time to get messages from our MQ server (around 1,500 ms per message).
Also, while the new application was running, the amqzlaa0 processes on our MQ server seemed to reach some sort of bottleneck, as it was reaching a plateau at 42% CPU usage. There was still some CPU available on the MQ server, but the processes were not consuming it. There was no bottleneck on the I/O side, the I/O subsystem was actually performing optimally and the number of IOPS on the disks was reasonable.
-----
The response queue that was being read by the client application had a queue depth of 800 messages.
At one point, we cleared this same response queue, and we saw that the response time was then much better (just a few ms, instead of 1,500 ms per message before it was cleared).
We were able to reproduce this isuse in our test environement, and the response time scaled as follows:
- Queue depth of 1 message: A few ms
- Queue depth of 800 messages: 1,500 ms
- Queue depth of 4,000 messages: 15,000 ms
-----
The MQ development guides, for both MQ v7.5 ans MQ v9, provide the following guidelines:
| Quote: |
Messaging performance
Selecting messages from a queue requires IBM WebSphere MQ to sequentially inspect each message on the queue. Messages are inspected until a message is found that matches the selection criteria or there are no more messages to examine. Therefore, messaging performance suffers if message selection is used on deep queues.
To optimize message selection on deep queues when selection is based on JMSCorrelationID or JMSMessageID, use a selection string of the form JMSCorrelationID = ... or JMSMessageID = ... and reference only one property.
This method offers a significant improvement in performance for selection on JMSCorrelationID and offers a marginal performance improvement for JMSMessageID.
|
We traced the requests being sent by the client application (using tcpdump and Wireshark), and we see that is seems to behave accordingly, sending the following selector when doing the OPEN operation on the response queue: JMSCorrelationID='45030c70-b06c-49bd-9d01-d97243b6bd88'
So as far as we can tell, our application complies to IBM's recommendations, but it still scales very badly with the queue depth.
-----
I would like to know if there is something else we should look into to get a better performance on (somewhat) deep queues. |
|
| Back to top |
|
 |
| hughson |
| Posted: Thu Aug 06, 2020 5:54 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1967
Location: Bay of Plenty, New Zealand
|
Message Selection on message properties on a deep queue can indeed poorly perform, as you have read.
If all you need to do is select by MessageID or CorrelationID however, that should not need to use MQOPEN SelectionString, but instead it should use MQGET selection looking at the fields in the MQMD on input.
The quote you have found in Knowledge Center showing the specific way to provide a selection for one of these two fields is supposed to cause the underlying MQ API calls to switch from using MQOPEN SelectionString, to using MQGET MessageID or CorrelationID selection.
When you traced the API calls, you said that you saw the selector on MQOPEN. That suggests that it hasn't caused this switch.
The quote you found doesn't mention the need to start the value with "ID:" and then the hex string of the ID. There is/was a Tech Not explaining this, but in the IBM website reshuffle it does not appear to exist any more, so I am struggling to find you a reference.
There is some example code in an answer to this SO question which might help.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 06, 2020 6:04 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9486
Location: US: west coast, almost. Otherwise, enroute.
|
Does the app disconnect/reconnect for each message?
Id be interested in seeing results of an API trace.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Aug 06, 2020 11:25 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7723
|
Consider a queue with 100 messages with a CorrelID of various names, and message #100 has a Correl ID of PETER.
By doing a selective JMS GET with a CorrelID of PETER, your JMS app will issue 100 MQGETs with Browse, and then one more MQGET under cursor to finally destructively pull the message off of the queue.
If the JMS GET is for Correl ID "ID:(HexValueForPETER)", then you will see only one MQGET occurring, and it is a destructive MQGET right off of bat.
101 MQGETs versus 1 MQGET. Obviously, the deeper the queue, the more important this is, and if the queue is close to zero as it should be, the less important it is.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bef1 |
| Posted: Thu Aug 06, 2020 11:47 am Post subject: |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
| hughson wrote: |
If all you need to do is select by MessageID or CorrelationID however, that should not need to use MQOPEN SelectionString, but instead it should use MQGET selection looking at the fields in the MQMD on input.
|
OK that makes sense, MQ would then only need to parse the headers instead of inspecting the messages payload. Also, from my understanding, the MQMH correlation ID is indexed on Distributed MQ, so it would obviously perform much better if it was being used as a selector.
One issue I have is that the client application is using Spring JMS frameworks, so they currently do not mess with MQMH headers. They stick to the JMS layer.
We might have to look if they are able to set MQMH headers instead of JMSCorrelationID as a selector, but I'm not sure if that is possible using the same framworks that are currently being used.
| hughson wrote: |
The quote you have found in Knowledge Center showing the specific way to provide a selection for one of these two fields is supposed to cause the underlying MQ API calls to switch from using MQOPEN SelectionString, to using MQGET MessageID or CorrelationID selection.
When you traced the API calls, you said that you saw the selector on MQOPEN. That suggests that it hasn't caused this switch.
|
That's being done on the client side, right? So the mq client JAR would be expected to figure out that it must set the MQMH correlation ID, based on the fact that the JMSCorrelationID was set, correct?
I was hoping upgrading MQ to a more modern version would fix this issue, but in that case, I understand it wouldn't, since the behaviour is controlled by the client side, and the MQ client JAR is already up to date.
| hughson wrote: |
The quote you found doesn't mention the need to start the value with "ID:" and then the hex string of the ID. There is/was a Tech Not explaining this, but in the IBM website reshuffle it does not appear to exist any more, so I am struggling to find you a reference.
There is some example code in an answer to this SO question which might help.
|
Maybe there might be some sort of formatting issue being done by our client application which would format the JMSCorrelationID in an improper way, which would not allow the MQ client to switch to using the MQMH correlation ID...? I was inspecting the headers using a tcpdump and Wireshark, so the JMSCorrelationID='45030c70-b06c-49bd-9d01-d97243b6bd88' format is how the MQ client sends the request "on the wire".
I just got the MQ client trace and will try to post it later. Before that, I need to mask some data in the trace, which might take some time. |
|
| Back to top |
|
|
| bef1 |
| Posted: Thu Aug 06, 2020 11:54 am Post subject: |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
| PeterPotkay wrote: |
Consider a queue with 100 messages with a CorrelID of various names, and message #100 has a Correl ID of PETER.
By doing a selective JMS GET with a CorrelID of PETER, your JMS app will issue 100 MQGETs with Browse, and then one more MQGET under cursor to finally destructively pull the message off of the queue.
If the JMS GET is for Correl ID "ID:(HexValueForPETER)", then you will see only one MQGET occurring, and it is a destructive MQGET right off of bat.
101 MQGETs versus 1 MQGET. Obviously, the deeper the queue, the more important this is, and if the queue is close to zero as it should be, the less important it is. |
That would makes sense. However, in this case the client is only retrieving one single message (there are no duplicate IDs). And it is a single "REQUEST_MSGS" request that the MQ server is processing during many seconds before returning a single "ASYNC_MESSAGE".
This is what I saw by analysing the requests using a tcpdump and Wireshark. I will try to post an MQ client trace when I get some time to mask the data in it. |
|
| Back to top |
|
|
| bef1 |
| Posted: Thu Aug 06, 2020 12:26 pm Post subject: |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
| bruce2359 wrote: |
Does the app disconnect/reconnect for each message?
Id be interested in seeing results of an API trace. |
There is only one message that has the request JMSCorrelationID. The MQ server takes many to process the request.
The MQ client request goes as follows:
| Code: |
MQ Client -- [OPEN <Response_queue> with JMSCorrelationID as a selector] --> MQ Server
MQ Server -- [OPEN response] --> MQ Client
MQ Client -- [REQUEST_MSGS *without any selector set*] --> MQ Server
MQ Server -- [ASYNC_MESSAGE] --> MQ Client
MQ Client -- [MQCLOSE] --> MQ Server
MQ Server -- [MQCLOSE_REPLY] --> MQ Client
MQ Client -- [MQDISC] --> MQ Server
MQ Server -- [MQDISC_REPLY] --> MQ Client
|
I will try to post an MQ client trace later if I'm able to mask the data in the trace. |
|
| Back to top |
|
|
| bef1 |
| Posted: Thu Aug 06, 2020 1:39 pm Post subject: |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
The complete MQ client trace is pretty huge (over 30K lines), and masking the sensitive data would be very time consuming and error-prone.
So I will instead share a sample of the tcpdump capture analysis using Wireshark.
As you can see in this trace, the client retrieves one message from the response queue using the following steps:
1. The client opens the queue using the JMSCorrelationID selector, but without setting the MQMH CorrelationID
2. The MQ server responds to the previous request
3. The client sends a request_message request without any selector
<There is a 15s delay before the next step, while the MQ server is processing the request. The response time increases in a linear or maybe even incremental manner in relation to the queue depth.>
4. The MQ server sends back the message matching the JMSCorrelationID (that was set during the Open operation, on step 1) to the client
1. MQ Client --[SPI (Open)]--> MQ Server
| Code: |
WebSphere MQ (SPI)
Transmission Segment Header
StructID..: TSHM
MQSegmLen.: 632
Convers ID: 5
Request ID: 0
Byte order: Big endian (0x01)
SegmType..: SPI (0x8c)
Ctl Flag 1: 0x30, Last Seg, First Seg
0... .... = DLQ used: Not set
.0.. .... = Req accept: Not set
..1. .... = Last Seg: Set
...1 .... = First Seg: Set
.... 0... = Close Chnl: Not set
.... .0.. = Req close: Not set
.... ..0. = Error: Not set
.... ...0 = Confirm Req: Not set
Ctl Flag 2: 0x00
.... 0... = CommitIntvl: Not set
.... .0.. = CSH: Not set
.... ..0. = MSG Comp: Not set
.... ...0 = HDR Comp: Not set
LUW Ident.: 0000000000000000
Encoding..: 111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
CCSID.....: (819)
Reserved..: 0x0000
API Header
Reply len..: 10872
Compl Code.: MQCC_OK (0)
Reason Code: MQRC_NONE (0)
Object Hdl.: 0xffffffff
SPI
SPI Verb: OPEN (12)
Version: 2
Max reply size: 552
SPOU
SPI Structid: SPOU
Version: 1
Length: 12
SPOI
SPI Structid: SPOI
Version: 2
Length: 556
LPOO
StructID......: LPOO
Version.......: 3
LpiVersion....: 0x00002029
LpiOpts.......: 0x00000023, SAVE_ORIGIN_CTXT, SAVE_IDENTITY_CTXT
.... .... .... .... .... .... .... .0.. = SAVE_USER_CTXT: Not set
.... .... .... .... .... .... .... ..1. = SAVE_ORIGIN_CTXT: Set
.... .... .... .... .... .... .... ...1 = SAVE_IDENTITY_CTXT: Set
DefPersistence: MQPER_PERSISTENCE_AS_PARENT (-1)
DefPutRespType: Unknown (-1)
DefReadAHead..: Unknown (-1)
PropertyCtl...: -1
qprotect......:
Options: 0x00000000
Bind/Read Ahead As Q Def
ExtraData.....: 10240
Object Descriptor
StructID.........: OD
version..........: 4
ObjType..........: MQOT_Q (1)
ObjName..........: QA.REP.QUEUE
ObjQMgr..........:
DynQName.........: AMQ.*
AltUserID........:
NbrRecord........: 0
Known Dest Count.: 0
Unknown Dest Cnt.: 0
Invalid Dest Cnt.: 0
Offset of 1st OR.: 0
Offset of 1st RR.: 0
Addr of 1st OR.: 0x00000000
Addr of 1st RR.: 0x00000000
Alt security id..:
Resolved Q Name..:
Resolved QMgrName:
Object string - [Empty]
VLStr Addr.: 0x00000000
VLStr Offs.: 400
VLStr BufSz: 0
VLStr Len..: 0
VLStr Ccsid: 1208
VLStr Value: [Empty]
Selection string -
VLStr Addr.: 0x00000000
VLStr Offs.: 400
VLStr BufSz: 10240
VLStr Len..: 55
VLStr Ccsid: 1208
VLStr Value:
Resolved object string - [Empty]
VLStr Addr.: 0x00000000
VLStr Offs.: 456
VLStr BufSz: 0
VLStr Len..: 0
VLStr Ccsid: 1208
VLStr Value: [Empty] *****COMMENT: See next comment in regards to this value*****
Resolv Obj Type..: MQOT_NONE (0)
Unknown (0x4a4d5343) --> *****COMMENT: For some reason, this value is unfortunately not recognized by the Wireshark MQ protocol dissector, but it is obviously the "VLStr Value", as it contains this ASCII string: [i]JMSCorrelationID='43020c70-b06c-48bd-8d01-d97233b6bd77'[/i]. So it is clear that the client application sets the JMSCorrelationID selector on the OPEN operation (not on the REQUEST_MSGS operation).*****
|
2. MQ Server --[SPI (Open) reply]--> MQ Client
| Code: |
WebSphere MQ (SPI_REPLY)
Transmission Segment Header
StructID..: TSHM
MQSegmLen.: 848
Convers ID: 5
Request ID: 0
Byte order: Big endian (0x01)
SegmType..: SPI_REPLY (0x9c)
Ctl Flag 1: 0x30, Last Seg, First Seg
0... .... = DLQ used: Not set
.0.. .... = Req accept: Not set
..1. .... = Last Seg: Set
...1 .... = First Seg: Set
.... 0... = Close Chnl: Not set
.... .0.. = Req close: Not set
.... ..0. = Error: Not set
.... ...0 = Confirm Req: Not set
Ctl Flag 2: 0x00
.... 0... = CommitIntvl: Not set
.... .0.. = CSH: Not set
.... ..0. = MSG Comp: Not set
.... ...0 = HDR Comp: Not set
LUW Ident.: 0000000000000000
Encoding..: 111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
CCSID.....: (819)
Reserved..: 0x0000
API Header
Reply len..: 840
Compl Code.: MQCC_OK (0)
Reason Code: MQRC_NONE (0)
Object Hdl.: 0x00000002
SPI
SPI Verb: OPEN (12)
Version: 2
Max reply size: 552
SPOU
SPI Structid: SPOU
Version: 1
Length: 12
SPOO
SPI Structid: SPOO
Version: 2
Length: 772
LPOO
StructID......: LPOO
Version.......: 3
LpiVersion....: 0x00002029
LpiOpts.......: 0x00000023, SAVE_ORIGIN_CTXT, SAVE_IDENTITY_CTXT
.... .... .... .... .... .... .... .0.. = SAVE_USER_CTXT: Not set
.... .... .... .... .... .... .... ..1. = SAVE_ORIGIN_CTXT: Set
.... .... .... .... .... .... .... ...1 = SAVE_IDENTITY_CTXT: Set
DefPersistence: MQPER_PERSISTENT (1)
DefPutRespType: MQPRT_SYNC_RESPONSE (1)
DefReadAHead..: MQREADA_NO (0)
PropertyCtl...: 0
qprotect......: SYSTEM.PROTECTION.ERROR.QUEUE
Options: 0x0000022c, PASS_ALL_CONTEXT, INQUIRE, BROWSE, INPUT_EXCLUSIVE
.... .... .0.. .... .... .... .... .... = BIND_ON_GROUP: Not set
.... .... ..0. .... .... .... .... .... = NO_MULTICAST: Not set
.... .... ...0 .... .... .... .... .... = READ_AHEAD: Not set
.... .... .... 0... .... .... .... .... = NO_READ_AHEAD: Not set
.... .... .... .0.. .... .... .... .... = RESOLVE_LOCAL_Q: Not set
.... .... .... ..0. .... .... .... .... = CO_OP: Not set
.... .... .... ...0 .... .... .... .... = RESOLVE_NAMES: Not set
.... .... .... .... 0... .... .... .... = BIND_NOT_FIXED: Not set
.... .... .... .... .0.. .... .... .... = BIND_ON_OPEN: Not set
.... .... .... .... ..0. .... .... .... = FAIL_IF_QUIESCING: Not set
.... .... .... .... ...0 .... .... .... = ALTERNATE_USER_AUTHORITY: Not set
.... .... .... .... .... 0... .... .... = SET_ALL_CONTEXT: Not set
.... .... .... .... .... .0.. .... .... = SET_IDENTITY_CONTEXT: Not set
.... .... .... .... .... ..1. .... .... = PASS_ALL_CONTEXT: Set
.... .... .... .... .... ...0 .... .... = PASS_IDENTITY_CONTEXT: Not set
.... .... .... .... .... .... 0... .... = SAVE_ALL_CONTEXT: Not set
.... .... .... .... .... .... .0.. .... = SET: Not set
.... .... .... .... .... .... ..1. .... = INQUIRE: Set
.... .... .... .... .... .... ...0 .... = OUTPUT: Not set
.... .... .... .... .... .... .... 1... = BROWSE: Set
.... .... .... .... .... .... .... .1.. = INPUT_EXCLUSIVE: Set
.... .... .... .... .... .... .... ..0. = INPUT_SHARED: Not set
.... .... .... .... .... .... .... ...0 = INPUT_AS_Q_DEF: Not set
ExtraData.....: 216
Object Descriptor
StructID.........: OD
version..........: 4
ObjType..........: MQOT_Q (1)
ObjName..........: QA.REP.QUEUE
ObjQMgr..........:
DynQName.........: AMQ.*
AltUserID........:
NbrRecord........: 0
Known Dest Count.: 1
Unknown Dest Cnt.: 0
Invalid Dest Cnt.: 0
Offset of 1st OR.: 0
Offset of 1st RR.: 0
Addr of 1st OR.: 0x00000000
Addr of 1st RR.: 0x00000000
Alt security id..:
Resolved Q Name..: QL.REP.QUEUE
Resolved QMgrName: MQSERVER
Object string - [Empty]
VLStr Addr.: 0x00000000
VLStr Offs.: 400
VLStr BufSz: 0
VLStr Len..: 0
VLStr Ccsid: 1208
VLStr Value: [Empty]
Selection string -
VLStr Addr.: 0x00000000
VLStr Offs.: 400
VLStr BufSz: 10240
VLStr Len..: 55
VLStr Ccsid: 1208
VLStr Value:
[Expert Info (Warning/Undecoded): Trailing stray characters]
[Trailing stray characters]
[Severity level: Warning]
[Group: Undecoded]
Resolved object string - [Empty]
VLStr Addr.: 0x00000000
VLStr Offs.: 456
VLStr BufSz: 0
VLStr Len..: 0
VLStr Ccsid: 1208
VLStr Value: [Empty]
Resolv Obj Type..: MQOT_Q (1)
|
3. MQ Client --[REQUEST_MSGS]--> MQ Server
| Code: |
WebSphere MQ (REQUEST_MSGS)
Transmission Segment Header
StructID..: TSHM
MQSegmLen.: 148
Convers ID: 5
Request ID: 1
Byte order: Big endian (0x01)
SegmType..: REQUEST_MSGS (0x0e)
Ctl Flag 1: 0x30, Last Seg, First Seg
0... .... = DLQ used: Not set
.0.. .... = Req accept: Not set
..1. .... = Last Seg: Set
...1 .... = First Seg: Set
.... 0... = Close Chnl: Not set
.... .0.. = Req close: Not set
.... ..0. = Error: Not set
.... ...0 = Confirm Req: Not set
Ctl Flag 2: 0x00
.... 0... = CommitIntvl: Not set
.... .0.. = CSH: Not set
.... ..0. = MSG Comp: Not set
.... ...0 = HDR Comp: Not set
LUW Ident.: 0000000000000000
Encoding..: 111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
CCSID.....: (819)
Reserved..: 0x0000
REQUEST MESSAGE
version..: 0x00000001 (1)
handle...: 0x00000002
RecvBytes: 0x00000000 (0)
RqstBytes: 0x00000001 (1)
MaxMsgLen: 0x00001000 (4096)
GetMsgOpt: 0x02003001, PROPERTIES_FORCE_MQRFH2, FAIL_IF_QUIESCING, SYNCPOINT_IF_PERSISTENT, WAIT
...0 .... .... .... .... .... .... .... = PROPERTIES_COMPATIBILITY: Not set
.... 0... .... .... .... .... .... .... = PROPERTIES_IN_HANDLE: Not set
.... .0.. .... .... .... .... .... .... = NO_PROPERTIES: Not set
.... ..1. .... .... .... .... .... .... = PROPERTIES_FORCE_MQRFH2: Set
.... ...0 .... .... .... .... .... .... = UNMARKED_BROWSE_MSG: Not set
.... .... 0... .... .... .... .... .... = UNMARK_BROWSE_HANDLE: Not set
.... .... .0.. .... .... .... .... .... = UNMARK_BROWSE_CO_OP: Not set
.... .... ..0. .... .... .... .... .... = MARK_BROWSE_CO_OP: Not set
.... .... ...0 .... .... .... .... .... = MARK_BROWSE_HANDLE: Not set
.... .... .... .0.. .... .... .... .... = ALL_SEGMENTS_AVAILABLE: Not set
.... .... .... ..0. .... .... .... .... = ALL_MSGS_AVAILABLE: Not set
.... .... .... ...0 .... .... .... .... = COMPLETE_MSG: Not set
.... .... .... .... 0... .... .... .... = LOGICAL_ORDER: Not set
.... .... .... .... .0.. .... .... .... = CONVERT: Not set
.... .... .... .... ..1. .... .... .... = FAIL_IF_QUIESCING: Set
.... .... .... .... ...1 .... .... .... = SYNCPOINT_IF_PERSISTENT: Set
.... .... .... .... .... 0... .... .... = BROWSE_MSG_UNDER_CURSOR: Not set
.... .... .... .... .... .0.. .... .... = UNLOCK: Not set
.... .... .... .... .... ..0. .... .... = LOCK: Not set
.... .... .... .... .... ...0 .... .... = MSG_UNDER_CURSOR: Not set
.... .... .... .... .... .... 0... .... = MARK_SKIP_BACKOUT: Not set
.... .... .... .... .... .... .0.. .... = ACCEPT_TRUNCATED_MSG: Not set
.... .... .... .... .... .... ..0. .... = BROWSE_NEXT: Not set
.... .... .... .... .... .... ...0 .... = BROWSE_FIRST: Not set
.... .... .... .... .... .... .... 0... = SET_SIGNAL: Not set
.... .... .... .... .... .... .... .0.. = NO_SYNCPOINT: Not set
.... .... .... .... .... .... .... ..0. = SYNCPOINT: Not set
.... .... .... .... .... .... .... ...1 = WAIT: Set
WaitIntrv: 0x00003a98 (15000)
QueStatus: 0x00000000
RqstFlags: 0x00000018 (24), REQ_MSG_SELECTION, REQ_MSG_F00000008
.... .... .... .... .... .... ...1 .... = REQ_MSG_SELECTION: Set
.... .... .... .... .... .... .... 1... = REQ_MSG_F00000008: Set
.... .... .... .... .... .... .... .0.. = REQ_MSG_F00000004: Not set
.... .... .... .... .... .... .... ..0. = REQ_MSG_F00000002: Not set
GlbMsgIdx: 0x00000000 (0)
SelectIdx: 0x0001 (1)
MQMDVers.: 0x0002 (2)
CCSID....: (1208)
Encoding.: 111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
MsgSeqNum: 0x00000000 (0)
Offset...: 0x00000000 (0)
MatchOpt.: 0x00000003, MATCH_CORREL_ID, MATCH_MSG_ID
.... .... .... .... .... .... ..0. .... = MATCH_MSG_TOKEN: Not set
.... .... .... .... .... .... ...0 .... = MATCH_OFFSET: Not set
.... .... .... .... .... .... .... 0... = MATCH_MSG_SEQ_NUMBER: Not set
.... .... .... .... .... .... .... .0.. = MATCH_GROUP_ID: Not set
.... .... .... .... .... .... .... ..1. = MATCH_CORREL_ID: Set
.... .... .... .... .... .... .... ...1 = MATCH_MSG_ID: Set
mtchMsgId: 000000000000000000000000000000000000000000000000 *****COMMENT: As you can see, the MQMH Message ID is not set on the request message request.*****
mtchCorID: 000000000000000000000000000000000000000000000000 *****COMMENT: As you can see, the MQMH Correlation ID is not set on the request message request.*****
|
2. MQ Server --[ASYNC_MESSAGE]--> MQ Client
| Code: |
WebSphere MQ [ASYNC_MESSAGE of a Full MQ Segment] Hdl=0x0002 GlbMsgIdx=1, SegIdx=0, SegLen=3864
Transmission Segment Header
StructID..: TSHM
MQSegmLen.: 4332
Convers ID: 5
Request ID: 1
Byte order: Big endian (0x01)
SegmType..: ASYNC_MESSAGE (0x0d)
Ctl Flag 1: 0x30, Last Seg, First Seg
0... .... = DLQ used: Not set
.0.. .... = Req accept: Not set
..1. .... = Last Seg: Set
...1 .... = First Seg: Set
.... 0... = Close Chnl: Not set
.... .0.. = Req close: Not set
.... ..0. = Error: Not set
.... ...0 = Confirm Req: Not set
Ctl Flag 2: 0x00
.... 0... = CommitIntvl: Not set
.... .0.. = CSH: Not set
.... ..0. = MSG Comp: Not set
.... ...0 = HDR Comp: Not set
LUW Ident.: 0000000000000000
Encoding..: 111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
CCSID.....: (819)
Reserved..: 0x0000
ASYNC MESSAGE
version..: 0x00000001 (1)
handle...: 0x00000002
MsgIndex.: 0x00000001 (1)
GlbMsgIdx: 0x00000001 (1)
SegLength: 0x00000f18 (3864)
SegmIndex: 0x0000 (0)
SeleIndex: 0x0001 (1)
ReasonCod: MQRC_NONE (0)
TotMsgLen: 0x00000f18 (3864)
ActMsgLen: 0x00000f18 (3864)
MsgToken.: 5eb5e06d000000410000000000000853
status...: 0x0001
resolQNLn: 12
resolQNme: QL.REP.QUEUE
Padding..: 00
Message Descriptor
StructID.: MD
Version..: 2
Report...: 0
Msg Type.: MQMT_REPLY (2)
Expiry .: -1
Feedback.: 0
Encoding.: 311-785 (FLT_S390/DEC_NORMAL/INT_NORMAL)
CCSID....: (500)
Format...: MQHRF2
Priority.: 4
Persist..: MQPER_NOT_PERSISTENT (0)
Msg ID...: c3e2d840d4d8e5f14040404040404040d8532c2d5397e595
CorrelID.: 414d51204d51444556312020202020205f2abac72131ce61
BackoCnt.: 0
ReplyToQ.:
RepToQMgr: MQV1
UserId...: mqcartd
AccntTok.: 000000000000000000000000000000000000000000000000...
AppIdData:
PutAppTyp: MQAT_XCF (20)
PutAppNme: DESJD1 IMSDEA
PutDatGMT: 20200805
PutTimGMT: 14492085
AppOriDat:
GroupId..: 000000000000000000000000000000000000000000000000
MsgSeqNum: 1
Offset...: 0
Msg flags: 0x00000000
Orig len.: -1
Rules and Formatting Header
Structid: RFH
version.: 2
Length..: 424
Encoding: 311-785 (FLT_S390/DEC_NORMAL/INT_NORMAL)
CCSID...: (500)
Format..: MQIMS
Flags...: 0x00000000
NmeValCCSID: (1208)
NameValue: <mcd><Msd>jms_bytes</Msd></mcd>
Len.: 32
Val.: *****COMMENT: phpBB doesn't accept these non-ASCII characters, so I am truncating them.*****
NameValue: <jms><Dst>queue:///QAC.REQ.QUEUE</Dst><Rto>queue:///QA.REP.QUEUE</Rto><Tms>1596638959834</Tms><Cid>43020c70-b06c-48bd-8d01-d97233b6bd77</Cid><Dlv>1</Dlv></jms>
Len.: 160
Val.: *****COMMENT: phpBB doesn't accept these non-ASCII characters, so I am truncating them.*****
NameValue: <usr><b3>cb4719cfd44917360a1319f299b6bbb3-4702e37e04f1b618-0</b3><Format>MQIMS </Format><Report dt='i4'>0</Report><Encoding dt='i4'>273</Encoding><MsgType dt='i4'>1</MsgType></usr>
Len.: 184
Val.: *****COMMENT: phpBB doesn't accept these non-ASCII characters, so I am truncating them.*****
Data (3440 bytes)
Data: *****COMMENT: I am removing this data from the post*****
[Length: 3440]
|
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Aug 06, 2020 2:06 pm Post subject: Re: JMS selector - performance issue on deep queues |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
| bef1 wrote: |
We traced the requests being sent by the client application (using tcpdump and Wireshark), and we see that is seems to behave accordingly, sending the following selector when doing the OPEN operation on the response queue: JMSCorrelationID='45030c70-b06c-49bd-9d01-d97243b6bd88'
So as far as we can tell, our application complies to IBM's recommendations, but it still scales very badly with the queue depth.
-----
I would like to know if there is something else we should look into to get a better performance on (somewhat) deep queues. |
Just look at the format given to your JMSCorrelationID and you'll see that it does not match the format of the string obtained through
| Code: |
msg.getJMSMessageID ;
// or
msg.getJMSCorrelationID();
// the correct format for the selector looks like:
// "ID:414d5120465232202020202020202020da69295f2310a130" |
So no you don't conform to the IBM recommendation and each message will have to be inspected, hence your deep queue times.
Also this correlation ID looks very suspiciously like a correlation ID from a different (non MQ) Messaging System. Do you per any chance run a bridge to MQ? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bef1 |
| Posted: Thu Aug 06, 2020 3:35 pm Post subject: Re: JMS selector - performance issue on deep queues |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
Edit: removed this post. Please refer to next post instead.
Last edited by bef1 on Thu Aug 06, 2020 6:15 pm; edited 2 times in total |
|
| Back to top |
|
|
| bef1 |
| Posted: Thu Aug 06, 2020 4:41 pm Post subject: Re: JMS selector - performance issue on deep queues |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
| fjb_saper wrote: |
Just look at the format given to your JMSCorrelationID and you'll see that it does not match the format of the string obtained through
| Code: |
msg.getJMSMessageID ;
// or
msg.getJMSCorrelationID();
// the correct format for the selector looks like:
// "ID:414d5120465232202020202020202020da69295f2310a130" |
So no you don't conform to the IBM recommendation and each message will have to be inspected, hence your deep queue times.
Also this correlation ID looks very suspiciously like a correlation ID from a different (non MQ) Messaging System. |
I spent some time reviewing IBM's documentation, and I think that we actually conform to IBM's specifications:
| Quote: |
The MQMD CorrelId field can hold a standard WebSphere MQ Correlation ID of 48 hexadecimal digits (24 bytes). The JMSCorrelationID can be a byte[] value, a string value containing hexadecimal characters and prefixed with ID:, or an arbitrary string value not beginning ID:. The first two of these represent a standard WebSphere MQ Correlation ID and map directly to or from the MQMD CorrelId field (truncated or padded with zeros as applicable); they do not use the MQRFH2 jms.Cid field. The third (arbitrary string) uses the MQRFH2 jms.Cid field; the first 24 bytes of the string, in UTF-8 format, are written into the MQMD CorrelID.
|
I have reviewed a TCP trace, and we are using the 3rd scenario. Here is an example from one PUT message:
RFH header:
| Code: |
Rules and Formatting Header
Structid: RFH
version.: 2
Length..: 424
Encoding: 111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
CCSID...: (1208)
Format..: MQIMS
Flags...: 0x00000000
NmeValCCSID: (1208)
NameValue: <mcd><Msd>jms_bytes</Msd></mcd>
Len.: 32
Val.: <mcd><Msd>jms_bytes</Msd></mcd>
NameValue: <jms><Dst>queue:///QAC.REQ.QUEUE</Dst><Rto>queue:///QA.REP.QUEUE</Rto><Tms>1596638959834</Tms><Cid>43020c70-b06c-48bd-8d01-d97233b6bd77</Cid><Dlv>1</Dlv></jms>
Len.: 160
Val.: <jms><Dst>queue:///QAC.REQ.QUEUE</Dst><Rto>queue:///QA.REP.ICDE</Rto><Tms>1596638959834</Tms><Cid>43020c70-b06c-48bd-8d01-d97233b6bd77</Cid><Dlv>1</Dlv></jms>
NameValue: <usr><b3>cb4719cfd44917360a1319f299b6bbb3-4702e37e04f1b618-0</b3><Format>MQIMS </Format><Report dt='i4'>0</Report><Encoding dt='i4'>273</Encoding><MsgType dt='i4'>1</MsgType></usr>
Len.: 184
Val.: <usr><b3>cb4719cfd44917360a1319f299b6bbb3-4702e37e04f1b618-0</b3><Format>MQIMS </Format><Report dt='i4'>0</Report><Encoding dt='i4'>273</Encoding><MsgType dt='i4'>1</MsgType></usr>
|
MQMH Correlation ID:
| Code: |
| 34333032306337302d623036632d343862642d386430312d |
This hex value, from MQMH header (34333032306337302d623036632d343862642d386430312d) can be converted to ASCII as follows, which corresponds to the first 24 bytes of the JMSCorrelationID: 43020c70-b06c-48bd-8d01-
So everything looks conform to the specs, as far as I can tell.
Then, the next question would be: Does this is this 3rd scenario perform as well as the first 2 (the ones that start with "ID:" and that you were expecting to see). I will quote IBM MQ's v9 developer guide on this:
| Quote: |
Selecting messages from a queue requires IBM MQ to sequentially inspect each message on the queue.
Messages are inspected until a message is found that matches the selection criteria or there are no more
messages to examine. Therefore, messaging performance suffers if message selection is used on deep
queues.
To optimize message selection on deep queues when selection is based on JMSCorrelationID or
JMSMessageID, use a selection string of the form JMSCorrelationID = ... or JMSMessageID
= ... and reference only one property.
This method offers a significant improvement in performance for selection on JMSCorrelationID and
offers a marginal performance improvement for JMSMessageID.
|
So it sounds to me like everything the client application is doing looks conform to IBM's specifications and recommandations.
Am I missing something? |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Aug 06, 2020 9:58 pm Post subject: Re: JMS selector - performance issue on deep queues |
|
|
Padawan
Joined: 09 May 2013
Posts: 1967
Location: Bay of Plenty, New Zealand
|
| bef1 wrote: |
| Am I missing something? |
What does the actual client code that sets the JMS Correlation ID look like?
I don't see anywhere where you show that you are using the "ID:" format. That is what I think you are missing.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| bef1 |
| Posted: Thu Aug 06, 2020 10:48 pm Post subject: Re: JMS selector - performance issue on deep queues |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
| hughson wrote: |
| I don't see anywhere where you show that you are using the "ID:" format. That is what I think you are missing. |
I am not sure if you saw my previous post on this thread. Short version: I confirm that the client is not using the ID: format. It is instead using the 3rd format that the MQ developer’s guide allows, i.e. an arbitrary string value not beginning ID: [...] The third (arbitrary string) uses the MQRFH2 jms.Cid field; the first 24 bytes of the string, in UTF-8 format, are written into the MQMD CorrelID. .
If this is the root cause of our performance issue, then I only see one possible reason: MQ does not behave accordingly to IBM’s documentation, since it is not specified anywhere that this 3rd format (an arbitrary string) will lead MQ to have to parse the messages, instead of using the arbitrary string to find messages having matching MDMH correlation ID. |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Aug 07, 2020 1:58 am Post subject: Re: JMS selector - performance issue on deep queues |
|
|
Padawan
Joined: 09 May 2013
Posts: 1967
Location: Bay of Plenty, New Zealand
|
| bef1 wrote: |
| I confirm that the client is not using the ID: format. |
Can you try it out to discover whether what we say is true - that it will solve your performance problem. It was certainly added in order to achieve exactly that.
| bef1 wrote: |
| It is instead using the 3rd format that the MQ developer's guide allows, i.e. an arbitrary string value not beginning ID: [...] The third (arbitrary string) uses the MQRFH2 jms.Cid field; the first 24 bytes of the string, in UTF-8 format, are written into the MQMD CorrelID. . |
Could you provide a link to this please? Is this for MQPUT or MQGET that it is describing?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| bef1 |
| Posted: Fri Aug 07, 2020 3:26 am Post subject: Re: JMS selector - performance issue on deep queues |
|
|
Novice
Joined: 29 Aug 2019
Posts: 15
|
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
