ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Java / JMS » JMS selector - performance issue on deep queues

Post new topic  Reply to topic Goto page 1, 2  Next
 JMS selector - performance issue on deep queues « View previous topic :: View next topic » 
Author Message
bef1
PostPosted: Thu Aug 06, 2020 4:59 am    Post subject: JMS selector - performance issue on deep queues Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

- Our MQ server is running on version 7.5.0.7 (yeah, I know...)
- The JMS client being used by the application causing the issue is using an MQ v9 client.

A new JMS client application was running a few days ago.

The MQ number of requests sent by this new application to our MQ server was not very high, in the range of a few dozen of requests every minute.

However, while the new application was running, it was experiencing slow response time to get messages from our MQ server (around 1,500 ms per message).

Also, while the new application was running, the amqzlaa0 processes on our MQ server seemed to reach some sort of bottleneck, as it was reaching a plateau at 42% CPU usage. There was still some CPU available on the MQ server, but the processes were not consuming it. There was no bottleneck on the I/O side, the I/O subsystem was actually performing optimally and the number of IOPS on the disks was reasonable.

-----

The response queue that was being read by the client application had a queue depth of 800 messages.

At one point, we cleared this same response queue, and we saw that the response time was then much better (just a few ms, instead of 1,500 ms per message before it was cleared).

We were able to reproduce this isuse in our test environement, and the response time scaled as follows:
- Queue depth of 1 message: A few ms
- Queue depth of 800 messages: 1,500 ms
- Queue depth of 4,000 messages: 15,000 ms

-----

The MQ development guides, for both MQ v7.5 ans MQ v9, provide the following guidelines:

Quote:

Messaging performance
Selecting messages from a queue requires IBM WebSphere MQ to sequentially inspect each message on the queue. Messages are inspected until a message is found that matches the selection criteria or there are no more messages to examine. Therefore, messaging performance suffers if message selection is used on deep queues.

To optimize message selection on deep queues when selection is based on JMSCorrelationID or JMSMessageID, use a selection string of the form JMSCorrelationID = ... or JMSMessageID = ... and reference only one property.

This method offers a significant improvement in performance for selection on JMSCorrelationID and offers a marginal performance improvement for JMSMessageID.


We traced the requests being sent by the client application (using tcpdump and Wireshark), and we see that is seems to behave accordingly, sending the following selector when doing the OPEN operation on the response queue: JMSCorrelationID='45030c70-b06c-49bd-9d01-d97243b6bd88'

So as far as we can tell, our application complies to IBM's recommendations, but it still scales very badly with the queue depth.

-----

I would like to know if there is something else we should look into to get a better performance on (somewhat) deep queues.
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Aug 06, 2020 5:54 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

Message Selection on message properties on a deep queue can indeed poorly perform, as you have read.

If all you need to do is select by MessageID or CorrelationID however, that should not need to use MQOPEN SelectionString, but instead it should use MQGET selection looking at the fields in the MQMD on input.

The quote you have found in Knowledge Center showing the specific way to provide a selection for one of these two fields is supposed to cause the underlying MQ API calls to switch from using MQOPEN SelectionString, to using MQGET MessageID or CorrelationID selection.

When you traced the API calls, you said that you saw the selector on MQOPEN. That suggests that it hasn't caused this switch.

The quote you found doesn't mention the need to start the value with "ID:" and then the hex string of the ID. There is/was a Tech Not explaining this, but in the IBM website reshuffle it does not appear to exist any more, so I am struggling to find you a reference.

There is some example code in an answer to this SO question which might help.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
bruce2359
PostPosted: Thu Aug 06, 2020 6:04 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Does the app disconnect/reconnect for each message?

Id be interested in seeing results of an API trace.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Aug 06, 2020 11:25 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Consider a queue with 100 messages with a CorrelID of various names, and message #100 has a Correl ID of PETER.

By doing a selective JMS GET with a CorrelID of PETER, your JMS app will issue 100 MQGETs with Browse, and then one more MQGET under cursor to finally destructively pull the message off of the queue.

If the JMS GET is for Correl ID "ID:(HexValueForPETER)", then you will see only one MQGET occurring, and it is a destructive MQGET right off of bat.

101 MQGETs versus 1 MQGET. Obviously, the deeper the queue, the more important this is, and if the queue is close to zero as it should be, the less important it is.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bef1
PostPosted: Thu Aug 06, 2020 11:47 am    Post subject: Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

hughson wrote:

If all you need to do is select by MessageID or CorrelationID however, that should not need to use MQOPEN SelectionString, but instead it should use MQGET selection looking at the fields in the MQMD on input.


OK that makes sense, MQ would then only need to parse the headers instead of inspecting the messages payload. Also, from my understanding, the MQMH correlation ID is indexed on Distributed MQ, so it would obviously perform much better if it was being used as a selector.

One issue I have is that the client application is using Spring JMS frameworks, so they currently do not mess with MQMH headers. They stick to the JMS layer.

We might have to look if they are able to set MQMH headers instead of JMSCorrelationID as a selector, but I'm not sure if that is possible using the same framworks that are currently being used.

hughson wrote:

The quote you have found in Knowledge Center showing the specific way to provide a selection for one of these two fields is supposed to cause the underlying MQ API calls to switch from using MQOPEN SelectionString, to using MQGET MessageID or CorrelationID selection.

When you traced the API calls, you said that you saw the selector on MQOPEN. That suggests that it hasn't caused this switch.


That's being done on the client side, right? So the mq client JAR would be expected to figure out that it must set the MQMH correlation ID, based on the fact that the JMSCorrelationID was set, correct?

I was hoping upgrading MQ to a more modern version would fix this issue, but in that case, I understand it wouldn't, since the behaviour is controlled by the client side, and the MQ client JAR is already up to date.

hughson wrote:

The quote you found doesn't mention the need to start the value with "ID:" and then the hex string of the ID. There is/was a Tech Not explaining this, but in the IBM website reshuffle it does not appear to exist any more, so I am struggling to find you a reference.

There is some example code in an answer to this SO question which might help.


Maybe there might be some sort of formatting issue being done by our client application which would format the JMSCorrelationID in an improper way, which would not allow the MQ client to switch to using the MQMH correlation ID...? I was inspecting the headers using a tcpdump and Wireshark, so the JMSCorrelationID='45030c70-b06c-49bd-9d01-d97243b6bd88' format is how the MQ client sends the request "on the wire".

I just got the MQ client trace and will try to post it later. Before that, I need to mask some data in the trace, which might take some time.
Back to top
View user's profile Send private message
bef1
PostPosted: Thu Aug 06, 2020 11:54 am    Post subject: Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

PeterPotkay wrote:
Consider a queue with 100 messages with a CorrelID of various names, and message #100 has a Correl ID of PETER.

By doing a selective JMS GET with a CorrelID of PETER, your JMS app will issue 100 MQGETs with Browse, and then one more MQGET under cursor to finally destructively pull the message off of the queue.

If the JMS GET is for Correl ID "ID:(HexValueForPETER)", then you will see only one MQGET occurring, and it is a destructive MQGET right off of bat.

101 MQGETs versus 1 MQGET. Obviously, the deeper the queue, the more important this is, and if the queue is close to zero as it should be, the less important it is.

That would makes sense. However, in this case the client is only retrieving one single message (there are no duplicate IDs). And it is a single "REQUEST_MSGS" request that the MQ server is processing during many seconds before returning a single "ASYNC_MESSAGE".

This is what I saw by analysing the requests using a tcpdump and Wireshark. I will try to post an MQ client trace when I get some time to mask the data in it.
Back to top
View user's profile Send private message
bef1
PostPosted: Thu Aug 06, 2020 12:26 pm    Post subject: Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

bruce2359 wrote:
Does the app disconnect/reconnect for each message?

Id be interested in seeing results of an API trace.


There is only one message that has the request JMSCorrelationID. The MQ server takes many to process the request.

The MQ client request goes as follows:

Code:

MQ Client -- [OPEN <Response_queue> with JMSCorrelationID as a selector] --> MQ Server
MQ Server -- [OPEN response] --> MQ Client

MQ Client -- [REQUEST_MSGS *without any selector set*] --> MQ Server
MQ Server -- [ASYNC_MESSAGE] --> MQ Client

MQ Client -- [MQCLOSE] --> MQ Server
MQ Server -- [MQCLOSE_REPLY] --> MQ Client

MQ Client -- [MQDISC] --> MQ Server
MQ Server -- [MQDISC_REPLY] --> MQ Client


I will try to post an MQ client trace later if I'm able to mask the data in the trace.
Back to top
View user's profile Send private message
bef1
PostPosted: Thu Aug 06, 2020 1:39 pm    Post subject: Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

The complete MQ client trace is pretty huge (over 30K lines), and masking the sensitive data would be very time consuming and error-prone.

So I will instead share a sample of the tcpdump capture analysis using Wireshark.

As you can see in this trace, the client retrieves one message from the response queue using the following steps:
1. The client opens the queue using the JMSCorrelationID selector, but without setting the MQMH CorrelationID
2. The MQ server responds to the previous request
3. The client sends a request_message request without any selector
<There is a 15s delay before the next step, while the MQ server is processing the request. The response time increases in a linear or maybe even incremental manner in relation to the queue depth.>
4. The MQ server sends back the message matching the JMSCorrelationID (that was set during the Open operation, on step 1) to the client

1. MQ Client --[SPI (Open)]--> MQ Server
Code:

WebSphere MQ (SPI)
    Transmission Segment Header
        StructID..: TSHM
        MQSegmLen.: 632
        Convers ID: 5
        Request ID: 0
        Byte order: Big endian (0x01)
        SegmType..: SPI (0x8c)
        Ctl Flag 1: 0x30, Last Seg, First Seg
            0... .... = DLQ used: Not set
            .0.. .... = Req accept: Not set
            ..1. .... = Last Seg: Set
            ...1 .... = First Seg: Set
            .... 0... = Close Chnl: Not set
            .... .0.. = Req close: Not set
            .... ..0. = Error: Not set
            .... ...0 = Confirm Req: Not set
        Ctl Flag 2: 0x00
            .... 0... = CommitIntvl: Not set
            .... .0.. = CSH: Not set
            .... ..0. = MSG Comp: Not set
            .... ...0 = HDR Comp: Not set
        LUW Ident.: 0000000000000000
        Encoding..:      111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
        CCSID.....:  (819)
        Reserved..: 0x0000
    API Header
        Reply len..: 10872
        Compl Code.: MQCC_OK (0)
        Reason Code: MQRC_NONE (0)
        Object Hdl.: 0xffffffff
    SPI
        SPI Verb: OPEN (12)
        Version: 2
        Max reply size: 552
    SPOU
        SPI Structid: SPOU
        Version: 1
        Length: 12
    SPOI
        SPI Structid: SPOI
        Version: 2
        Length: 556
    LPOO
        StructID......: LPOO
        Version.......: 3
        LpiVersion....: 0x00002029
        LpiOpts.......: 0x00000023, SAVE_ORIGIN_CTXT, SAVE_IDENTITY_CTXT
            .... .... .... .... .... .... .... .0.. = SAVE_USER_CTXT: Not set
            .... .... .... .... .... .... .... ..1. = SAVE_ORIGIN_CTXT: Set
            .... .... .... .... .... .... .... ...1 = SAVE_IDENTITY_CTXT: Set
        DefPersistence: MQPER_PERSISTENCE_AS_PARENT (-1)
        DefPutRespType: Unknown (-1)
        DefReadAHead..: Unknown (-1)
        PropertyCtl...: -1
        qprotect......:                                                 
        Options: 0x00000000
            Bind/Read Ahead As Q Def
        ExtraData.....: 10240
    Object Descriptor
        StructID.........: OD 
        version..........: 4
        ObjType..........: MQOT_Q (1)
        ObjName..........: QA.REP.QUEUE                                     
        ObjQMgr..........:                                                 
        DynQName.........: AMQ.*                                           
        AltUserID........:             
        NbrRecord........: 0
        Known Dest Count.: 0
        Unknown Dest Cnt.: 0
        Invalid Dest Cnt.: 0
        Offset of 1st OR.: 0
        Offset of 1st RR.: 0
        Addr   of 1st OR.: 0x00000000
        Addr   of 1st RR.: 0x00000000
        Alt security id..:
        Resolved Q Name..:
        Resolved QMgrName:
        Object string - [Empty]
            VLStr Addr.: 0x00000000
            VLStr Offs.: 400
            VLStr BufSz: 0
            VLStr Len..: 0
            VLStr Ccsid: 1208
            VLStr Value: [Empty]
        Selection string -
            VLStr Addr.: 0x00000000
            VLStr Offs.: 400
            VLStr BufSz: 10240
            VLStr Len..: 55
            VLStr Ccsid: 1208
            VLStr Value:
        Resolved object string - [Empty]
            VLStr Addr.: 0x00000000
            VLStr Offs.: 456
            VLStr BufSz: 0
            VLStr Len..: 0
            VLStr Ccsid: 1208
            VLStr Value: [Empty] *****COMMENT: See next comment in regards to this value*****
        Resolv Obj Type..: MQOT_NONE (0)
    Unknown (0x4a4d5343) --> *****COMMENT: For some reason, this value is unfortunately not recognized by the Wireshark MQ protocol dissector, but it is obviously the "VLStr Value", as it contains this ASCII string: [i]JMSCorrelationID='43020c70-b06c-48bd-8d01-d97233b6bd77'[/i]. So it is clear that the client application sets the JMSCorrelationID selector on the OPEN operation (not on the REQUEST_MSGS operation).*****


2. MQ Server --[SPI (Open) reply]--> MQ Client
Code:

WebSphere MQ (SPI_REPLY)
    Transmission Segment Header
        StructID..: TSHM
        MQSegmLen.: 848
        Convers ID: 5
        Request ID: 0
        Byte order: Big endian (0x01)
        SegmType..: SPI_REPLY (0x9c)
        Ctl Flag 1: 0x30, Last Seg, First Seg
            0... .... = DLQ used: Not set
            .0.. .... = Req accept: Not set
            ..1. .... = Last Seg: Set
            ...1 .... = First Seg: Set
            .... 0... = Close Chnl: Not set
            .... .0.. = Req close: Not set
            .... ..0. = Error: Not set
            .... ...0 = Confirm Req: Not set
        Ctl Flag 2: 0x00
            .... 0... = CommitIntvl: Not set
            .... .0.. = CSH: Not set
            .... ..0. = MSG Comp: Not set
            .... ...0 = HDR Comp: Not set
        LUW Ident.: 0000000000000000
        Encoding..:      111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
        CCSID.....:  (819)
        Reserved..: 0x0000
    API Header
        Reply len..: 840
        Compl Code.: MQCC_OK (0)
        Reason Code: MQRC_NONE (0)
        Object Hdl.: 0x00000002
    SPI
        SPI Verb: OPEN (12)
        Version: 2
        Max reply size: 552
    SPOU
        SPI Structid: SPOU
        Version: 1
        Length: 12
    SPOO
        SPI Structid: SPOO
        Version: 2
        Length: 772
    LPOO
        StructID......: LPOO
        Version.......: 3
        LpiVersion....: 0x00002029
        LpiOpts.......: 0x00000023, SAVE_ORIGIN_CTXT, SAVE_IDENTITY_CTXT
            .... .... .... .... .... .... .... .0.. = SAVE_USER_CTXT: Not set
            .... .... .... .... .... .... .... ..1. = SAVE_ORIGIN_CTXT: Set
            .... .... .... .... .... .... .... ...1 = SAVE_IDENTITY_CTXT: Set
        DefPersistence: MQPER_PERSISTENT (1)
        DefPutRespType: MQPRT_SYNC_RESPONSE (1)
        DefReadAHead..: MQREADA_NO (0)
        PropertyCtl...: 0
        qprotect......: SYSTEM.PROTECTION.ERROR.QUEUE                   
        Options: 0x0000022c, PASS_ALL_CONTEXT, INQUIRE, BROWSE, INPUT_EXCLUSIVE
            .... .... .0.. .... .... .... .... .... = BIND_ON_GROUP: Not set
            .... .... ..0. .... .... .... .... .... = NO_MULTICAST: Not set
            .... .... ...0 .... .... .... .... .... = READ_AHEAD: Not set
            .... .... .... 0... .... .... .... .... = NO_READ_AHEAD: Not set
            .... .... .... .0.. .... .... .... .... = RESOLVE_LOCAL_Q: Not set
            .... .... .... ..0. .... .... .... .... = CO_OP: Not set
            .... .... .... ...0 .... .... .... .... = RESOLVE_NAMES: Not set
            .... .... .... .... 0... .... .... .... = BIND_NOT_FIXED: Not set
            .... .... .... .... .0.. .... .... .... = BIND_ON_OPEN: Not set
            .... .... .... .... ..0. .... .... .... = FAIL_IF_QUIESCING: Not set
            .... .... .... .... ...0 .... .... .... = ALTERNATE_USER_AUTHORITY: Not set
            .... .... .... .... .... 0... .... .... = SET_ALL_CONTEXT: Not set
            .... .... .... .... .... .0.. .... .... = SET_IDENTITY_CONTEXT: Not set
            .... .... .... .... .... ..1. .... .... = PASS_ALL_CONTEXT: Set
            .... .... .... .... .... ...0 .... .... = PASS_IDENTITY_CONTEXT: Not set
            .... .... .... .... .... .... 0... .... = SAVE_ALL_CONTEXT: Not set
            .... .... .... .... .... .... .0.. .... = SET: Not set
            .... .... .... .... .... .... ..1. .... = INQUIRE: Set
            .... .... .... .... .... .... ...0 .... = OUTPUT: Not set
            .... .... .... .... .... .... .... 1... = BROWSE: Set
            .... .... .... .... .... .... .... .1.. = INPUT_EXCLUSIVE: Set
            .... .... .... .... .... .... .... ..0. = INPUT_SHARED: Not set
            .... .... .... .... .... .... .... ...0 = INPUT_AS_Q_DEF: Not set
        ExtraData.....: 216
    Object Descriptor
        StructID.........: OD 
        version..........: 4
        ObjType..........: MQOT_Q (1)
        ObjName..........: QA.REP.QUEUE                                     
        ObjQMgr..........:                                                 
        DynQName.........: AMQ.*                                           
        AltUserID........:             
        NbrRecord........: 0
        Known Dest Count.: 1
        Unknown Dest Cnt.: 0
        Invalid Dest Cnt.: 0
        Offset of 1st OR.: 0
        Offset of 1st RR.: 0
        Addr   of 1st OR.: 0x00000000
        Addr   of 1st RR.: 0x00000000
        Alt security id..:
        Resolved Q Name..: QL.REP.QUEUE                                     
        Resolved QMgrName: MQSERVER                                         
        Object string - [Empty]
            VLStr Addr.: 0x00000000
            VLStr Offs.: 400
            VLStr BufSz: 0
            VLStr Len..: 0
            VLStr Ccsid: 1208
            VLStr Value: [Empty]
        Selection string -
            VLStr Addr.: 0x00000000
            VLStr Offs.: 400
            VLStr BufSz: 10240
            VLStr Len..: 55
            VLStr Ccsid: 1208
            VLStr Value:
                [Expert Info (Warning/Undecoded): Trailing stray characters]
                    [Trailing stray characters]
                    [Severity level: Warning]
                    [Group: Undecoded]
        Resolved object string - [Empty]
            VLStr Addr.: 0x00000000
            VLStr Offs.: 456
            VLStr BufSz: 0
            VLStr Len..: 0
            VLStr Ccsid: 1208
            VLStr Value: [Empty]
        Resolv Obj Type..: MQOT_Q (1)


3. MQ Client --[REQUEST_MSGS]--> MQ Server
Code:

WebSphere MQ (REQUEST_MSGS)
    Transmission Segment Header
        StructID..: TSHM
        MQSegmLen.: 148
        Convers ID: 5
        Request ID: 1
        Byte order: Big endian (0x01)
        SegmType..: REQUEST_MSGS (0x0e)
        Ctl Flag 1: 0x30, Last Seg, First Seg
            0... .... = DLQ used: Not set
            .0.. .... = Req accept: Not set
            ..1. .... = Last Seg: Set
            ...1 .... = First Seg: Set
            .... 0... = Close Chnl: Not set
            .... .0.. = Req close: Not set
            .... ..0. = Error: Not set
            .... ...0 = Confirm Req: Not set
        Ctl Flag 2: 0x00
            .... 0... = CommitIntvl: Not set
            .... .0.. = CSH: Not set
            .... ..0. = MSG Comp: Not set
            .... ...0 = HDR Comp: Not set
        LUW Ident.: 0000000000000000
        Encoding..:      111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
        CCSID.....:  (819)
        Reserved..: 0x0000
    REQUEST MESSAGE
        version..: 0x00000001 (1)
        handle...: 0x00000002
        RecvBytes: 0x00000000 (0)
        RqstBytes: 0x00000001 (1)
        MaxMsgLen: 0x00001000 (4096)
        GetMsgOpt: 0x02003001, PROPERTIES_FORCE_MQRFH2, FAIL_IF_QUIESCING, SYNCPOINT_IF_PERSISTENT, WAIT
            ...0 .... .... .... .... .... .... .... = PROPERTIES_COMPATIBILITY: Not set
            .... 0... .... .... .... .... .... .... = PROPERTIES_IN_HANDLE: Not set
            .... .0.. .... .... .... .... .... .... = NO_PROPERTIES: Not set
            .... ..1. .... .... .... .... .... .... = PROPERTIES_FORCE_MQRFH2: Set
            .... ...0 .... .... .... .... .... .... = UNMARKED_BROWSE_MSG: Not set
            .... .... 0... .... .... .... .... .... = UNMARK_BROWSE_HANDLE: Not set
            .... .... .0.. .... .... .... .... .... = UNMARK_BROWSE_CO_OP: Not set
            .... .... ..0. .... .... .... .... .... = MARK_BROWSE_CO_OP: Not set
            .... .... ...0 .... .... .... .... .... = MARK_BROWSE_HANDLE: Not set
            .... .... .... .0.. .... .... .... .... = ALL_SEGMENTS_AVAILABLE: Not set
            .... .... .... ..0. .... .... .... .... = ALL_MSGS_AVAILABLE: Not set
            .... .... .... ...0 .... .... .... .... = COMPLETE_MSG: Not set
            .... .... .... .... 0... .... .... .... = LOGICAL_ORDER: Not set
            .... .... .... .... .0.. .... .... .... = CONVERT: Not set
            .... .... .... .... ..1. .... .... .... = FAIL_IF_QUIESCING: Set
            .... .... .... .... ...1 .... .... .... = SYNCPOINT_IF_PERSISTENT: Set
            .... .... .... .... .... 0... .... .... = BROWSE_MSG_UNDER_CURSOR: Not set
            .... .... .... .... .... .0.. .... .... = UNLOCK: Not set
            .... .... .... .... .... ..0. .... .... = LOCK: Not set
            .... .... .... .... .... ...0 .... .... = MSG_UNDER_CURSOR: Not set
            .... .... .... .... .... .... 0... .... = MARK_SKIP_BACKOUT: Not set
            .... .... .... .... .... .... .0.. .... = ACCEPT_TRUNCATED_MSG: Not set
            .... .... .... .... .... .... ..0. .... = BROWSE_NEXT: Not set
            .... .... .... .... .... .... ...0 .... = BROWSE_FIRST: Not set
            .... .... .... .... .... .... .... 0... = SET_SIGNAL: Not set
            .... .... .... .... .... .... .... .0.. = NO_SYNCPOINT: Not set
            .... .... .... .... .... .... .... ..0. = SYNCPOINT: Not set
            .... .... .... .... .... .... .... ...1 = WAIT: Set
        WaitIntrv: 0x00003a98 (15000)
        QueStatus: 0x00000000
        RqstFlags: 0x00000018 (24), REQ_MSG_SELECTION, REQ_MSG_F00000008
            .... .... .... .... .... .... ...1 .... = REQ_MSG_SELECTION: Set
            .... .... .... .... .... .... .... 1... = REQ_MSG_F00000008: Set
            .... .... .... .... .... .... .... .0.. = REQ_MSG_F00000004: Not set
            .... .... .... .... .... .... .... ..0. = REQ_MSG_F00000002: Not set
        GlbMsgIdx: 0x00000000 (0)
        SelectIdx: 0x0001 (1)
        MQMDVers.: 0x0002 (2)
        CCSID....:  (1208)
        Encoding.:      111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
        MsgSeqNum: 0x00000000 (0)
        Offset...: 0x00000000 (0)
        MatchOpt.: 0x00000003, MATCH_CORREL_ID, MATCH_MSG_ID
            .... .... .... .... .... .... ..0. .... = MATCH_MSG_TOKEN: Not set
            .... .... .... .... .... .... ...0 .... = MATCH_OFFSET: Not set
            .... .... .... .... .... .... .... 0... = MATCH_MSG_SEQ_NUMBER: Not set
            .... .... .... .... .... .... .... .0.. = MATCH_GROUP_ID: Not set
            .... .... .... .... .... .... .... ..1. = MATCH_CORREL_ID: Set
            .... .... .... .... .... .... .... ...1 = MATCH_MSG_ID: Set
        mtchMsgId: 000000000000000000000000000000000000000000000000 *****COMMENT: As you can see, the MQMH Message ID is not set on the request message request.*****
        mtchCorID: 000000000000000000000000000000000000000000000000 *****COMMENT: As you can see, the MQMH Correlation ID is not set on the request message request.*****


2. MQ Server --[ASYNC_MESSAGE]--> MQ Client
Code:

WebSphere MQ [ASYNC_MESSAGE of a Full MQ Segment] Hdl=0x0002 GlbMsgIdx=1, SegIdx=0, SegLen=3864
    Transmission Segment Header
        StructID..: TSHM
        MQSegmLen.: 4332
        Convers ID: 5
        Request ID: 1
        Byte order: Big endian (0x01)
        SegmType..: ASYNC_MESSAGE (0x0d)
        Ctl Flag 1: 0x30, Last Seg, First Seg
            0... .... = DLQ used: Not set
            .0.. .... = Req accept: Not set
            ..1. .... = Last Seg: Set
            ...1 .... = First Seg: Set
            .... 0... = Close Chnl: Not set
            .... .0.. = Req close: Not set
            .... ..0. = Error: Not set
            .... ...0 = Confirm Req: Not set
        Ctl Flag 2: 0x00
            .... 0... = CommitIntvl: Not set
            .... .0.. = CSH: Not set
            .... ..0. = MSG Comp: Not set
            .... ...0 = HDR Comp: Not set
        LUW Ident.: 0000000000000000
        Encoding..:      111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
        CCSID.....:  (819)
        Reserved..: 0x0000
    ASYNC MESSAGE
        version..: 0x00000001 (1)
        handle...: 0x00000002
        MsgIndex.: 0x00000001 (1)
        GlbMsgIdx: 0x00000001 (1)
        SegLength: 0x00000f18 (3864)
        SegmIndex: 0x0000 (0)
        SeleIndex: 0x0001 (1)
        ReasonCod: MQRC_NONE (0)
        TotMsgLen: 0x00000f18 (3864)
        ActMsgLen: 0x00000f18 (3864)
        MsgToken.: 5eb5e06d000000410000000000000853
        status...: 0x0001
        resolQNLn: 12
        resolQNme: QL.REP.QUEUE
        Padding..: 00
    Message Descriptor
        StructID.: MD 
        Version..: 2
        Report...: 0
        Msg Type.: MQMT_REPLY (2)
        Expiry  .: -1
        Feedback.: 0
        Encoding.:      311-785 (FLT_S390/DEC_NORMAL/INT_NORMAL)
        CCSID....:  (500)
        Format...: MQHRF2 
        Priority.: 4
        Persist..: MQPER_NOT_PERSISTENT (0)
        Msg ID...: c3e2d840d4d8e5f14040404040404040d8532c2d5397e595
        CorrelID.: 414d51204d51444556312020202020205f2abac72131ce61
        BackoCnt.: 0
        ReplyToQ.:                                                 
        RepToQMgr: MQV1                                           
        UserId...: mqcartd     
        AccntTok.: 000000000000000000000000000000000000000000000000...
        AppIdData:                                 
        PutAppTyp: MQAT_XCF (20)
        PutAppNme: DESJD1  IMSDEA             
        PutDatGMT: 20200805
        PutTimGMT: 14492085
        AppOriDat:     
        GroupId..: 000000000000000000000000000000000000000000000000
        MsgSeqNum: 1
        Offset...: 0
        Msg flags: 0x00000000
        Orig len.: -1
    Rules and Formatting Header
        Structid: RFH
        version.: 2
        Length..: 424
        Encoding:      311-785 (FLT_S390/DEC_NORMAL/INT_NORMAL)
        CCSID...:  (500)
        Format..: MQIMS   
        Flags...: 0x00000000
        NmeValCCSID:  (1208)
        NameValue: <mcd><Msd>jms_bytes</Msd></mcd>
            Len.: 32
            Val.: *****COMMENT: phpBB doesn't accept these non-ASCII characters, so I am truncating them.*****
        NameValue: <jms><Dst>queue:///QAC.REQ.QUEUE</Dst><Rto>queue:///QA.REP.QUEUE</Rto><Tms>1596638959834</Tms><Cid>43020c70-b06c-48bd-8d01-d97233b6bd77</Cid><Dlv>1</Dlv></jms>
            Len.: 160
            Val.: *****COMMENT: phpBB doesn't accept these non-ASCII characters, so I am truncating them.*****
        NameValue: <usr><b3>cb4719cfd44917360a1319f299b6bbb3-4702e37e04f1b618-0</b3><Format>MQIMS   </Format><Report dt='i4'>0</Report><Encoding dt='i4'>273</Encoding><MsgType dt='i4'>1</MsgType></usr>
            Len.: 184
            Val.: *****COMMENT: phpBB doesn't accept these non-ASCII characters, so I am truncating them.*****
    Data (3440 bytes)
        Data: *****COMMENT: I am removing this data from the post*****
        [Length: 3440]
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Aug 06, 2020 2:06 pm    Post subject: Re: JMS selector - performance issue on deep queues Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

bef1 wrote:

We traced the requests being sent by the client application (using tcpdump and Wireshark), and we see that is seems to behave accordingly, sending the following selector when doing the OPEN operation on the response queue: JMSCorrelationID='45030c70-b06c-49bd-9d01-d97243b6bd88'

So as far as we can tell, our application complies to IBM's recommendations, but it still scales very badly with the queue depth.

-----

I would like to know if there is something else we should look into to get a better performance on (somewhat) deep queues.

Just look at the format given to your JMSCorrelationID and you'll see that it does not match the format of the string obtained through
Code:
msg.getJMSMessageID ;
// or
msg.getJMSCorrelationID();
// the correct format for the selector looks like:
// "ID:414d5120465232202020202020202020da69295f2310a130"

So no you don't conform to the IBM recommendation and each message will have to be inspected, hence your deep queue times.

Also this correlation ID looks very suspiciously like a correlation ID from a different (non MQ) Messaging System. Do you per any chance run a bridge to MQ?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
bef1
PostPosted: Thu Aug 06, 2020 3:35 pm    Post subject: Re: JMS selector - performance issue on deep queues Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

Edit: removed this post. Please refer to next post instead.

Last edited by bef1 on Thu Aug 06, 2020 6:15 pm; edited 2 times in total
Back to top
View user's profile Send private message
bef1
PostPosted: Thu Aug 06, 2020 4:41 pm    Post subject: Re: JMS selector - performance issue on deep queues Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

fjb_saper wrote:

Just look at the format given to your JMSCorrelationID and you'll see that it does not match the format of the string obtained through
Code:
msg.getJMSMessageID ;
// or
msg.getJMSCorrelationID();
// the correct format for the selector looks like:
// "ID:414d5120465232202020202020202020da69295f2310a130"

So no you don't conform to the IBM recommendation and each message will have to be inspected, hence your deep queue times.

Also this correlation ID looks very suspiciously like a correlation ID from a different (non MQ) Messaging System.


I spent some time reviewing IBM's documentation, and I think that we actually conform to IBM's specifications:

Quote:

The MQMD CorrelId field can hold a standard WebSphere MQ Correlation ID of 48 hexadecimal digits (24 bytes). The JMSCorrelationID can be a byte[] value, a string value containing hexadecimal characters and prefixed with ID:, or an arbitrary string value not beginning ID:. The first two of these represent a standard WebSphere MQ Correlation ID and map directly to or from the MQMD CorrelId field (truncated or padded with zeros as applicable); they do not use the MQRFH2 jms.Cid field. The third (arbitrary string) uses the MQRFH2 jms.Cid field; the first 24 bytes of the string, in UTF-8 format, are written into the MQMD CorrelID.


I have reviewed a TCP trace, and we are using the 3rd scenario. Here is an example from one PUT message:

RFH header:
Code:

Rules and Formatting Header
    Structid: RFH
    version.: 2
    Length..: 424
    Encoding:      111-273 (FLT_IEEE_NORMAL/DEC_NORMAL/INT_NORMAL)
    CCSID...:  (1208)
    Format..: MQIMS   
    Flags...: 0x00000000
    NmeValCCSID:  (1208)
    NameValue: <mcd><Msd>jms_bytes</Msd></mcd>
        Len.: 32
        Val.: <mcd><Msd>jms_bytes</Msd></mcd>
    NameValue: <jms><Dst>queue:///QAC.REQ.QUEUE</Dst><Rto>queue:///QA.REP.QUEUE</Rto><Tms>1596638959834</Tms><Cid>43020c70-b06c-48bd-8d01-d97233b6bd77</Cid><Dlv>1</Dlv></jms>
        Len.: 160
        Val.: <jms><Dst>queue:///QAC.REQ.QUEUE</Dst><Rto>queue:///QA.REP.ICDE</Rto><Tms>1596638959834</Tms><Cid>43020c70-b06c-48bd-8d01-d97233b6bd77</Cid><Dlv>1</Dlv></jms>   
    NameValue: <usr><b3>cb4719cfd44917360a1319f299b6bbb3-4702e37e04f1b618-0</b3><Format>MQIMS   </Format><Report dt='i4'>0</Report><Encoding dt='i4'>273</Encoding><MsgType dt='i4'>1</MsgType></usr>
        Len.: 184
        Val.: <usr><b3>cb4719cfd44917360a1319f299b6bbb3-4702e37e04f1b618-0</b3><Format>MQIMS   </Format><Report dt='i4'>0</Report><Encoding dt='i4'>273</Encoding><MsgType dt='i4'>1</MsgType></usr> 


MQMH Correlation ID:
Code:
34333032306337302d623036632d343862642d386430312d

This hex value, from MQMH header (34333032306337302d623036632d343862642d386430312d) can be converted to ASCII as follows, which corresponds to the first 24 bytes of the JMSCorrelationID: 43020c70-b06c-48bd-8d01-

So everything looks conform to the specs, as far as I can tell.

Then, the next question would be: Does this is this 3rd scenario perform as well as the first 2 (the ones that start with "ID:" and that you were expecting to see). I will quote IBM MQ's v9 developer guide on this:
Quote:

Selecting messages from a queue requires IBM MQ to sequentially inspect each message on the queue.
Messages are inspected until a message is found that matches the selection criteria or there are no more
messages to examine. Therefore, messaging performance suffers if message selection is used on deep
queues.

To optimize message selection on deep queues when selection is based on JMSCorrelationID or
JMSMessageID, use a selection string of the form JMSCorrelationID = ... or JMSMessageID
= ... and reference only one property.

This method offers a significant improvement in performance for selection on JMSCorrelationID and
offers a marginal performance improvement for JMSMessageID.


So it sounds to me like everything the client application is doing looks conform to IBM's specifications and recommandations.

Am I missing something?
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Aug 06, 2020 9:58 pm    Post subject: Re: JMS selector - performance issue on deep queues Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

bef1 wrote:
Am I missing something?


What does the actual client code that sets the JMS Correlation ID look like?

I don't see anywhere where you show that you are using the "ID:" format. That is what I think you are missing.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
bef1
PostPosted: Thu Aug 06, 2020 10:48 pm    Post subject: Re: JMS selector - performance issue on deep queues Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

hughson wrote:
I don't see anywhere where you show that you are using the "ID:" format. That is what I think you are missing.


I am not sure if you saw my previous post on this thread. Short version: I confirm that the client is not using the ID: format. It is instead using the 3rd format that the MQ developer’s guide allows, i.e. an arbitrary string value not beginning ID: [...] The third (arbitrary string) uses the MQRFH2 jms.Cid field; the first 24 bytes of the string, in UTF-8 format, are written into the MQMD CorrelID. .

If this is the root cause of our performance issue, then I only see one possible reason: MQ does not behave accordingly to IBM’s documentation, since it is not specified anywhere that this 3rd format (an arbitrary string) will lead MQ to have to parse the messages, instead of using the arbitrary string to find messages having matching MDMH correlation ID.
Back to top
View user's profile Send private message
hughson
PostPosted: Fri Aug 07, 2020 1:58 am    Post subject: Re: JMS selector - performance issue on deep queues Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

bef1 wrote:
I confirm that the client is not using the ID: format.

Can you try it out to discover whether what we say is true - that it will solve your performance problem. It was certainly added in order to achieve exactly that.

bef1 wrote:
It is instead using the 3rd format that the MQ developer's guide allows, i.e. an arbitrary string value not beginning ID: [...] The third (arbitrary string) uses the MQRFH2 jms.Cid field; the first 24 bytes of the string, in UTF-8 format, are written into the MQMD CorrelID. .

Could you provide a link to this please? Is this for MQPUT or MQGET that it is describing?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
bef1
PostPosted: Fri Aug 07, 2020 3:26 am    Post subject: Re: JMS selector - performance issue on deep queues Reply with quote

Novice

Joined: 29 Aug 2019
Posts: 15

hughson wrote:

Can you try it out to discover whether what we say is true - that it will solve your performance problem. It was certainly added in order to achieve exactly that.


Yes, I’ll try to get that done today.

hughson wrote:
Could you provide a link to this please? Is this for MQPUT or MQGET that it is describing?


It comes from this page: https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/rjc0007_.html
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » IBM MQ Java / JMS » JMS selector - performance issue on deep queues
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.