| Author |
Message
|
| chmd |
 Posted: Wed Dec 16, 2015 6:26 am Post subject: |
 |
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
| bruce2359 wrote: |
| chmd wrote: |
This code is still not faster when used with the production queue, though. If anything, the option makes the code slower. |
What option? Please be a bit more precise when you post. |
Sorry. I am talking about the option CMQC.MQOO_READ_AHEAD. You can see in my run that I get a very slightly better performance (13 TPS vs 15 TPS) on my production queue when I am *not* using it, which is counter intuitive. On the other hand, when I use the same program on another queue, I get a very good performance with CMQC.MQOO_READ_AHEAD (300 TPS vs 20 TPS).
| fjb_saper wrote: |
Don't build the MQMD and the GMO each time around in the loop.
Build it once outside, and assign the value to the actual field in each loop iteration...  |
No, this is actually needed. If I reuse md and gmo, then I get this bug that I can pull only one message from the queue, and I exit with the error code 2033 while the queue is in fact not empty. You do need to use a different message descriptor every time if you don't want to face this bug. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Dec 16, 2015 6:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You need to clear the msgid and correlid on the mqmd... or make sure you don't match on either.
The code object/structure is reusable.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 16, 2015 7:35 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| chmd wrote: |
Sorry. I am talking about the option CMQC.MQOO_READ_AHEAD. You can see in my run that I get a very slightly better performance (13 TPS vs 15 TPS) on my production queue when I am *not* using it, which is counter intuitive. On the other hand, when I use the same program on another queue, I get a very good performance with CMQC.MQOO_READ_AHEAD (300 TPS vs 20 TPS). |
Are the message that are put to the two queues the exact same messages? Exact same message length? Same data content?
Something must be different that causes the different timing. If the queue attributes are identical, and the exact same application is consuming the messages, what is different?
Does the same application produce the messages that end up in the two queues?
Have you embedded a wait or timer of any kind in your application?
Is there a database involved in your application?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 16, 2015 11:05 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| chmd wrote: |
| fjb_saper wrote: |
Don't build the MQMD and the GMO each time around in the loop.
Build it once outside, and assign the value to the actual field in each loop iteration... |
No, this is actually needed. If I reuse md and gmo, then I get this bug that I can pull only one message from the queue, and I exit with the error code 2033 while the queue is in fact not empty. You do need to use a different message descriptor every time if you don't want to face this bug. |
| mqjeff wrote: |
You need to clear the msgid and correlid on the mqmd... or make sure you don't match on either.
The code object/structure is reusable. |
So you need to declare / define the structures twice.
Once for initialization (think of them as constants) and once for use inside the loop. Then at each iteration you initialize the value of the loop field with that of the corresponding constant field. It may shave off a few cycles off your loop.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 16, 2015 11:54 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| fjb_saper wrote: |
So you need to declare / define the structures twice.
Once for initialization (think of them as constants) and once for use inside the loop. Then at each iteration you initialize the value of the loop field with that of the corresponding constant field. It may shave off a few cycles off your loop. |
This is a best programming practice - instantiate an MQMD to use as a template for resetting its field values to factory (initial) values, and another MQMD to use for MQGETs and MQPUTs.
Values in the MQMD are NOT defaults - they are not automatically reset to factory values after use. The Application Programming Ref. manual correctly describes fields in MQMD and other structures as initial values - and not defaults.
Before an MQGET or MQPUT, copy the MQMD template over the MQMD your app used for MQGET/MQPUT.
The size of the MQMD is a few hundred bytes. The number of processor cycles to copy the MQMD template over the recently-used MQMD is insignificant.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| chmd |
| Posted: Fri Dec 18, 2015 3:05 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
All right, I just did an interesting test: I copied about 26000 messages from my production queue to a file, then to my test queue. Now I get the same problem on the test queue!
The thing is, I checked all of them, and they are all non persistent!
| Code: |
1 : dis qstatus(TEST.Q) all
AMQ8450: Display queue status details.
QUEUE(TEST.Q) TYPE(QUEUE)
CURDEPTH(26770) IPPROCS(0)
LGETDATE( ) LGETTIME( )
LPUTDATE( ) LPUTTIME( )
MEDIALOG( ) MONQ(OFF)
MSGAGE( ) OPPROCS(0)
QTIME( , ) UNCOM(NO)
|
| Code: |
$ amqsbcg TEST.Q QMGR|grep "Persistence : 0" | wc -l
26770
|
However, my code is super slow when dealing with those:
| Code: |
time QUEUE_NAME=TEST.Q READ_AHEAD=1 ./read_ahead_test
Starting.
Using read_ahead
TEST.Q: Got 41 messages (8.2 tps, 122.3 KiB/s, 14.9 KiB/msg)
TEST.Q: Got 47 messages (9.4 tps, 155.6 KiB/s, 16.6 KiB/msg)
TEST.Q: Got 44 messages (8.8 tps, 144.8 KiB/s, 16.5 KiB/msg)
|
The same code gets 230 TPS when dealing with test messages on the same size, loaded with the q utility like this:
| Code: |
q -af -oTEST.Q -lmqic -M'#!50000/15360'
|
It is definitely not a problem of the queue, but really with the messages. This is weird, because the ony thing that is required should be non-persistence... |
|
| Back to top |
|
|
| chmd |
| Posted: Fri Dec 18, 2015 5:21 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
Ok, I compared some test messages with what I get from my prod queue. The only difference that seems significant is the encoding!
Production message:
| Code: |
Encoding : 273 CodedCharSetId : 819
|
Test message:
| Code: |
Encoding : 546 CodedCharSetId : 819
|
Maybe read ahead was working all along? Maybe there is some conversion that I did not explicitly request that takes place and screws up the performance? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 18, 2015 5:47 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
So you have the same CCSID but a different encoding... What is the FORMAT on the MQMD of your messages as shown by the browse tool?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| chmd |
| Posted: Fri Dec 18, 2015 7:07 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
| fjb_saper wrote: |
| So you have the same CCSID but a different encoding... What is the FORMAT on the MQMD of your messages as shown by the browse tool? |
I get
Now, what I did: In my dump file, I modified this format to
(8 spaces)with a search and replace, and reloaded the messages in the queue. Then I used my program again to pull the messages.
The program does not run any faster, which means that changing the Format to None will not help  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 18, 2015 7:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Are you reading messages at the same time as you are writing them - from whatever you are using instead of qload?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Dec 18, 2015 8:14 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Exactly what platform (o/s, hardware) is the message created on?
Exactly what platform (o/s, hardware) is the qmgr?
Exactly what platform (o/s, hardware) is the message consumer?
Are any exits involved?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| chmd |
| Posted: Mon Dec 21, 2015 1:56 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
| mqjeff wrote: |
| Are you reading messages at the same time as you are writing them - from whatever you are using instead of qload? |
No, just reading. I do not touch the production queue anymore now that I can reproduce the problem on the test queue. When loaded with test messages (15KiB-30KiB messages generated with the 'q' utility), the queue reads at 130-200 TPS. When reading messages from my production batch (about 15KiB on average), it reads at 8-10TPS.
| bruce2359 wrote: |
Exactly what platform (o/s, hardware) is the message created on?
Exactly what platform (o/s, hardware) is the qmgr?
Exactly what platform (o/s, hardware) is the message consumer?
Are any exits involved? |
- The qmgr is running on a Red Hat Enterprise Linux Server release 6.6.
- The consumer is running on a Debian 7.9. It runs exactly this code.
- I don't know what you mean by 'exits'. As you can see if you read the gist I linked, the code drops the messages in order to avoid IO related problems and focus on MQ performance.
As for the hardware: Both are virtual machines, with plenty of resources. I don't have more relevant information, but I guarantee that there is no issue with it: we extensively checked this aspect and made sure no other VM was stealing resources. |
|
| Back to top |
|
|
| chmd |
| Posted: Mon Dec 21, 2015 2:08 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
For the record, I am adding all the meta info I can have about the messages.
Production (slow)
| Code: |
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 273 CodedCharSetId : 819
Format : 'MQSTR '
Priority : 0 Persistence : 0
MsgId : X'(removed)'
CorrelId : X'(removed)'
BackoutCount : 0
ReplyToQ : '(removed)'
ReplyToQMgr : '(removed)'
** Identity Context
UserIdentifier : '(removed)'
AccountingToken :
X'0000000000000000000000000000000000000000000000000000000000000000'
ApplIdentityData : ' '
** Origin Context
PutApplType : '6'
PutApplName : '(removed) '
PutDate : '20151218' PutTime : '10292976'
ApplOriginData : ' '
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset : '0'
MsgFlags : '0'
OriginalLength : '-1'
**** Message ****
[...]
|
Test (fast)
| Code: |
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 546 CodedCharSetId : 819
Format : 'MQSTR '
Priority : 0 Persistence : 0
MsgId : X'(removed)'
CorrelId : X'(removed)'
BackoutCount : 0
ReplyToQ : '(removed) '
ReplyToQMgr : '(removed)'
** Identity Context
UserIdentifier : '(removed) '
AccountingToken :
X'0000000000000000000000000000000000000000000000000000000000000000'
ApplIdentityData : ' '
** Origin Context
PutApplType : '6'
PutApplName : 'q'
PutDate : '20151218' PutTime : '12375004'
ApplOriginData : ' '
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset : '0'
MsgFlags : '0'
OriginalLength : '-1'
**** Message ****
[...]
|
|
|
| Back to top |
|
|
| chmd |
| Posted: Mon Dec 21, 2015 7:18 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
All right, seems like the issue is unrelated to encoding in the end.
I tried 2 more things:
- Changing the Encoding metadata in the dumpfile to 546 and retrying: still slow
- Downloading the messages with q instead of my own program: It takes only 1m18 to dowload all the 26919 messages -> 340 TPS!
So there is a bug in my own code that makes fetching those messages super slow, and I have no idea what it is. |
|
| Back to top |
|
|
| jasonn |
| Posted: Thu Dec 29, 2022 1:30 am Post subject: |
|
|
Newbie
Joined: 22 Dec 2022
Posts: 6
|
To increase the throughput, you should first check if the MQOO_READ_AHEAD option is enabled. To do this, you can use the following MQSC command:
DISPLAY QLOCAL(NOT.REAL.QUEUE.NAME)
The output should include a line with "DEFREADA(YES)" which indicates that the read ahead option is enabled.
If the option is not enabled, you can enable it by issuing the following MQSC command:
ALTER QLOCAL(NOT.REAL.QUEUE.NAME) DEFREADA(YES)
If the option is already enabled, you should check if there are any other bottlenecks that could be causing the slow throughput. This could include network latency, lack of resources on the queue manager (such as memory or CPU) or a slow disk subsystem.
You can also optimize the code to reduce the overhead of the MQ calls, making sure that unnecessary calls are not made, and that large chunks of data are transferred in a single call rather than multiple calls. Finally, you can increase the number of threads that the client application uses to process messages.
EDITED: To remove the url buried in one of the display commands... |
|
| Back to top |
|
|
|
|
