| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| How can I check if read-ahead is actually enabled? |
« View previous topic :: View next topic » |
| Author |
Message
|
| bruce2359 |
 Posted: Tue Dec 15, 2015 3:44 am Post subject: |
 |
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
Try again with MQGMO_WAIT. There were some performance issues with clients doing read-ahead with MQGMO_NO_WAIT in v7.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
 |
| chmd |
| Posted: Tue Dec 15, 2015 5:56 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
I managed to have READ_AHEAD working on a small example! I uploaded the script as a github gist for everyone to see it. It is only 138 lines, so please consider having a look. The problem is: it works fine with test messages, but it does not work on my production queue. Here are 2 examples with a test queue, where I put 5000 messages beforehand each time.
With READ_AHEAD:
| Code: |
$ time QUEUE_NAME=TEST.Q READ_AHEAD=1 ./read_ahead_test
read_ahead_test:: Starting.
read_ahead_test:: TEST.Q: Got 1800 messages (360.0 tps, 885.1 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 1598 messages (319.6 tps, 772.1 KiB/s, 2.4 KiB/msg)
read_ahead_test:: Got a total of 5000 messages
Traceback (most recent call last):
File "./read_ahead_test", line 138, in <module>
main()
File "./read_ahead_test", line 127, in main
message = queue.get(None, md, gmo)
File "/home/mqclient/venv/local/lib/python2.7/site-packages/pymqi/__init__.py", line 1743, in get
raise MQMIError(rv[-2], rv[-1])
pymqi.MQMIError: MQI Error. Comp: 2, Reason 2033: FAILED: MQRC_NO_MSG_AVAILABLE
real 0m15.476s
user 0m0.836s
sys 0m0.146s
|
Without:
| Code: |
$ time QUEUE_NAME=TEST.Q ./read_ahead_test
read_ahead_test:: Starting.
read_ahead_test:: TEST.Q: Got 113 messages (22.6 tps, 60.3 KiB/s, 2.7 KiB/msg)
read_ahead_test:: TEST.Q: Got 119 messages (23.8 tps, 59.3 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 126 messages (25.2 tps, 62.3 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 121 messages (24.2 tps, 55.8 KiB/s, 2.3 KiB/msg)
read_ahead_test:: TEST.Q: Got 130 messages (26.0 tps, 71.5 KiB/s, 2.7 KiB/msg)
read_ahead_test:: TEST.Q: Got 115 messages (23.0 tps, 54.1 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 127 messages (25.4 tps, 58.4 KiB/s, 2.3 KiB/msg)
read_ahead_test:: TEST.Q: Got 123 messages (24.6 tps, 60.6 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 114 messages (22.8 tps, 56.0 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 123 messages (24.6 tps, 60.2 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 127 messages (25.4 tps, 59.8 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 126 messages (25.2 tps, 59.5 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 122 messages (24.4 tps, 59.6 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 117 messages (23.4 tps, 59.1 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 124 messages (24.8 tps, 62.5 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 120 messages (24.0 tps, 57.1 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 124 messages (24.8 tps, 57.3 KiB/s, 2.3 KiB/msg)
read_ahead_test:: TEST.Q: Got 117 messages (23.4 tps, 54.4 KiB/s, 2.3 KiB/msg)
read_ahead_test:: TEST.Q: Got 115 messages (23.0 tps, 61.4 KiB/s, 2.7 KiB/msg)
read_ahead_test:: TEST.Q: Got 131 messages (26.2 tps, 62.6 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 120 messages (24.0 tps, 62.2 KiB/s, 2.6 KiB/msg)
read_ahead_test:: TEST.Q: Got 129 messages (25.8 tps, 62.4 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 120 messages (24.0 tps, 55.5 KiB/s, 2.3 KiB/msg)
read_ahead_test:: TEST.Q: Got 112 messages (22.4 tps, 49.4 KiB/s, 2.2 KiB/msg)
read_ahead_test:: TEST.Q: Got 124 messages (24.8 tps, 61.9 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 109 messages (21.8 tps, 54.4 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 120 messages (24.0 tps, 60.7 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 131 messages (26.2 tps, 59.2 KiB/s, 2.3 KiB/msg)
read_ahead_test:: TEST.Q: Got 118 messages (23.6 tps, 63.4 KiB/s, 2.7 KiB/msg)
read_ahead_test:: TEST.Q: Got 127 messages (25.4 tps, 61.1 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 122 messages (24.4 tps, 58.7 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 131 messages (26.2 tps, 65.0 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 119 messages (23.8 tps, 57.1 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 114 messages (22.8 tps, 52.7 KiB/s, 2.3 KiB/msg)
read_ahead_test:: TEST.Q: Got 126 messages (25.2 tps, 59.9 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 116 messages (23.2 tps, 58.8 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 134 messages (26.8 tps, 63.2 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 110 messages (22.0 tps, 58.1 KiB/s, 2.6 KiB/msg)
read_ahead_test:: TEST.Q: Got 121 messages (24.2 tps, 62.2 KiB/s, 2.6 KiB/msg)
read_ahead_test:: TEST.Q: Got 123 messages (24.6 tps, 54.4 KiB/s, 2.2 KiB/msg)
read_ahead_test:: TEST.Q: Got 132 messages (26.4 tps, 61.8 KiB/s, 2.3 KiB/msg)
read_ahead_test:: Got a total of 5000 messages
Traceback (most recent call last):
File "./read_ahead_test", line 138, in <module>
main()
File "./read_ahead_test", line 127, in main
message = queue.get(None, md, gmo)
File "/home/mqclient/venv/local/lib/python2.7/site-packages/pymqi/__init__.py", line 1743, in get
raise MQMIError(rv[-2], rv[-1])
pymqi.MQMIError: MQI Error. Comp: 2, Reason 2033: FAILED: MQRC_NO_MSG_AVAILABLE
real 3m25.416s
user 0m2.062s
sys 0m0.388s
|
Now on my production queue:
| Code: |
1 : display qstatus(PROD.Q)
AMQ8450: Display queue status details.
QUEUE(PROD.Q) TYPE(QUEUE)
CURDEPTH(4223) IPPROCS(0)
LGETDATE( ) LGETTIME( )
LPUTDATE( ) LPUTTIME( )
MEDIALOG( ) MONQ(OFF)
MSGAGE( ) OPPROCS(8)
QTIME( , ) UNCOM(NO)
|
| Code: |
$ time QUEUE_NAME=PROD.Q ./read_ahead_test
read_ahead_test:: Starting.
read_ahead_test:: Got a total of 1 messages
Traceback (most recent call last):
File "./read_ahead_test", line 138, in <module>
main()
File "./read_ahead_test", line 127, in main
message = queue.get(None, md, gmo)
File "/home/mqclient/venv/local/lib/python2.7/site-packages/pymqi/__init__.py", line 1743, in get
raise MQMIError(rv[-2], rv[-1])
pymqi.MQMIError: MQI Error. Comp: 2, Reason 2033: FAILED: MQRC_NO_MSG_AVAILABLE
real 0m1.066s
user 0m0.165s
sys 0m0.009s
|
So it seems that when pulling out of the production queue, this code exits after one message. I really only change the queue, and I showed you that the queue had 4223 messages when I started the program. There is no other program pulling from there. Channel, encryption, all other parameters stay the same across the runs. At this point, I don't know what is the problem. You can see that I don't do anything with the messages (appart counting the number of bytes for statistics purpose). And the error is purely mq related (see the message), not due to some parsing of the messages.
I can't do that. Well I did at some point, but it was a bad idea. In my "real" code, I need to cycle between queues because I am pulling from a bunch of them. I don't want to wait on an empty queue that some process puts some message on it. What I do is that I catch the exception, and that is the signal that I should move to another queue. The other signal is a timeout has been reached. I get a very good performance with this approach when running the program on the queue manager directly (1500+ TPS). |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Dec 15, 2015 6:41 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Is there a particular reason your program doesn't use more than one thread, each reading from a dedicated queue?
That would allow you to do GET WITH WAIT, and ensure that messages being processed from queue1 don't interfere with messages being processed from queue2.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| chmd |
| Posted: Tue Dec 15, 2015 7:18 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
No reason apart from the fact I tend to only use threads whenever strictly necessary because they are harder to debug. So I just tried changing to MQGMO_WAIT, because if it solves my problem, I am happy to use threads, but it does not change anything. Same results with my test queue, the queue is read until we get:
| Code: |
pymqi.MQMIError: MQI Error. Comp: 2, Reason 2033: FAILED: MQRC_NO_MSG_AVAILABLE
|
And same results also with my production queue, the queue reads one message, and I get this error messages even though the queue is full of messages. Here are the (updated) setups of the queues. The same channel is used in both cases, so I think we can rule that out.
| Code: |
1 : display qlocal(PROD.Q) all
AMQ8409: Display Queue details.
QUEUE(PROD.Q) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2015-12-15)
ALTTIME(12.03.39) BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLCHNAME( )
CLWLPRTY(0) CLWLRANK(0)
CLWLUSEQ(QMGR) CRDATE(2015-06-22)
CRTIME(14.11.03) CURDEPTH(0)
CUSTOM( ) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DEFREADA(NO)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR( ) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(1)
MAXDEPTH(50000) MAXMSGL(4194304)
MONQ(QMGR) MSGDLVSQ(PRIORITY)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS(8) PROCESS( )
PUT(ENABLED) PROPCTL(COMPAT)
QDEPTHHI(80) QDEPTHLO(20)
QDPHIEV(DISABLED) QDPLOEV(DISABLED)
QDPMAXEV(ENABLED) QSVCIEV(NONE)
QSVCINT(999999999) RETINTVL(999999999)
SCOPE(QMGR) SHARE
STATQ(QMGR) TRIGDATA( )
TRIGDPTH(1) TRIGMPRI(0)
TRIGTYPE(FIRST) USAGE(NORMAL)
|
| Code: |
1 : display qlocal(TEST.Q) all
AMQ8409: Display Queue details.
QUEUE(TEST.Q) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2015-12-14)
ALTTIME(13.21.08) BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLCHNAME( )
CLWLPRTY(0) CLWLRANK(0)
CLWLUSEQ(QMGR) CRDATE(2015-12-14)
CRTIME(13.21.08) CURDEPTH(0)
CUSTOM( ) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DEFREADA(NO)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR( ) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(0)
MAXDEPTH(5000) MAXMSGL(4194304)
MONQ(QMGR) MSGDLVSQ(PRIORITY)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS(0) PROCESS( )
PUT(ENABLED) PROPCTL(COMPAT)
QDEPTHHI(80) QDEPTHLO(20)
QDPHIEV(DISABLED) QDPLOEV(DISABLED)
QDPMAXEV(ENABLED) QSVCIEV(NONE)
QSVCINT(999999999) RETINTVL(999999999)
SCOPE(QMGR) SHARE
STATQ(QMGR) TRIGDATA( )
TRIGDPTH(1) TRIGMPRI(0)
TRIGTYPE(FIRST) USAGE(NORMAL)
|
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Dec 15, 2015 7:53 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
See your test queue set up
Change it to YES and see what happens. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 15, 2015 9:19 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| chmd wrote: |
And same results also with my production queue, the queue reads one message, and I get this error messages even though the queue is full of messages. Here are the (updated) setups of the queues. The same channel is used in both cases, so I think we can rule that out.
| Quote: |
1 : display qlocal(PROD.Q) all
AMQ8409: Display Queue details.
QUEUE(PROD.Q) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2015-12-15)
ALTTIME(12.03.39) BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLCHNAME( )
CLWLPRTY(0) CLWLRANK(0)
CLWLUSEQ(QMGR) CRDATE(2015-06-22)
CRTIME(14.11.03) CURDEPTH(0)
CUSTOM( ) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DEFREADA(NO)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR( ) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(1)
MAXDEPTH(50000) MAXMSGL(4194304)
MONQ(QMGR) MSGDLVSQ(PRIORITY)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS( PROCESS( )
PUT(ENABLED) PROPCTL(COMPAT)
QDEPTHHI(80) QDEPTHLO(20)
QDPHIEV(DISABLED) QDPLOEV(DISABLED)
QDPMAXEV(ENABLED) QSVCIEV(NONE)
QSVCINT(999999999) RETINTVL(999999999)
SCOPE(QMGR) SHARE
STATQ(QMGR) TRIGDATA( )
TRIGDPTH(1) TRIGMPRI(0)
TRIGTYPE(FIRST) USAGE(NORMAL)
|
| Quote: |
1 : display qlocal(TEST.Q) all
AMQ8409: Display Queue details.
QUEUE(TEST.Q) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2015-12-14)
ALTTIME(13.21.0 BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLCHNAME( )
CLWLPRTY(0) CLWLRANK(0)
CLWLUSEQ(QMGR) CRDATE(2015-12-14)
CRTIME(13.21.0 CURDEPTH(0)
CUSTOM( ) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DEFREADA(NO)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR( ) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(0)
MAXDEPTH(5000) MAXMSGL(4194304)
MONQ(QMGR) MSGDLVSQ(PRIORITY)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS(0) PROCESS( )
PUT(ENABLED) PROPCTL(COMPAT)
QDEPTHHI(80) QDEPTHLO(20)
QDPHIEV(DISABLED) QDPLOEV(DISABLED)
QDPMAXEV(ENABLED) QSVCIEV(NONE)
QSVCINT(999999999) RETINTVL(999999999)
SCOPE(QMGR) SHARE
STATQ(QMGR) TRIGDATA( )
TRIGDPTH(1) TRIGMPRI(0)
TRIGTYPE(FIRST) USAGE(NORMAL)
|
|
With CURDEPTH(0), ReasonCode 2033 is appropriate - no message available.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| chmd |
| Posted: Tue Dec 15, 2015 9:37 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
| bruce2359 wrote: |
With CURDEPTH(0), ReasonCode 2033 is appropriate - no message available. |
I ran this command after I emptied the queues. When my code was running the queues were full.
| fjb_saper wrote: |
See your test queue set up
Change it to YES and see what happens. |
The only effect it has is that the queue is now always open with READ_AHEAD, while before the client was deciding:
| Code: |
AMQ8409: Display Queue details.
QUEUE(TEST.Q) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2015-12-15)
ALTTIME(17.19.15) BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLCHNAME( )
CLWLPRTY(0) CLWLRANK(0)
CLWLUSEQ(QMGR) CRDATE(2015-12-14)
CRTIME(13.21.08) CURDEPTH(5000)
CUSTOM( ) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DEFREADA(YES)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR( ) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(0)
MAXDEPTH(5000) MAXMSGL(4194304)
MONQ(QMGR) MSGDLVSQ(PRIORITY)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS(0) PROCESS( )
PUT(ENABLED) PROPCTL(COMPAT)
QDEPTHHI(80) QDEPTHLO(20)
QDPHIEV(DISABLED) QDPLOEV(DISABLED)
QDPMAXEV(ENABLED) QSVCIEV(NONE)
QSVCINT(999999999) RETINTVL(999999999)
SCOPE(QMGR) SHARE
STATQ(QMGR) TRIGDATA( )
TRIGDPTH(1) TRIGMPRI(0)
TRIGTYPE(FIRST) USAGE(NORMAL)
|
Then
| Code: |
$ time QUEUE_NAME=TEST.Q READ_AHEAD=1 ./read_ahead_test
read_ahead_test:: Starting.
read_ahead_test:: TEST.Q: Got 1660 messages (332.0 tps, 815.6 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 1535 messages (307.0 tps, 748.8 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 1559 messages (311.8 tps, 760.9 KiB/s, 2.4 KiB/msg)
read_ahead_test:: Got a total of 5000 messages
Traceback (most recent call last):
File "./read_ahead_test", line 138, in <module>
main()
File "./read_ahead_test", line 127, in main
message = queue.get(None, md, gmo)
File "/home/mqclient/venv/local/lib/python2.7/site-packages/pymqi/__init__.py", line 1743, in get
raise MQMIError(rv[-2], rv[-1])
pymqi.MQMIError: MQI Error. Comp: 2, Reason 2033: FAILED: MQRC_NO_MSG_AVAILABLE
real 0m16.864s
user 0m0.840s
sys 0m0.145s
|
or
| Code: |
$ unset READ_AHEAD
$ time QUEUE_NAME=TEST.Q ./read_ahead_test
read_ahead_test:: Starting.
read_ahead_test:: TEST.Q: Got 1419 messages (283.8 tps, 695.9 KiB/s, 2.5 KiB/msg)
read_ahead_test:: TEST.Q: Got 1658 messages (331.6 tps, 806.2 KiB/s, 2.4 KiB/msg)
read_ahead_test:: TEST.Q: Got 1592 messages (318.4 tps, 781.3 KiB/s, 2.5 KiB/msg)
read_ahead_test:: Got a total of 5000 messages
Traceback (most recent call last):
File "./read_ahead_test", line 138, in <module>
main()
File "./read_ahead_test", line 127, in main
message = queue.get(None, md, gmo)
File "/home/mqclient/venv/local/lib/python2.7/site-packages/pymqi/__init__.py", line 1743, in get
raise MQMIError(rv[-2], rv[-1])
pymqi.MQMIError: MQI Error. Comp: 2, Reason 2033: FAILED: MQRC_NO_MSG_AVAILABLE
real 0m15.505s
user 0m0.851s
sys 0m0.150s
|
On the production queue, it has no effect. I still pull only one message while the queue has plenty more. |
|
| Back to top |
|
|
| chmd |
| Posted: Tue Dec 15, 2015 9:39 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
Edit: quoted myself instead of editing my message.
I wanted to add that it probably has to do with CMQC.MQOO_INPUT_AS_Q_DEF which now gets the READ_AHEAD as part of the queue definition settings. Sadly it has no effect on the production queue  |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 15, 2015 9:47 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| chmd wrote: |
Edit: quoted myself instead of editing my message.
I wanted to add that it probably has to do with CMQC.MQOO_INPUT_AS_Q_DEF which now gets the READ_AHEAD as part of the queue definition settings. Sadly it has no effect on the production queue |
No. MQOO_INPUT_AS_Q_DEF determines SHARED or EXCLUSIVE as set at the queue def.
MQOO_READ-AHEAD-AS-Q-DEF determines read-ahead as set at the queue def.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| tczielke |
| Posted: Tue Dec 15, 2015 11:05 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
One other clarification. It is really the absensce of both of these bit flags in the option options that causes MQOO_READ_AHEAD_AS_Q_DEF to be in effect:
#define MQOO_READ_AHEAD 0x00100000
#define MQOO_NO_READ_AHEAD 0x00080000
MQOO_READ_AHEAD_AS_Q_DEF is defined as the following (it sets no bit flags in the options), and is really only useful for documentation purposes, when you do NOT specify either MQOO_READ_AHEAD or MQOO_NO_READ_AHEAD in the open options.
#define MQOO_READ_AHEAD_AS_Q_DEF 0x00000000
I personally find this approach of flags that map to nothing misleading, and would prefer only flags that explicitly set a bit were defined.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 15, 2015 11:19 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
..._AS_Q_DEF are handy in situations where the administrator takes responsibility for determining PERSISTENCE or PRIORITY or INPUT_EXCLUSIVE or INPUT_SHARED, or in this case READ_AHEAD.
I dislike ..._AS_Q_DEF for precisely that reason. When the business requirement dictates PERSISTENT messages, I will explicitly code for PERSISTENT messages, and not use PERSISTENCE_AS_Q_DEF.
As this post demonstrates, doing problem-determination becomes far more complicated when the app depends on queue definitions to dictate app behaviors or behaviours. (In full disclosure, I must admit that I do enjoy helping someone else work through this type of problem, rather than me or my client having the problem.)
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| chmd |
| Posted: Wed Dec 16, 2015 4:50 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
Ok, I found what caused my code to exit after one message when pulling out of the production queue. I moved the lines
| Code: |
md = pymqi.MD()
gmo = pymqi.GMO()
gmo.Options = CMQC.MQGMO_WAIT | \
CMQC.MQGMO_FAIL_IF_QUIESCING
|
Inside the loop, such that a new message descriptor and new get message options are created every time. For performance reason, I thought I could reuse the object, but it turns out to be a bad idea. The updated code sits at https://gist.github.com/chmduquesne/5d730dcf297d064abde5
This code is still not faster when used with the production queue, though. If anything, the option makes the code slower.
| Code: |
$ time QUEUE_NAME=1A.INCOMING.PUP.L ./read_ahead_test
Starting.
Not using read_ahead
1A.INCOMING.PUP.L: Got 58 messages (11.6 tps, 223.2 KiB/s, 19.2 KiB/msg)
1A.INCOMING.PUP.L: Got 67 messages (13.4 tps, 223.9 KiB/s, 16.7 KiB/msg)
1A.INCOMING.PUP.L: Got 67 messages (13.4 tps, 228.3 KiB/s, 17.0 KiB/msg)
1A.INCOMING.PUP.L: Got 84 messages (16.8 tps, 225.8 KiB/s, 13.4 KiB/msg)
1A.INCOMING.PUP.L: Got 82 messages (16.4 tps, 230.6 KiB/s, 14.1 KiB/msg)
^CGot a total of 409 messages
Traceback (most recent call last):
File "./read_ahead_test", line 134, in <module>
main()
File "./read_ahead_test", line 120, in main
message = queue.get(None, md, gmo)
File "/home/mqclient/venv/local/lib/python2.7/site-packages/pymqi/__init__.py", line 1726, in get
mDesc.pack(), getOpts.pack(), length)
KeyboardInterrupt
real 0m28.861s
user 0m0.467s
sys 0m0.060s
$ time QUEUE_NAME=1A.INCOMING.PUP.L READ_AHEAD=1 ./read_ahead_test
Starting.
Using read_ahead
1A.INCOMING.PUP.L: Got 68 messages (13.6 tps, 157.5 KiB/s, 11.6 KiB/msg)
1A.INCOMING.PUP.L: Got 51 messages (10.2 tps, 159.8 KiB/s, 15.7 KiB/msg)
1A.INCOMING.PUP.L: Got 45 messages (9.0 tps, 176.0 KiB/s, 19.6 KiB/msg)
1A.INCOMING.PUP.L: Got 46 messages (9.2 tps, 165.5 KiB/s, 18.0 KiB/msg)
1A.INCOMING.PUP.L: Got 46 messages (9.2 tps, 149.4 KiB/s, 16.2 KiB/msg)
^CGot a total of 270 messages
Traceback (most recent call last):
File "./read_ahead_test", line 134, in <module>
main()
File "./read_ahead_test", line 120, in main
message = queue.get(None, md, gmo)
File "/home/mqclient/venv/local/lib/python2.7/site-packages/pymqi/__init__.py", line 1750, in get
mDesc.pack(), getOpts.pack(), length)
KeyboardInterrupt
real 0m26.907s
user 0m0.423s
sys 0m0.089s
|
|
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 16, 2015 5:21 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| chmd wrote: |
This code is still not faster when used with the production queue, though. If anything, the option makes the code slower. |
What option? Please be a bit more precise when you post.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 16, 2015 5:53 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
Don't build the MQMD and the GMO each time around in the loop.
Build it once outside, and assign the value to the actual field in each loop iteration...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Dec 16, 2015 5:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I forget if there's a supplied C sample that does read ahead... If there is, make sure to try that with the same setup (machine, conninfo, etc etc etc).
Also check how you are committing messages. It's possible (and again, I forget) that committing each message means you only get read ahead on the first message...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
