| Author |
Message
|
| chmd |
 Posted: Fri Dec 11, 2015 8:36 am Post subject: How can I check if read-ahead is actually enabled? |
 |
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
Hi,
I wrote a client application that pulls messages from a queue and discards them. I am trying to maximize the throughput (I will do something with the messages later). This client code works, but it is slow (typically 8-10 transactions per second or 160 KiB/s). I suspect that the MQOO_READ_AHEAD option is ignored. How can I check that?
If you think that MQOO_READ_AHEAD is already enabled, how can I increase the throughput? We mesured the performance of the link between the client machine and the queue manager with iperf, and found out that the link is capable of 500MiB/s, which is much more that 160kiB/s, so the network is clearly not the issue. I think that the problem is either in the MQ configuration or in my code. And since the same code, running locally on the queue manager, easily reaches 1500 transactions per second (200 times faster), I am more inclined to believe that MQ is misconfigured for what I am attempting to do.
Here is the channel I pull from:
| Code: |
1 : display channel(NOT.REAL.CHANNEL.NAME)
AMQ8414: Display Channel details.
CHANNEL(NOT.REAL.CHANNEL.NAME) CHLTYPE(SVRCONN)
ALTDATE(2015-12-04) ALTTIME(13.51.29)
COMPHDR(NONE) COMPMSG(NONE)
DESCR( ) DISCINT(0)
HBINT(300) KAINT(AUTO)
MAXINST(999999999) MAXINSTC(999999999)
MAXMSGL(4194304) MCAUSER( )
MONCHL(QMGR) RCVDATA( )
RCVEXIT( ) SCYDATA( )
SCYEXIT( ) SENDDATA( )
SENDEXIT( ) SHARECNV(10)
SSLCAUTH(REQUIRED)
SSLCIPH(TLS_RSA_WITH_AES_256_CBC_SHA)
SSLPEER( ) TRPTYPE(TCP)
|
The queue itself has been altered to have DEFREADA to YES.
| Code: |
1 : display qlocal(NOT.REAL.QUEUE.NAME)
AMQ8409: Display Queue details.
QUEUE(NOT.REAL.QUEUE.NAME) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2015-12-09)
ALTTIME(16.16.01) BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLCHNAME( )
CLWLPRTY(0) CLWLRANK(0)
CLWLUSEQ(QMGR) CRDATE(2015-06-22)
CRTIME(14.11.03) CURDEPTH(0)
CUSTOM( ) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFPRESP(SYNC) DEFREADA(YES)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR( ) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(1)
MAXDEPTH(50000) MAXMSGL(4194304)
MONQ(QMGR) MSGDLVSQ(PRIORITY)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS(8) PROCESS( )
PUT(ENABLED) PROPCTL(COMPAT)
QDEPTHHI(80) QDEPTHLO(20)
QDPHIEV(DISABLED) QDPLOEV(DISABLED)
QDPMAXEV(ENABLED) QSVCIEV(NONE)
QSVCINT(999999999) RETINTVL(999999999)
SCOPE(QMGR) SHARE
STATQ(QMGR) TRIGDATA( )
TRIGDPTH(1) TRIGMPRI(0)
TRIGTYPE(FIRST) USAGE(NORMAL)
|
We made sure that this queue only gets non persistent messages. I had doubts so recently I checked by myself and only found non persistent messages:
| Code: |
>/opt/mqm/samp/bin/amqsbcg NOT.REAL.QUEUE.NAME NOT.REAL.QMGR.NAME | grep Persistence
Priority : 0 Persistence : 0
Priority : 0 Persistence : 0
Priority : 0 Persistence : 0
...
|
The queue manager is running MQ 7.5 and runs on a Red Hat server.
The client code runs on another machine (VM on debian). I am using pymqi, compiled against latest version of mqclient (8.0.0.4-WS-MQC-Redist-LinuxX64.tar.gz). It connects over SSL (certificates have been configured and work just fine).
In the client code, the queue is open with the following options:
| Code: |
CMQC.MQOO_INPUT_AS_Q_DEF |
CMQC.MQOO_READ_AHEAD
|
If you need more information, please let me know!
Thank you,
Christophe-Marie |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Dec 11, 2015 2:11 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Christophe Marie, mon ami, tu devrais aussi utiliser dans tes OpenOptions
MQOO et MQGMO la petite perle, oh combien importante: FAIL_IF_QUIESCE...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Dec 11, 2015 4:02 pm Post subject: Re: How can I check if read-ahead is actually enabled? |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| chmd wrote: |
Here is the channel I pull from:
| Quote: |
1 : display channel(NOT.REAL.CHANNEL.NAME)
AMQ8414: Display Channel details.
CHANNEL(NOT.REAL.CHANNEL.NAME) CHLTYPE(SVRCONN)
ALTDATE(2015-12-04) ALTTIME(13.51.29)
COMPHDR(NONE) COMPMSG(NONE)
DESCR( ) DISCINT(0)
HBINT(300) KAINT(AUTO)
MAXINST(999999999) MAXINSTC(999999999)
MAXMSGL(4194304) MCAUSER( )
MONCHL(QMGR) RCVDATA( )
RCVEXIT( ) SCYDATA( )
SCYEXIT( ) SENDDATA( )
SENDEXIT( ) SHARECNV(10)
SSLCAUTH(REQUIRED)
SSLCIPH(TLS_RSA_WITH_AES_256_CBC_SHA)
SSLPEER( ) TRPTYPE(TCP) |
|
What is the SHARECNV value for the CLNTCONN?
I ask because:
| Quote: |
| High SHARECNV limits have the advantage of reducing queue manager thread usage. However, if many conversations sharing a socket are all busy, there is a possibility of delays as the conversations contend with one another to use the receiving thread. In this situation, a lower SHARECNV value is better. |
If the app is faster when run locally (still client-bindings) to a nearby qmgr, and slower when the qmgr is miles or thousands of miles away, then network contention is still a possibility.
Désolé , mon français est nul.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 11, 2015 8:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Note that the OP only showed the MQOO options. The READ_AHEAD might also be needed in the MQGMO options for it to happen.
And as advised please change the sharecnv to 1 (best practice on MQ8).
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tczielke |
| Posted: Sat Dec 12, 2015 7:37 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
When you OPEN the queue with MQOO_READ_AHEAD, there are some conditions that need to be met for it to be enabled -> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q027330_.htm?lang=en
| Quote: |
•Both the client and remote queue manager must be at WebSphere® MQ Version 7 or later.
•The client application must be compiled and linked against the threaded IBM MQ MQI client libraries.
•The client channel must be using TCP/IP protocol
•The channel must have a non-zero SharingConversations (SHARECNV) setting in both the client and server channel definitions.
|
When you compiled your client code on Linux, did you link with mqic or mqic_r? Note the requirement that you should be linking against mqic_r.
One simple test you could do is remove the MQOO_READ_AHEAD option and see if your performance changes for the worse. If the performance does not really change, you probably are not using read ahead. I did some tests, and using read ahead made some very significant and noticeable performance improvements (the overall elapsed time went from around 72 seconds to 10 seconds with read ahead enabled).
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| chmd |
| Posted: Mon Dec 14, 2015 4:52 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
Thank you for all the answers!
| Quote: |
Christophe Marie, mon ami, tu devrais aussi utiliser dans tes OpenOptions
MQOO et MQGMO la petite perle, oh combien importante: FAIL_IF_QUIESCE...
|
My open options are:
| Code: |
CMQC.MQGMO_NO_WAIT | CMQC.MQGMO_FAIL_IF_QUIESCING
|
| Quote: |
The READ_AHEAD might also be needed in the MQGMO options for it to happen.
|
I just tried, but it does not seem to be a valid get message option (as in: the code crashes).
| Quote: |
If the app is faster when run locally (still client-bindings) to a nearby qmgr, and slower when the qmgr is miles or thousands of miles away, then network contention is still a possibility.
|
I agree. The machines are physically very close to each other, so at first I did not consider the possibility. Then when running the code on the queue manager vs on this machine we had this problem, that was my first clue. So to be sure, we actually measured: we installed iperf on the queue manager and confirmed that we were geting 500MiB/s (!) to this very client machine.
| Quote: |
And as advised please change the sharecnv to 1 (best practice on MQ8).
|
I did as you asked. No noticeable change. Note that when running on the queue manager, I am using a different channel with no ssl encryption. The value of SHARECNV is 10 on that one. I also considered that the slowness could be blamed on the encryption, but the messages coming to the queue are coming through a different SSL encrypted channel, which has no performance problem whatsoever. It could be the cipher, though (I will check).
| Quote: |
When you compiled your client code on Linux, did you link with mqic or mqic_r? Note the requirement that you should be linking against mqic_r.
|
It is definitely linked to mqic_r. When I alter LD_LIBRARY_PATH and remove /opt/mqm/lib64 from there, the program crashes with
| Code: |
ImportError: libmqic_r.so: cannot open shared object file: No such file or directory
|
As for the other criterias: the qmgr version is 7.5, the client lib is 8.0.0.4-WS-MQC-Redist-LinuxX64; The connection is TCP; SHARECNV is non-zero on the queue manager.
| Quote: |
What is the SHARECNV value for the CLNTCONN?
|
Good question! I have no idea how to check that. I am using pymqi and I found no information in the source code of the lib telling me if there was a default setting in the lib. Do you know how does one specify that in a C/C++ client? The python binding is no more than a fine wrapper around the C API, so I should be able to derive from a sample.[/list] |
|
| Back to top |
|
|
| tczielke |
| Posted: Mon Dec 14, 2015 6:14 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
I am not sure if you have the MQ Client installed on your client server. It sounds like you might not since you mention the redistributable client. I think would need the MQ Client software installed to run a client trace (i.e. strmqtrc). If you do have the MQ Client, it looks like you can validate if read ahead is enabled or not by doing the following tracing on the MQ Client side:
#turn tracing on for the client application myprog
strmqtrc -t all -p myprog
Run your program. Your traces be default on Linux will go to /var/mqm/trace. You need to format your traces with dspmqtrc. In the formatted trace for your client program, search for the text "ahead".
This is what I saw in the trace for a queue I was opening when MQOO_READ_AHEAD was set:
| Code: |
07:29:04.101929 3101.1 RSESS:000001 Requested read-ahead=0x00100000
07:29:04.101934 3101.1 RSESS:000001 Requested Maximum Q size=100KB
|
This is what I saw in the trace for a queue I was opening when MQOO_READ_AHEAD was NOT set and the queue had DEFREADA(NO):
| Code: |
07:36:39.597749 14101.1 RSESS:000001 Read-ahead taken from object def [DefReadAhead=0x00000000]
07:36:39.597755 14101.1 RSESS:000001 Requested read-ahead=0x00080000
07:36:39.597760 14101.1 RSESS:000001 Requested Maximum Q size=100KB
|
Here are the hex values for MQOO_ options of read ahead:
0x00080000 MQOO_NO_READ_AHEAD
0x00100000 MQOO_READ_AHEAD
You can see in the above examples that read ahead resolved to 0x00100000 MQOO_READ_AHEAD when read ahead was set in the open options and it resolved to 0x00080000 MQOO_NO_READ_AHEAD when read ahead was not set in the open options and the queue had DEFREADA(NO).
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| tczielke |
| Posted: Mon Dec 14, 2015 6:40 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
You can also use the Application Activity Trace to find the negotiated SharingConversations for your channel instance. You need to have TraceLevel set to high for the activity trace. After you get your activity trace (you could use amqsact), if you look for the MQCD Structure for your CONNX of your channel and look near the bottom, you will see something like the following:
| Code: |
00000750: 0000 0032 0000 0011 FFFF FFFD 0000 000A '...2............' <- SharingConversations starts at offset 75C for 4 bytes and is 10 = 0x0A
00000760: 0000 0000 3B9A C9FF 3B9A C9FF 0000 0000 '....;...;.......'
00000770: 0000 0001 0000 1388 0000 0002 0000 0000 '................'
00000780: 2020 2020 2020 2020 2020 2020 2020 2020 ' ' <- CertificateLabel starts at offset 780 (NOTE: edited and was initially incorrect at offset 770)
00000790: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000007A0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
000007B0: 2020 2020 2020 2020 2020 2020 2020 2020 ' '
|
CertificateLabel should be at the bottom of the MQCD structure. SharingConversations should be 3 lines above the CertificateLabel and the last 4 byte on that third line.
_________________
Working with MQ since 2010.
Last edited by tczielke on Mon Dec 14, 2015 8:31 am; edited 1 time in total |
|
| Back to top |
|
|
| chmd |
| Posted: Mon Dec 14, 2015 7:22 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
I do have the strmqtrc program on my client machine.
I tried to run
| Code: |
| strmqtrc -t all -p mq2kafka #does not do anything with kafka yet |
However, /var/mqm/trace was not populated after running my program.Since this is a python program calling the lib, I also tried
| Code: |
| strmqtrc -t all -p python |
with no luck.
However, we did confirm that read_ahead *can* work with this installation, by running "q" on the client machine. We created a test queue TEST.Q with all default options, and a test channel with all default options too.
Synchronous put:
| Code: |
time ./q -oTEST.Q -lmqic -M'#!1000/-5000'
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.
real 0m21.905s
user 0m0.035s
sys 0m0.033s
|
Synchronous get:
| Code: |
time ./q -ITEST.Q -lmqic -q
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.
No more messages.
real 0m31.980s
user 0m0.038s
sys 0m0.036s
|
Asynchronous put:
| Code: |
time ./q -af -oTEST.Q -lmqic -M'#!1000/-5000'
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.
real 0m0.586s
user 0m0.003s
sys 0m0.005s
|
Asynchronous get:
| Code: |
time ./q -aa -ITEST.Q -lmqic -q
MQSeries Q Program by Paul Clarke [ V6.0.0 Build:May 2 2012 ]
Connecting ...connected to 'NOT.REAL.QMGR.NAME'.
No more messages.
real 0m1.985s
user 0m0.014s
sys 0m0.010s
|
Now that we have established that the client installation and the server installation can work in read_ahead mode, I am rewriting my python code to do exactly the same as q, on this very test queue, with this very channel. I will try to narrow it down to very simple code that I can hopefully paste here. |
|
| Back to top |
|
|
| tczielke |
| Posted: Mon Dec 14, 2015 7:25 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
If you do the following, it will trace any client program:
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| chmd |
| Posted: Mon Dec 14, 2015 8:27 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
All right!
So it turns out that the traces do not go to /var/mqm. In my case, they go to a directory IBM/MQ/data/trace that is created in the current directory.
I cleaned it and rerun the program, then formatted all the trace files. Only one of them showed infos:
| Code: |
16:09:32.440594 20302.1 RSESS:000001 Read-ahead taken from object def [DefReadAhead=0x00000000]
16:09:32.440597 20302.1 RSESS:000001 Requested read-ahead=0x00080000
16:09:32.440599 20302.1 RSESS:000001 Requested Maximum Q size=100KB
|
Then all the subsequent matches are
| Code: |
16:09:32.851279 20302.1 RSESS:000001 Requested read-ahead=0x00100000
16:09:32.851282 20302.1 RSESS:000001 Requested Maximum Q size=100KB
|
Something weird is going on. I will make sure I only pull from one queue, because right now my program is a bit more complicated than necessary. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Dec 14, 2015 9:17 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
This channel isn't being used by other apps, is it? I'd recommend using test channels, as well as test queues, while you do your debugging.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| chmd |
| Posted: Mon Dec 14, 2015 1:14 pm Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
| No, it is a brand new channel specially created for that purpose. |
|
| Back to top |
|
|
| tczielke |
| Posted: Mon Dec 14, 2015 1:28 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
This manual document also lists the MQGMO options that are not allowed for read ahead -> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q026320_.htm?lang=en
One that caught my eye was this one:
MQGMO_SYNCPOINT
"If these options are specified on the first MQGET call then read ahead is disabled. If these options are specified on a subsequent MQGET call a reason code MQRC_OPTIONS_ERROR is returned."
You had mentioned earlier about transactions. Are you using MQGMO_SYNCPOINT for your GETs?
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| chmd |
| Posted: Tue Dec 15, 2015 2:21 am Post subject: |
|
|
Novice
Joined: 11 Dec 2015
Posts: 18
|
No, my get options are
| Code: |
gmo.Options = CMQC.MQGMO_NO_WAIT | \
CMQC.MQGMO_FAIL_IF_QUIESCING
|
|
|
| Back to top |
|
|
|
|
