| Author |
Message
|
| andres |
 Posted: Fri Apr 12, 2013 6:16 am Post subject: best practices on MQ Java Client |
 |
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Hi all,
Im doing a bulk process from iSeries to MQ.
IM using the Java Client for V7.1. I build a daemon in iSeries that get data from iSeries data queues and send to MQ.
So basically, i fill the iSeries data queue with around 20.000 msg and the java daemon will read one by one those entries and send them to MQ. I will do this process several times in a day.
ACtually i open and close connection for every message.
Could be better to open a connection and close after every bulk of msgs? |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Fri Apr 12, 2013 7:02 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
It doesn't hurt to reset the conx once in a while, especially when processing bulk messages. But you should make it a configurable parameter so you can tune how many messages pass before you reset conx.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Apr 12, 2013 7:33 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Yes, keep the connection open, keep the queue open. Absoutely no need to repeat these for each message. Terrible overhead.
You only need to close the queue and connection when you have reached the end of the messages (i.e. the MQGET returns RC 2033).
This is really basic MQI stuff.
You might want to use syncpointing to improve throughput and commit the messages in batches (e.g. every 200).
Your current design will commit every message individually as well as waste resources performing the connection/open/close/disconnect each time. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Apr 12, 2013 7:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You should connect to MQ as few times as possible.
A better approach overall is to leave your Java daemon running all the time, and react as quickly as it can to new messages in the data queue. Then it can maintain a connection object (or more than one) in a persistent environment.
This also allows time on the other side for the MQ applications to react immediately, rather than having to plow through messages on a deep queue. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Apr 12, 2013 7:48 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Agree, and that means using MQGET with MQGMO_WAIT and a suitable interval.
Don't forget FAIL_IF_QUIESCING options. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Apr 12, 2013 8:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
Agree, and that means using MQGET with MQGMO_WAIT and a suitable interval.
Don't forget FAIL_IF_QUIESCING options. |
The app in question is only doing PUTS, not GETs. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Apr 12, 2013 10:49 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| mqjeff wrote: |
| react as quickly as it can to new messages in the data queue. |
So what does this phrase mean, if not a MQGET?
I guess you meant to include the word "put". |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Apr 13, 2013 4:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
| mqjeff wrote: |
| react as quickly as it can to new messages in the data queue. |
So what does this phrase mean, if not a MQGET?
I guess you meant to include the word "put". |
andres is building a bridging application between an iSeries data queue and a WebSphere MQ queue.
Different things.
No MQGET involved. |
|
| Back to top |
|
|
| andres |
| Posted: Mon Apr 15, 2013 12:33 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Thanks all.
Like mqjeff say, im using the Java Client on iSeries.
In a loop (that keeps waiting for some msg to come into a data queue) i use:
Loop (wait for data queue)
MQMessage MQmsg = new MQMessage();
MQmsg.format = MQC.MQFMT_STRING;
MQmsg.characterSet = 1208;
MQPutMessageOptions pmo = new MQPutMessageOptions();
MQmsg.writeString(msg);
queue.put(MQmsg, pmo);
queue.close();
queueManager.disconnect();
endloop
I could just probably open the connection just when i start the daemon, write to the MQ while reading the data queue.
I wonder how to stop the daemon closing the connection to the queue and the manager. And if stop and start daemon again, for how long will be active the first one? |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Apr 15, 2013 1:41 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Unless I am wrong about this - just setting the CCSID to 1208 will not change the data into UTF-8.
So if your data queue is presenting EBCDIC data, you need to set the MQMD code page to 500, 037, 285 or a suitable value, so that MQGMO_CONVERT (done by the recipient) will work properly.
The end of the loop should immediately be after the MQPUT. Do not repeat the queue close and QM disconnect for each message.
You will need to decide when and how to stop this task (daemon or whatever term you use) if it waits for more data in a loop. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Mon Apr 15, 2013 3:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| zpat wrote: |
| Unless I am wrong about this - just setting the CCSID to 1208 will not change the data into UTF-8. |
It depends on which Java write method the application uses: writeString() converts a String to a byte sequence, using the given characterSet. |
|
| Back to top |
|
|
| andres |
| Posted: Tue Apr 16, 2013 6:05 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Hi felas,
i just did some testing,comparing different approaches:
1. Send 20.000 msgs: open/send/close queue...
2. Send 20.000 msgs: open/send/send/send.. and close every 100 msgs
3. Send 20.000 msgs: open/send 20.000 / close.
First time took 5-6 minutes (test enviroment) and more CPU in iSeries.
Second took 2-3 minutes and lest CPU.
in this case, could i cause soem troubles in others queues in the same QM?
But i got exception reason code 2017, because the maximin handles of que QM is 256.
Could it be a good idea to increase that value? |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Apr 16, 2013 6:19 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
No, just follow normal good practice. Do not connect to the queue manager more than once. Disconnect for each connection made.
Absolutely no need to keep closing/opening the queue. Why are you doing this?
I will say this only ONCE more.
ONE CONNECT
ONE OPEN
MANY PUTS (optionally with SYNCPOINT, and then a COMMIT).
ONE CLOSE
ONE DISCONNECT
If you want to optimise throughput - use MQPMO_SYNCPOINT and then issue MQCOMMIT every 200 (or so) messages. You could close and re-open the queue to commit, but this ONLY makes it commit at that time if you have coded SYNCPOINT on the put (otherwise every put is separately commited immediately when issued) |
|
| Back to top |
|
|
| andres |
| Posted: Tue Apr 16, 2013 7:11 am Post subject: |
|
|
Apprentice
Joined: 12 Apr 2013
Posts: 27
|
Well, the program is a daemon, so it´s hard to say when i can do the commit or not, that´s because i tested open/send/close.
i could process 10 msgs or 20.000, depends of the application. So it´s hard to say when a bunch of msgs will finish or not, and this is related how the dataques on iSeries works ( i cant know that the last msgs is already the last message to do a commit or disconnect).
I´ll try to figure this out and use open/commit.
thanks for your help |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Apr 16, 2013 7:19 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| andres wrote: |
Well, the program is a daemon, so it´s hard to say when i can do the commit or not, that´s because i tested open/send/close.
i could process 10 msgs or 20.000, depends of the application. So it´s hard to say when a bunch of msgs will finish or not, and this is related how the dataques on iSeries works ( i cant know that the last msgs is already the last message to do a commit or disconnect).
I´ll try to figure this out and use open/commit.
thanks for your help |
Committing after n messages or t time if n messages not reached? If the stuff on the iSeries data-queue only occurs during known hours-of-service you could also put in a commit prior to the daemon shutting down (if it's done any uncommitted puts), which should be after the cessation of hours-of-service.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
|
|
