| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| messageId and java |
« View previous topic :: View next topic » |
| Author |
Message
|
| meriton |
 Posted: Wed Aug 07, 2002 9:16 pm Post subject: messageId and java |
 |
|
Novice
Joined: 07 Aug 2002
Posts: 24
|
Hi All,
I am working on a module where there is a requirement to create a flat file after reading messages from a MQQueue.
I have tried following method,
Browse the queue and keep the mesages in a vector.After getting all the message for a batch(its part of message),connect the queue get the message from the queue( with option MQGMO_NO_SYNCPOINT) after comparing the message in vector with the message On queue,to make sure I get the message of the right batch.
This method works If the batch size is small (<20,000),for large batch of (>20,000),adding large number of message strings to vector generates run time out of memory exception.
Following are my Questions.
1.What are the options of doing the above requirement?
2.How can I store message id in a vector instead of whole message,then get the right message from the queue by checking the message id from vector.
The problem I am facing here is the message ids are stored in byte[],converting it to a int or integer is a pain.
Any help is appreciated.
[/quote] |
|
| Back to top |
|
 |
| RogerLacroix |
| Posted: Thu Aug 08, 2002 8:55 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
Ok, I've gotta ask, "What are you trying to accomplish?" (what are the requirements?)
There has to be a better way of doing it. e.g. Message Grouping, Browsing the queue, etc...
Please provide more details of what you are trying to accomplish.
later
Roger Lacroix
Enterprise Architect
Capitalware Inc.
http://www.capitalware.biz
----------------------------------------
IBM Certified Specialist - MQSeries
IBM Certified Developer - MQSeries
IBM Certified Solutions Expert - MQSeries
---------------------------------------- |
|
| Back to top |
|
|
| meriton |
| Posted: Thu Aug 08, 2002 9:25 pm Post subject: |
|
|
Novice
Joined: 07 Aug 2002
Posts: 24
|
Hi Roger,
Here is our requirement in brief.
1)We have got two queues,control queue & reply queue.
2)For each batch One control message is sent to control queue and corresponding data MQmessages(can be any number) are being sent to a reply queue from mainframe,there is a identifier within the message text for batch id(for both data and control message).
3)A java collation program is written which is invoked by mq trigger on control queue.
4)The collation program reads the message from control queue,grabs batchid from the message text and browses(MQOO_BROWSE) data messages .All the matched data message for a batch id is stored in a vector.
5)The size of the vector is same as the number of message expected by control message(noOfDataRecs flag withing control message),collation program connects to quue(MQGMO_NO_SYNCPOINT) gets all message that are present in the vector and writes to a flat file.
PROBLEM:
The above design works for small batches (<15000) for large batches (>15,000)adding to vector,causes out of memry run time exception.
TIA |
|
| Back to top |
|
|
| bduncan |
| Posted: Fri Aug 09, 2002 7:35 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Oh man... I would strongly recommend using grouping. Imagine doing 20,000 browses just to figure out what messages to retrieve, and then doing 20,000 MQGETs matching against MsgId! I can see why you are having problems with large batch sizes. If you use groups, MQSeries will logically associate a given number of messages. Here are some of the advantages:
1) They all have the same GroupId, which you can match against when retrieving the messages
2) You can specify MQGET options such that the group won't even be retreivable from the queue until ALL messages for that group have arrived (this removes the need for all the browses)
3) You can specify MQGET options such that the messages in the group are given to you in order, such that you don't need to match against MsgId, etc., you just say, "Gimme the next message in the group."
As you can see, this will provide huge performance increases for you.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Fri Aug 09, 2002 8:37 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
Definitely, Message Grouping is the way to go for your requirements. But the only problem is that Message Grouping is only available on the most recent version of MQ for OS/390 (mainframe).
Therefore, if you have an older version of MQ on OS/390 and are not planning on upgrading in a while then I would change your program logic.
A little Java threading can go a long way.
You should change the program to do the following:
Mainline:- read a message from the "control queue" to get the "batch id"
- loop
- spin-off a new thread with the "batch id"
- read a message from the "control queue" to get the "batch id" - end-loop
- exit
Thread:- i=0;
- loop while i < noOfDataRecs
- Get (not browse) matching message from the data queue
- Write it to the file
- i++ -endloop
- thread dies
later
Roger... |
|
| Back to top |
|
|
| meriton |
| Posted: Sun Aug 11, 2002 3:16 pm Post subject: |
|
|
Novice
Joined: 07 Aug 2002
Posts: 24
|
Thanks All,
Roger Thats what we are doing at present,the problem occurs(out of memory) when you store message in a vector (loop 1) so that you can match message and do a GET (in loop 2).
Correct me I am missed something from your post.
TIA |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Sun Aug 11, 2002 8:36 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
Don't store it in a vector - don't store anything at all.
When the mainline gets the "identifier" from a message in the control queue, start the thread and pass into it the "identifier". The thread browses the "data" queue, for the matching message by identifier, if a match is found then immediately do a destructive get and write it to the file.
The resources used by the mainline will be minimal. The resources used the thread should be low. The search time should be fine since the data queue will be constantly emptied by the worker threads. Your only overhead will be the starting and stoping of the threads.
Also, the mainline should be able to always keep the control queue at a zero depth. And the worker threads should be able to do a good job of keeping the current depth of the data queue to a low number (maybe even zero).
later
Roger... |
|
| Back to top |
|
|
| meriton |
| Posted: Wed Jul 09, 2003 3:59 pm Post subject: How to be sure aal messages arrived. |
|
|
Novice
Joined: 07 Aug 2002
Posts: 24
|
Hi Roger,
How this new thread would be sure that all data messages for a batch have been arrived at the queue before doing a destructive get.
TIA |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
