| Author |
Message
|
| daveeason |
 Posted: Mon Dec 11, 2006 7:22 pm Post subject: Using Segmentation with MQGet Node question |
 |
|
Novice
Joined: 14 Jul 2005
Posts: 18
Location: Canberra, Australia
|
This is a design question relating to the MQGet node provided in WMBv6,
Is it possible to use MQ segmentation with the MQGet node?
I have a small flow that is collating (batching) several messages that are all part of one logical batch. What I would like to do is use the MQ facility of message segmentation in the following manner;
1) MQInput (receive individual seperate batch msgs)
2) Compute (assign MQMF_SEGEMENT parms to each msg and write to temporary queue)
-- Then when entire batch has been received.
3) MQGet (using MQGMO_COMPLETE_MESSAGE to return the entire batch as a single message, read from temp queue)
4) MQOutput (single message collated)
Is this possible, I have not had any success at the moment and I am unsure whether MQGet can be used with the MQGMO_COMPLETE_MESSAGE option set? Has anyone tried to do this, can somebody recommend the appropriate use of the MQMF_ flags, or MQMD or MQGMO settings. Any assistance would be appreciated.
_________________
Dave Eason
Addis |
|
| Back to top |
|
 |
| elvis_gn |
| Posted: Tue Dec 12, 2006 12:37 am Post subject: Re: Using Segmentation with MQGet Node question |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi daveeason,
| daveeason wrote: |
1) MQInput (receive individual seperate batch msgs)
2) Compute (assign MQMF_SEGEMENT parms to each msg and write to temporary queue)
-- Then when entire batch has been received.
3) MQGet (using MQGMO_COMPLETE_MESSAGE to return the entire batch as a single message, read from temp queue)
4) MQOutput (single message collated) |
You received 1 message from the MQInput...
You coded for segmentation on that 1 message and put it into the temp queue...
You did the same for all the messages that you received from MQInput...
Now when you read back using MQGet node, it will only pick one message at a time, and that will be the merger of the 1st segmented message, which is nothing but the message you had received first...
Either I have not got what you are actually trying to do here, or you have not understood what segmentation actually is...
Regards. |
|
| Back to top |
|
|
| EddieA |
| Posted: Tue Dec 12, 2006 9:21 am Post subject: |
|
|
Jedi
Joined: 28 Jun 2001
Posts: 2453
Location: Los Angeles
|
Have you tried reading the messages, between steps 2 and 3, with an MQ application that does handle segmentation, to validate you set it up correctly in step 2.
Cheers,
_________________
Eddie Atherton
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
| daveeason |
| Posted: Tue Dec 12, 2006 2:16 pm Post subject: |
|
|
Novice
Joined: 14 Jul 2005
Posts: 18
Location: Canberra, Australia
|
Thanks elvis_gn, hmm it could be that I don't understand what segmentation actually is, I have been reading the infocenter but have not found some of the aspects particularly clear.
My understanding is, that there is Queue manager controlled segmentation (that is not what I am referring to here) and there is application controlled segmentation. I was under the impression that I should be able to use application controlled segmentation to specify each incoming message as a 'sub-segment' of the entire batch before writing it to a temporary queue. Then once the whole batch is received I would use the MQGet node with the MQGMO_COMPLETE_MESSAGE option specified to return all 'sub-segments' as one complete message.
Is it possible to do that using MQ segmentation (application controlled?)
EddieA, thanks for the suggestion I will look into it. Might take me some time to set up a stand alone MQ application.
_________________
Dave Eason
Addis |
|
| Back to top |
|
|
| TonyD |
| Posted: Tue Dec 12, 2006 2:17 pm Post subject: |
|
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
Dave,
I did some research on this question a year ago and seem to remember that I got a response from WMB Support to the effect that the MQGet node does not suport the functionality that you require. Unfortunately I cannot find the response; possibly someone from IBM could confirm whether this is correct. I just tried a test , similar to that suggested by EddieA, which uses an MQInput node to read a segmented message. That worked OK, but an MQGet set to read the same message left the five segments sitting on the queue, even though I specified 'MQGMO_LOCICAL_ORDER' and 'MQGMO_ALL_SEGMENTS_AVAILABLE' which are the properties that are set in the successful MQInput.
I will continue to hunt for the response from IBM. |
|
| Back to top |
|
|
| TonyD |
| Posted: Tue Dec 12, 2006 2:58 pm Post subject: |
|
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
Referring back to your initial question, logical grouping might be applicable rather than segmentation. If you have a set of messages arriving that you can identify as members of a group (batch) and you can tell when the last one has arrived, you can write them as a logical group to your temp queue and then read the group, when it is complete, in a second flow.
You need to set MQMD.GroupId, MQMD.MsgSeqNumber and MQMD.MsgFlags in your output messages to the temp queue (this example was processing a set of employee records (EmpCnt and NumofEmps):
| Code: |
SET OutputRoot.MQMD.GroupId = CAST(X'313131313131313132323232323232323333333333333333' AS BLOB);
SET OutputRoot.MQMD.MsgSeqNumber = EmpCnt;
SET OutputRoot.MQMD.MsgFlags = 8;
IF EmpCnt = NumOfEmps THEN
SET OutputRoot.MQMD.MsgFlags = 24;
END IF;
|
:and check the 'Logical Order' property in the MQInput node of the second flow. Depending upon your application's characteristics the second flow may need to test MQMD.MsgFlags to determine when it has read the last message in the group., e.g.:
| Code: |
IF Root.MQMD.MsgFlags = 24 THEN ...
|
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Dec 12, 2006 4:08 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If you use Logical Grouping, you will have to wire or code a loop around the MQGet in order to ensure you get the entire group.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| daveeason |
| Posted: Tue Dec 12, 2006 6:36 pm Post subject: |
|
|
Novice
Joined: 14 Jul 2005
Posts: 18
Location: Canberra, Australia
|
Jeff -> Thankyou 
Tony,
Yes the grouping solution you have suggested would work but I think that due to the fact that the batch may potentially contain thousands of records (individual messages that need to be collated) then grouping will require many Get's whereas I would prefer (if possible) to use MQ Segmentation to just do one get and retrieve the entire completed batch as one message.
You mentioned earlier that you were able to use an MQInput node to read in a segmented message just by specifying 'MQGMO_LOGICAL_ORDER' and 'MQGMO_ALL_SEGMENTS_AVAILBLE'? Are you referring to the 'Advanced > Logical Order and All Messages Available' properties on the MQInput node. I have been trying to replicate what you said but I just keep reading in an empty message, where you able to read a complete message (does the MQInput node use MQGMO_COMPLETE_MESSAGE ?) or have I misunderstood your previous point?
Thanks in advance you guys.
_________________
Dave Eason
Addis |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 13, 2006 4:21 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Segment the batch up front with a home grown program that will cut a message for each individual transaction in the batch. On the flow output keep the segmentation info(group info) and add an output to a special queue for statistics and feedback.
Treat then each message as the individual transaction it is... 
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| TonyD |
| Posted: Wed Dec 13, 2006 7:05 pm Post subject: |
|
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
Dave,
The 'MQGMO_' options were my attempt to get the MQGet node to read the segments (but it did not work!). In the Advanced Properties of the MQInput tick 'Logical Order' and 'All Messages Available'.
'MQGMO_COMPLETE_MESSAGE' is the default behaviour of the MQInput node and cannot be altered. However this bit of information might be useful; it comes from WMB Support:
| Quote: |
The setting of MQGMO_COMPLETE_MSG on the MQInput nodes MQGET is dependent on the Transaction mode setting in the input node. If the transaction mode is set to Yes (causes MQGMO_SYNCPOINT) or No (causes MQGMO_NO_SYNCPOINT) then MQGMO_COMPLETE_MSG is set. However if the Transaction mode is set to Automatic, then MQGMO_COMPLETE_MSG is NOT set. In this case we issue MQGMO_SYNCPOINT_IF_PERSISTENT and since the segments on the queue could be a mixture of persistent and non persistent messages we retrieve them as individual messages. |
If you cannot get your test to work I can package up my sample and send it to you; it is two very simple flows only. |
|
| Back to top |
|
|
| daveeason |
| Posted: Wed Dec 20, 2006 5:09 pm Post subject: |
|
|
Novice
Joined: 14 Jul 2005
Posts: 18
Location: Canberra, Australia
|
Thanks Tony,
OK so with Tony's help I now have this working thanks heaps mate.
I thought I would post what my solution was incase others were interested.
The key to this for me was a few things 1) Yes the MQInput node uses MQGMO_COMPLETE_MESSAGE, 2) In order to get this working I needed to specify the MQMD.Offset property for each segment to indicate the positioning of each segment in the final message and 3) All messages within a single batch share the same MQMD.MsgSeqNumber. Once those to pieces of the puzzle were complete the solution worked as follows;
I used two seperate message flows, the first to receive the seperate messages and assign the appropriate segmentation parameters to each of them and the second to read the complete batch as one single message then continue processing;
Message flow (1) receive msg and perform the following steps;
a) define as part of a segment (MQMD.MsgFlags = MQMF_SEGMENT),
b) specify sequence number, (MQMD.MsgSeqNumber),
c) I then set the CorrelId, MsgId and GroupId (appropriate to my solution),
d) set the segment offset (MQMD.Offset)
e) for the last message of the batch only, set MQMD.MsgFlags = MQMF_LAST_SEGMENT
Message flow (2) Get completed batch message.
Note: MQInput node has the following properties set;
Transaction Mode = 'Yes' (can use No but not Automatic),
Logical Order 'checked',
All Message available 'checked'.
That was pretty much it. Once again thanks TonyD for your assistance and all others who contributed.
Regards,
Dave
_________________
Dave Eason
Addis |
|
| Back to top |
|
|
| wbintegrator |
| Posted: Sun Jun 24, 2007 3:34 am Post subject: |
|
|
Voyager
Joined: 08 Feb 2006
Posts: 83
|
Hi all,
This post was really helpful to me in regard with making the MQInput get segmented message - I was missing the 'Automatic' thing about transactions, so thanks a lot
However, I have a very weird problem with this thing: every time the flow "eats" a segmented message it needs to be stopped and started before the MQInput eats another segmented message. I mean, you can put messages into the queue and nothings happens, like the flow is stuck or something.
All the messages I'm referring to are proper segments with the flags,offsets etc properly set by a c program.
Has anybody stumbled over a similar problem? Any suggestions?
Much thanks for help |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jun 24, 2007 7:02 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Coud it be that at the end of the logical message you are forgetting to clear correlationId and other get parameter ??
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| wbintegrator |
| Posted: Sun Jun 24, 2007 10:55 pm Post subject: |
|
|
Voyager
Joined: 08 Feb 2006
Posts: 83
|
Sorry, I didn't understood your question.
I'm using MQInput, so I'm not setting CorrelId and other get parameters. |
|
| Back to top |
|
|
|
|
