| Author |
Message
|
| vk |
 Posted: Fri Mar 17, 2006 4:14 pm Post subject: Aggregate Request Reply for Large Messages |
 |
|
Partisan
Joined: 20 Sep 2005
Posts: 302
Location: Houston
|
We have a message flow in WMQI v2.1 involving 3 message sets (Source, GBO and Destination) and 6 subflows. There is good amount of transformations and some database access. Recently in production, we faced memory allocation issues in the broker machine, when the flow was processing messages as big as 3 MB.
To avoid such issues in the future we are thinking of splitting the messages into smaller units, processing them individually and then concatenating the transformed messages back before sending to the destination application.
For this, is Aggregate Request\Reply a good solution? An aggregate request flow will split the messages and send the request to the existing flow. The existing flow will process the individual requests and send them to the reply queue. The aggregate reply flow will concatenate the replies and send them to the destination.
More than one messages can be sent from the source simultaneously. I tested a sample flow and the splitting and concatenating is working fine. But will I run into more issues with this design? Or is it better to use the segmentation and grouping facilities available in MQ and WMQI.
Thanks in advance,
VK. |
|
| Back to top |
|
 |
| elvis_gn |
| Posted: Sun Mar 19, 2006 7:37 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi vk,
Who proposed this solution ? I am not saying it is wrong, but instead of first trying to find an easier solution why are you going for a total revamp...especially when u r into production....
| Quote: |
| faced memory allocation issues |
Did you try to increase the memory....Hav you found out whether 3MB is the max size they would send or tomorrow they could increase it further....3MB is not toooo huge a message too...it is big, agreed.
| Quote: |
| is it better to use the segmentation and grouping facilities |
Dosen't this sound simpler....you won't have to change a lot of stuff....
This should also be a decision of your lead/architect/manager....someone senior....
Regards. |
|
| Back to top |
|
|
| vk |
| Posted: Mon Mar 20, 2006 4:21 am Post subject: |
|
|
Partisan
Joined: 20 Sep 2005
Posts: 302
Location: Houston
|
Thanks Elvis.
Memory is already very high. We are having around 16 GB RAM.
By moving to Aggregate Request Reply, I only have to come up with the Request flow which will split the message and give it to my existing flow, and the reply flow which will combine all the transformed messages sent by the existing flow.
The amount of change in the existing flow is very minimal - only copying the MsgId of the individual messages to the CorrelId, before putting into the Request queue.
Here are some of the statistics I got while testing based on this solution - A 2.3 MB message took 43 minutes to process in Production. With my current design, the processing time for the same message became 2 minutes 40 secs.
I dont know whether, this is the best solution, but I am definitely going to give it some more try. Maybe we can use the same Request and Reply flows in the future for any interface, provided, there is some means to specify the Aggregation Name, Timeout values and Folder Names dynamically.
One more question - If I combine the Request and Reply flows into a single flow, will the entire scenario be transactional? Will the entire message be rolled back (message put by the source application), if there is an error in the Reply flow or if all the replies do not come within the Timeout?
The reason for asking the same is, if the entire message is rolled back, then we have an option for correcting and resending the messages from the backout queue. Otherwise, we will have to ask the source application to send the message again. This might be an issue, when the message goes to the backout queue for system related issues and the incoming data is perfectly fine.
Meanwhile, we are also trying out segmentation and grouping at MQ level. Once we get the statistics there, we will make a decision on which route to take. But this definitely seems to involve more changes in existing code.
Thanks for you help.
VK. |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Mon Mar 20, 2006 4:32 am Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi vk,
| Quote: |
| Memory is already very high. We are having around 16 GB RAM. |
I am hoping this is the system RAM you are talking about  I am not talking about the same....Search this forum or memory heap size...you can increase the JVM of the Broker Execution groups....that should imporve the performance and maybe fix ur current problem...
About the transaction issue....U can prevent automatic commit in the nodes and thus have a rollback....
Regards. |
|
| Back to top |
|
|
| nathanw |
| Posted: Tue Mar 21, 2006 12:14 am Post subject: |
|
|
Knight
Joined: 14 Jul 2004
Posts: 550
|
Would also add that Aggregation in 2.1 was not as succesful as V5
But if you do decide to go this route let me know as i have had quite a bit of exposure to aggregation recently |
|
| Back to top |
|
|
| vk |
| Posted: Tue Mar 21, 2006 11:23 am Post subject: |
|
|
Partisan
Joined: 20 Sep 2005
Posts: 302
Location: Houston
|
Thanks Nathan.
I am not facing any issues in development, with the solution that I developed. Everything is working fine and I am seeing a tremendous improvement in processing time. I tried the same with another interface and there too I am seeing a remarkable improvement.
My only worry is whether we will run into unseen issues, if we move the same to Production. One case I can think of is broker going down after the control message has been sent by the request flow. Will the timeout information be retained when the broker comes up? Otherwise, all the valid replies will be rejected by the reply flow and the original message will have to be resent from the source application.
We are currently in WMQI v2.1. But we will soon be migrating to WBI MB v6.0. Are you aware of any issues with migrating flows involving aggregation?
Thanks,
VK. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Mar 22, 2006 1:51 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Hi vk,
I'd be interested to know which MRM physical formats you are using in your message sets ( XML/CWF/TDS ). Is the large message an XML message, or something else? |
|
| Back to top |
|
|
| vk |
| Posted: Wed Mar 22, 2006 5:00 am Post subject: |
|
|
Partisan
Joined: 20 Sep 2005
Posts: 302
Location: Houston
|
No. The source and destination are TDS message sets in MRM domian. But we also have an intermediate XML message set (GBO). Messages from multiple source applications get mapped to the same GBO. From the GBO, we map to different destinations.
I think the MRM parser is taking a lot of time to create the tree when the number of records in the message is repeating huge number of times - say 15000. Each record is of size 160 at the source and 200 at the destination.
Regards,
VK. |
|
| Back to top |
|
|
|
|
