| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Write to File vs. put to persistent queue |
« View previous topic :: View next topic » |
| Author |
Message
|
| kbsinc |
 Posted: Mon Feb 10, 2003 5:31 pm Post subject: Write to File vs. put to persistent queue |
 |
|
Newbie
Joined: 07 Apr 2002
Posts: 6
Location: Harrisburg, PA.
|
Customer has a legacy AIX application using Mercator as a translation broker that accepts input via files and produces remapped output to files. They want to plop this application in the middle of an existing clustered MQ setup to gain the benefits of workload balancing and increased availability. They have agreed to (obviously they have to) WebMQ enable their initial input map to receive input via a Q and also their final output MAP to produce a message to be placed on a Q.
Very often their translations are a multi-step process that takes an initial file as input and before eventually creating the final output file, creates (1-n) iterative output files along the way. Each file is input to the next MAP in the translation process. In other words, a translation that takes 5 steps to complete, would have created 5 distinct sequential files along the way. I contend that they should not only WebMQ enable the input and output MAPS, but also the intermediate MAPS as well. They are skeptical that writing to persistent Q's instead of sequential files along the way would be any more efficient due to the fact that MQ has to log the message to disk anyway. Although, there are probably plenty of other arguments for doing it MQ all the way, I am primarily focused on the performance question right now.
Has anyone come across published numbers that compares file I/O's vs. putting to persistent queues? Thanks...
_________________
cheers,
rick |
|
| Back to top |
|
 |
| RogerLacroix |
| Posted: Tue Feb 11, 2003 10:43 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
I am heavily involved with Mercator and MQ at my current client site.
I always try to get everything message enabled. Yes, there is a very minor performance hit with using MQ vs file (MQ is slower but only by milliseconds).
What you should be more concerned about 'is how do you handle restarts when things fail'. Lets say between maps #3 and #4, you get a return code of 4 (Mercator rc=28), invalid input (Map #3 outputted garbage!!).
Do you have the tools / resources to fix a message in a queue? What if you had 200 large messages and order was important? Remember you CANNOT 'edit in place' with MQ. When you edit a message and save it, the updated message with be put to the bottom of the queue!!!!
If it was the fourth message that was bad and message order was important, you have to edit 1 message to fix the problem then 'fake' edit 195 messages so that the order is preserved. Doing this at 5:00AM and not making a mistake is really important!!!
Some times speed or the 'latest-and-greatest-technology' is not the most important criteria when architecting a solution. The reason I originally wrote MQ Visual Edit is because the client I am at did not want to purchase a MQ editing tool (I am not kidding!!).
Hope that helps.
later
Roger...
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| tso0rxp |
| Posted: Wed Feb 12, 2003 5:48 am Post subject: |
|
|
Voyager
Joined: 07 Jan 2002
Posts: 85
|
We had this same design issue 'early on' in our Mercator development cycle. We used 'triggered' queues as the output of each map in the string to immediately spawn the next map. If a map fails along the way it put its messages to a separate 'error queue' for subsequent reprocessing. Order was not an issue for us so recovery was simple. We saw no performance hit using MQ vs. files and we pass rather large SAP IDOC's thru this interface.
We now have implemented Mercator Event Server which allows us to build this interface as a contained 'system'. All inputs/outputs are under the control of the Integration engine (now we use files to pass data) but the Event Server handles all I/O internally. Also, saves us from having to build separate queues for each map.
Given your circumstance I would definitely suggest using MQ for your intermediate data stores. |
|
| Back to top |
|
|
| kbsinc |
| Posted: Wed Feb 12, 2003 10:58 am Post subject: |
|
|
Newbie
Joined: 07 Apr 2002
Posts: 6
Location: Harrisburg, PA.
|
Thank you both for your insight. I have had discussions with the Mercator architect about restart issues etc... I will not pretend to be well versed in Mercator, but in design discussions I have had with him he feels we can handle the restart issues in either case (files or Q's) and also the 'order' of messages issue. Both those issues came up in our discussions, so you confirmed our concerns there. For this particular application, it basically came down to whether or not performance is going to be an issue in a high volume system. Since it appears it will not be too painful to convert a few of the more involved translation flows to Q's all the way through, we decided to do just that and do some runs in our test environment. If the performance impact is negligible we will go with the Q's. That way we can tie in with our Q monitoring tool on all phases of the translation process.
_________________
cheers,
rick |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
