| Author |
Message
|
| RocknRambo |
 Posted: Fri Mar 05, 2010 8:47 am Post subject: File Nodes Vs MQ |
 |
|
Partisan
Joined: 24 Sep 2003
Posts: 355
|
We have requirement to receive large files around 80 MB file (XML format), perform transfomations and route to destination. Our team has mixed opinion on using file nodes Vs MQ.
We understand the queue limit of 100 MB and the source system file does not exceed the limit. We are using WMQ/WMB v7, the document says large file handling has been enhanced. From architecture/performance perspective are there any trade off's in using file nodes Vs MQ as the transport layer.
Any pointings are appreciated.
-RR |
|
| Back to top |
|
 |
| zpat |
| Posted: Fri Mar 05, 2010 8:52 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Your file is almost certainly composed of logical records. Each of these should be sent as a separate MQ message.
Sending very large messages is usually incorrect unless you really have huge single objects like image files.
If you wish to link messages together, use grouped messages.
This can also be used to overcome the 100 MB limit. Much better to send a group of smaller messages, try to keep each one below the default of 4MB to avoid changes to channels, DLQs etc.
Messages should be atomic (i.e. individual items) like rows in a database table or records in a file. It is not correct to treat messages as a huge single item.
Messages are not analagous to files. They are analagous to records in a file. |
|
| Back to top |
|
|
| RocknRambo |
| Posted: Fri Mar 05, 2010 10:10 am Post subject: |
|
|
Partisan
Joined: 24 Sep 2003
Posts: 355
|
Agreed, if at well are doing from the scratch we would have leisure to build the system to restrict the message size to 4 MB and be compliance to all good stuff.
For this particular requirement, we cannot push back to the source system to segment or send the messages up to 4 MB. We have to build the solution for receive entire file (~80MB) and propagate to destination in multiple messages.
So, going back to file node Vs mq node, does anyone have a opinion in which is best in performance or management (error or failure any kind)
Thanks
-RR |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 05, 2010 10:34 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RocknRambo wrote: |
| So, going back to file node Vs mq node, does anyone have a opinion in which is best in performance or management (error or failure any kind) |
What will happen, if you select the MQ option, when the file size grows beyond 100Mb?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| RocknRambo |
| Posted: Fri Mar 05, 2010 10:50 am Post subject: |
|
|
Partisan
Joined: 24 Sep 2003
Posts: 355
|
We haven't seen the file size more 76 MB with an avg of 42 MB from the past 60 month history, but there could be potential in increasing. As this is one our sunset systems, we don't see that happening in near future.
the possibility of going more 100 MB are very slim (near to zero)
--
RR |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Fri Mar 05, 2010 11:24 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| Quote: |
| Your file is almost certainly composed of logical records. Each of these should be sent as a separate MQ message. |
While your source system may not be able to change, you could put a simple MQ app up that breaks the large file into much smaller, logical pieces to be passed to the broker for the transformation step. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 05, 2010 11:32 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RocknRambo wrote: |
| As this is one our sunset systems, we don't see that happening in near future. |
I've been surprised, over the year, how long it can sometimes take the sun to set.
| RocknRambo wrote: |
| the possibility of going more 100 MB are very slim (near to zero) |
So if you decide to use MQ, be sure to document that as a risk of the system so the stakeholders can agree to take the risk.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Mar 05, 2010 12:04 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I don't understand, you say MQ is your choice, but you also say you can't follow my advice and send multiple messages?
You can have one file in and multiple messages out of a flow (use propagate to do this).
If these messages need to be processed as a group by the receiver - set the groupid field and msgflags accordingly. If they can be processed atomically don't bother.
Don't send massive MQ messages, it's almost always wrong to do this and results from a lack of understanding of how to use grouped or segmented messages. If you are producing the message you do have the choice. |
|
| Back to top |
|
|
| RocknRambo |
| Posted: Mon Mar 08, 2010 8:44 am Post subject: |
|
|
Partisan
Joined: 24 Sep 2003
Posts: 355
|
Propagating to the destination is no issue, as you mentioned ESQL statement can handle.
Our concern is from the source, modifying the source in either option (group, small messages etc.,) is ruled out. We are down to MB to handle it in a optimum way (MB might not be the best choice, we have to do this for now).
We are down to a MB solution and weighing the best option b/w - reading a message from queue Vs reading a file from a folder, keeping in my mind about the new features/enhancements in MB/MB7 what's the best approach.
Does MB be better off reading file from a folder on large files ? or the other option
--Rr |
|
| Back to top |
|
|
| hallmark |
| Posted: Thu Apr 08, 2010 7:34 am Post subject: |
|
|
Voyager
Joined: 10 Mar 2005
Posts: 76
|
What does the source do again? What would be putting this 80Mb message on a queue?
To me an is more analogous to a batch of messages (especially if its structured enough for ESQL to propagate). So I would lean towards a FileInputNode that breaks the records in this manner...My main concern would be as it is with all files, transactionality, assured delivery etc. which is possibly why you are in a quandry.
I don't have the knowledge about which broker "prefers". Without doing some empirical tests (which you could do to determine the "best" format) I would guess there isn't much in it. However I am surprised you have the flexibility to stream to a file or write to MQ but aren't able to write as multiple MQ messages.
_________________
Rob |
|
| Back to top |
|
|
|
|
