| Author |
Message
|
| wilsonjohn24 |
 Posted: Wed Aug 12, 2009 5:51 am Post subject: File Input Node v MQ Input Node |
 |
|
Voyager
Joined: 02 Feb 2007
Posts: 93
Location: Scotland
|
Hi,
Context - is a large file of records is to be upload to Front end Web Application.
This Front end Web Applicationcan either send this on to WMB via either;
1) MPUT call and pass the file data in as normal message.
2) Save the file out a file System and have a WMB File Node pick up the file for processing.
Each record will then result in some services calls sort of record -> XML.
I am intrested in the forums view of the pro/cons of each approach?
thanks
John |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Aug 12, 2009 6:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
My 2 cents, other views valid, some of this may be rubbish.... 
The key question here is the content, if the application providing the data has no preference on method; if it's an old piece of COBOL that runs out files it's not worth changing it. If as in this case it's happily either/or then cosider the contents of the file. If it's a big list of disconnected records then you'll get better throughput using the MQInput node. Have multiple threads running, get the sending application to commit each record as it's ready, all the usual good stuff and you'll benefit from the asyncronous joy that is WMQ. Even if the file is committed onto the queue as a piece then you'll still do better.
If however the file is interlinked (e.g. a file header or trailer), the records have affinity or is to be processed as a single transaction then the FileInput is your better choice. This also gives you the option of retaining a copy of the file for audit purposes; experience teaches that the content of such files is often called into question by business.
I again draw the reader's attention to the disclaimer at the top.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Luke |
| Posted: Wed Aug 12, 2009 6:17 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
Hi John,
A large file of records? Bit like a batch type of thing is it?
Regardless of the transport you choose, I'd be a bit wary of this. If you're processing realtime traffic, then swamp your middleware with batch stuff, things are likely to get a bit messy, and slow.
How about some kind of batch job to process them instead? Alternatively, if it has to go through Broker, you might want to do something that allows you to drip-feed the messages in, so that you don't get so many problems with queue build-ups and slow responses ... |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Aug 12, 2009 6:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Luke wrote: |
If you're processing realtime traffic, then swamp your middleware with batch stuff, things are likely to get a bit messy, and slow.
|
Unless you hive the file stuff off to a different EG. But otherwise yes.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Luke |
| Posted: Wed Aug 12, 2009 6:29 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
Agree a different EG would help, but I'm also thinking that if John is planning on invoking service calls as he says, then there may still be knock on effects in terms of queue/xmitq depth build up (if services invoked over MQ), or the services themselves may struggle to handle the throughput.
I'm speculating here though, just things you may wish to consider. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Aug 12, 2009 6:38 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
File processing is inherently non-transactional. You run the risk of not FTPing a file at all, or FTPing it twice by accident.
FTP is a manual command, not suited to automation, it can result in partial transfers, it has no restart capability and so on.
Use MQ end to end, for assured, one-time, data delivery. |
|
| Back to top |
|
|
| wilsonjohn24 |
| Posted: Wed Aug 12, 2009 7:46 am Post subject: |
|
|
Voyager
Joined: 02 Feb 2007
Posts: 93
Location: Scotland
|
Yes very batch in nature - for example load 10 000 perspective customers.
For each customer call down to WMB exposed services ( most realized in the backend systems). Keeping a separate execution group as an option.
So Is a File InputNode any faster than a MQ Input node?
I was hoping the file input node would load provide a more optimized read of a large data set? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 12, 2009 8:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Yes, the FileInput node has access to a streaming model that can handle larger chunks of data in a more efficient manner than the MQInput node does.
The FileInput node is also capable of reading chunks of data that are larger than the 100MB that is the maximum size of an MQInput message.
You will find, however, that the largest determinant in processing performance is how you code your flow rather than anything else. |
|
| Back to top |
|
|
|
|
