| Author |
Message
|
| tatyeong |
 Posted: Sun Apr 18, 2010 8:16 pm Post subject: File Transfer Sequence |
 |
|
Acolyte
Joined: 16 Apr 2010
Posts: 52
|
Hi,
Suppose sequencing is important in the processing of the batch files.
Let's say there is an batch file interface in which the source system generates a flat file every 1 hour.
We configure the WMQ FTE using a file trigger mechanism. Once the file is placed in the specific folder, the file transfer will begin.
I am looking at the scenario where the file is stalled in transmission for whatever reason. Let's say this is the file generated by the 9am job at the source system. At 10am, the source system produces another file, and FTE will pick up the 2nd file (since it is multi-thread). And that the 2nd file is successfully transferred to target system before the 1st file. In that case, my target system will process the 2nd file first, hence losing the sequencing.
Will this scenario ever happen? Or it will always be FIFO? |
|
| Back to top |
|
 |
| mphillip |
| Posted: Mon Apr 19, 2010 6:14 am Post subject: |
|
|
Apprentice
Joined: 03 Feb 2009
Posts: 49
|
In the scenario you describe there would be no guarantee that with two totally independant transfer requests that the first would happen before the second.
For example, if the destination agent was stopped and 4 transfer requests built up, when the destination agent starts up the source agent will start all 4 transfer requests concurrently.
Even if you restricted the source agent to only sending one transfer at a time (maxSourceTransfers property for an agent), this would still not help in your situation, as Agents will queue up transfer requests that they cannot currently process and retry them later - this is so that a transfer to AgentB (which is currently stopped) does not block a transfer that could go ahead to AgentC (which is up and running). |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 19, 2010 6:36 am Post subject: Re: File Transfer Sequence |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| tatyeong wrote: |
| Suppose sequencing is important in the processing of the batch files. |
This is the old school version of message affinity. It's as bad a design in files as it is in messages. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fatherjack |
| Posted: Mon Apr 19, 2010 7:06 am Post subject: Re: File Transfer Sequence |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| Vitor wrote: |
| This is the old school version of message affinity. It's as bad a design in files as it is in messages. |
Agreed but unfortunately these things are generally dumped on us because "thats the way the legacy app works".
You probably need to implement some form of sequence checking in your app. E.g. get the sender to add a sequence number to the file name or in a file header and check it increases by 1 each time. If not ...... well good old manual intervention I guess to hold the file until you're back in sequence. |
|
| Back to top |
|
|
| tatyeong |
| Posted: Mon Apr 19, 2010 7:26 am Post subject: |
|
|
Acolyte
Joined: 16 Apr 2010
Posts: 52
|
I am new to MQ and MQFTE, so pardon me if my question doesn't make sense.
Since MQFTE is implemented on top of MQ, wouldn't it go by FIFO? If I browse the QM, there will be 1 queue for source agent and 1 queue for target agent.
fatherjack : Yes, managing at the app level would be an alternative that I can implement. Just wanted to know how MQFTE handles concurrent file transfer
vitor : there are a lot of valid scenarios that sequencing may be required, no? Say the first file contains info to create a customer record in target system. The second file contains some fields to be updated in the customer record created from the first file. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 19, 2010 7:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| tatyeong wrote: |
| vitor : there are a lot of valid scenarios that sequencing may be required, no? Say the first file contains info to create a customer record in target system. The second file contains some fields to be updated in the customer record created from the first file. |
In all these scenarios there are design solutions to eliminate sequence. Which are no worse and in many ways better than the solutions required to ensure sequence.
Take your case - if you're changing the app to manage sequence, why not change it so that sequence become irrelevant?
I accept this is often out of scope for such changes. It just annoys me when the Powers That Be cling to the old solution "because that's how it's always worked".
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mphillip |
| Posted: Mon Apr 19, 2010 7:36 am Post subject: |
|
|
Apprentice
Joined: 03 Feb 2009
Posts: 49
|
FTE is a bit more complicated that just taking a file and sticking it as a single message on a queue. There is a whole protocol of message exchange between two FTE agents that allows the file data to be split into small messages and transfered between the agents. FTE can also transfer multiple files concurrently.
So, the basic queueing 'FIFO' of messages does not actually apply to the transfer of files by FTE. |
|
| Back to top |
|
|
|
|
