| Author |
Message
|
| Accell |
 Posted: Sat Feb 07, 2015 2:01 am Post subject: Urgent!!-File transfer on condition |
 |
|
Novice
Joined: 07 Feb 2015
Posts: 24
|
Hi,
The requirement is that i need to FTP 3 files in a sequence at same time(ms difference) to a remote server.
source:remote server 1.
Target:remote server 2.
file name for eg (bcd123.txt , abc123.txt , xyz123.csv.txt)
files should not move until and unless all three files are present at local server.
file size: small(kbs)
Multiple batches of files can arrive in a day.
Thanks!! |
|
| Back to top |
|
 |
| nelson |
| Posted: Sat Feb 07, 2015 8:18 am Post subject: |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
| How are generated the files, in the same order they should be transferred? Why don´t you detect only the last file... then within the flow, you decide the order in which they should be transferred... |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Feb 07, 2015 9:42 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
IF the FTP destination is some distance away (on a different continent) AND the files differ wildy in size (100kb ans 10Mb) you will find that doing remote FTP and getting all three files to the destnation in a READABLE form within a few milliseconds impossible to avhieve.
IF this is the case then may I humbly suggest that you go back to whoever designed this system and ask them to sort out that problem first.
To be honest, I'd really plump for some sort of processing on the target system in the time difference is that important. Simple network congestion could make even short distance transfers have say a 1 second time difference.
Can you guarantee that this will NEVER EVER happen (even once) in a million sets of data?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Accell |
| Posted: Sat Feb 07, 2015 1:10 pm Post subject: |
|
|
Novice
Joined: 07 Feb 2015
Posts: 24
|
Hi,
Thanks for quick replies!!
file size did not differ much in size.
Lets say i collect these files locally via one flow,is there any way to trigger the 2nd flow only when all 3 files of a batch are present.?
Files will be kept at source location sequentially.
Many thanks!!! |
|
| Back to top |
|
|
| nelson |
| Posted: Sat Feb 07, 2015 4:01 pm Post subject: |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
| nelson wrote: |
| How are generated the files, in the same order they should be transferred? |
That´s why I asked you this. If you can detect the last file and the files are sequentially generated. Then one flow can detect this last file... and as it is supposed that the other files exist, so... transfer the first, the second, and the third sequentially (in the same flow). |
|
| Back to top |
|
|
| Accell |
| Posted: Sun Feb 08, 2015 12:28 am Post subject: |
|
|
Novice
Joined: 07 Feb 2015
Posts: 24
|
| nelson wrote: |
| nelson wrote: |
| How are generated the files, in the same order they should be transferred? |
That´s why I asked you this. If you can detect the last file and the files are sequentially generated. Then one flow can detect this last file... and as it is supposed that the other files exist, so... transfer the first, the second, and the third sequentially (in the same flow). |
Hi Nelson,the file names are HIG-1234-XY-01.csv
ABP-1234-XY-01.csv
PFS-1234-XY-01.csv
Lets say flow detects only 3rd file(PFS*),will it not only transfer the 3rd file and exclude the other two.?
May be i m missing smthing as i m new to WMB.
Many Thanks!!! |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Feb 08, 2015 12:35 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
My comments about being able to guarantee that all three files arrives at their destination in a readable form with a few milliseconds still stands.
IMHO, this is the big problem. how are you going to resolve that? What happens when midway through the transfer of the last file, someone else starts a network backup and loads the network?
AFAIK, you can't so you are really left with a simple choice
1) Change this milliseconds to seconds
2) Hold the files locally and trigger local renaming when the final file is received.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Accell |
| Posted: Sun Feb 08, 2015 12:42 am Post subject: |
|
|
Novice
Joined: 07 Feb 2015
Posts: 24
|
| smdavies99 wrote: |
My comments about being able to guarantee that all three files arrives at their destination in a readable form with a few milliseconds still stands.
IMHO, this is the big problem. how are you going to resolve that? What happens when midway through the transfer of the last file, someone else starts a network backup and loads the network?
AFAIK, you can't so you are really left with a simple choice
1) Change this milliseconds to seconds
2) Hold the files locally and trigger local renaming when the final file is received. |
Hi,okay lets say i change it to seconds.
Now for triggering part..How can i trigger the flow.Do i need a trigger file.?
Many Thanks!! |
|
| Back to top |
|
|
| nelson |
| Posted: Sun Feb 08, 2015 6:34 am Post subject: |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
| Accell wrote: |
Lets say flow detects only 3rd file(PFS*),will it not only transfer the 3rd file and exclude the other two.?
May be i m missing smthing as i m new to WMB.
Many Thanks!!! |
I imagine the flow doing something like this:
1. Read the third file. (as the flow's input)
2. Store the third file data.
3. Read the first file.
4. Send the first file.
5. Read the second file.
6. Send the second file.
7. Restore the third file data and send it.
This is the way that comes to my mind, as it is a batch process and requires order, maybe there is a better way. Maybe the destination could do a simple check whether the three files has arrived or not? and you only need to write a flow that transfers *.csv files (read and send)? 
Hope that helps. |
|
| Back to top |
|
|
| Accell |
| Posted: Sun Feb 08, 2015 3:13 pm Post subject: |
|
|
Novice
Joined: 07 Feb 2015
Posts: 24
|
Thanks!!
However ,could i please get some help on the triggering part.How to trigger the flow when the third file arrives.Do i need to write an ESQL code?
Many thanks!! |
|
| Back to top |
|
|
| nelson |
| Posted: Sun Feb 08, 2015 4:43 pm Post subject: |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
| Accell wrote: |
Thanks!!
However ,could i please get some help on the triggering part.How to trigger the flow when the third file arrives.Do i need to write an ESQL code? |
FileInput node? You should check FTP properties, as you said the source is a remote server. |
|
| Back to top |
|
|
| Accell |
| Posted: Mon Feb 09, 2015 11:40 am Post subject: |
|
|
Novice
Joined: 07 Feb 2015
Posts: 24
|
Yes but these properties needed to be set in file input node or smwhere else?
Many thanks!! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 09, 2015 11:44 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
First question - why are you creating three files in the first place? Are they coming from three separate programs?
Second question - why are you trying to solve this by changing the triggering of the file input, rather than using collection or aggregation?
Third question - how do you know when all three files *should* have arrived, to know if one of them *didn't*.
Fourth question - can you use a fourth file to trigger the process instead of one of the files in the set?
Fifth question - why are you using files in the first place, and not messages of some kind? |
|
| Back to top |
|
|
| Accell |
| Posted: Mon Feb 09, 2015 12:27 pm Post subject: |
|
|
Novice
Joined: 07 Feb 2015
Posts: 24
|
| mqjeff wrote: |
First question - why are you creating three files in the first place? Are they coming from three separate programs?
Second question - why are you trying to solve this by changing the triggering of the file input, rather than using collection or aggregation?
Third question - how do you know when all three files *should* have arrived, to know if one of them *didn't*.
Fourth question - can you use a fourth file to trigger the process instead of one of the files in the set?
Fifth question - why are you using files in the first place, and not messages of some kind? |
1.Client Requirement(3rd party generates the file)
2.I tried with collector node,but messages are getting piled up in SYSTEM.BROKER.EDA.EVENTS.
3.There should be least difference of time while transfer.
4.Yes,The Client agreed to provide a fourth file of 0 kb to trigger the process (Could you please brief and help me on this part?).
5.Again,Client requirement.
Many Thanks!!! |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 09, 2015 1:21 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Accell wrote: |
| 1.Client Requirement(3rd party generates the file) |
If a third party generates the 3 input files, why are they not sending them to the remote server? If you're processing the input files, why are you not storing them using the processing flow and then retrieving the data / splitting it back up into 3 files when the trigger arrives?
| Accell wrote: |
| 2.I tried with collector node,but messages are getting piled up in SYSTEM.BROKER.EDA.EVENTS. |
What do you think is causing that (hint - it's a problem with your flow)
| Accell wrote: |
| 3.There should be least difference of time while transfer. |
And how will the reading flow know how much time has elapsed so it can tell that (for example) the first and third files have arrived but the second one is missing? What action are you expected to take when one file is missing but the other two are present? What action will you take when all three files arrive but the trigger file does not?
| Accell wrote: |
| 4.Yes,The Client agreed to provide a fourth file of 0 kb to trigger the process (Could you please brief and help me on this part?). |
This doesn't help much - see my answer above.
| Accell wrote: |
| 5.Again,Client requirement. |
What requirement? What, exactly, do they need to happen? I expect that what you're trying to build is what they've asked for, but what do they actually need?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
