| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| WS MQFTE V7.0 combo with classic FTP in progress - Issue |
« View previous topic :: View next topic » |
| Author |
Message
|
| Nenad_SBS |
 Posted: Mon Feb 02, 2009 8:50 am Post subject: WS MQFTE V7.0 combo with classic FTP in progress - Issue |
 |
|
Newbie
Joined: 02 Feb 2009
Posts: 8
|
Hello,
I just wonder, if someone noticed already this behavior of WS MQFTE V7.0 :
[ so far, on this forum I can't find related topic(s) ]
-> This is the "topology" :
3 x Windows 2003 Server (SP 2)
- One machine has full installation of MQ V7 Server, MQ FTE V7 and MQ FTE V7 Remote Tool ( this tis default QManager and Coordination QManager machine)
- Other Two machines are WSMQ FTE AGENTS (clients)
-> Scheduled and/or Triggered transfers working fine (between those two agents) except:
When I try to put file Test_File.ZIP (or any other) to AGENT1OUT folder using classic FTP... #->
[triggered (file=Exist) or scheduled transfer, or combination of both, triggered and scheduled, created against that folder -AGENT1OUT- in manner that move file to AGENT2IN folder with deleting source file after transfer]
#->... WSMQ FTE TAKE (part) OF THIS file (AGENT1OUT\Test_File.zip) even FTP did not finished transfer, and move (with deleting source) a piece of file to AGENT2IN ...
After that, classic FTP (still in progress) become a bit confused ... 
Any suggestions/recommendations/links/etc ... ?
thanks in advance
Nenad |
|
| Back to top |
|
 |
| RichCumbers |
| Posted: Mon Feb 02, 2009 12:16 pm Post subject: |
|
|
Novice
Joined: 22 May 2006
Posts: 10
Location: Hursley, UK
|
Hi there,
So can I confirm the scenario before going any further.....
you have a trigger with a schedule for a given file, and when that file appears (which could be large in size) the action to take it to transfer that same file that is being ftp'd across and then delete it?
Cheers
Rich
WMQFTE Development
(http://twitter.com/cumbers) |
|
| Back to top |
|
|
| Nenad_SBS |
| Posted: Mon Feb 02, 2009 2:31 pm Post subject: |
|
|
Newbie
Joined: 02 Feb 2009
Posts: 8
|
Sorry if my explanation wasn't "clear" enough. Also I apologize about my English.
Scenario is:
1. Scheduled or trigger with a schedule (doesn't matter, I tried both of those combinations) against folder(AGENT1OUT) in fact, against any file on that folder ( *.* ). [if trigger is setted up, condition is" file=exist]
The file can be "created" using different "ways" ( classic FTP or by some user's Application or using copy/paste or ...) and files are with different size (from 300KB -> few hundreds MB).
1.a. Intention is:
if any file or files, appears on that specific folder(AGENT1OUT) (created via classic ftp or any other "way" ) MQ FTE Agent need to move that file, or more files, (with deletion on source folder) to another folder on another machine (AGENT2IN)
2. So, when classic FTP start with putting/creating file on that folder(AGENT1IN), if MQ FTE AGENT start with transfer(scheduled or trigger with schedule) then:
before "FTP finish his job", MQ FTE CUTTED OF that file of FTP(even FTP transfer didn't finished "work" ), and only part of the file become transfered to another FTE AGENT and at the same time FTP Is interrupted and cancelled.
Size of file is not issue, sometimes it is just few MBs, sometimes hundreds of MBS. For example:
-original file size is 388MB, MQ FTE Agent cutted of file, while FTP was in progress, in moment when only 138MB was transfered, or only 54MB or only 34MB ... (I tried several times).
-or original file size is 67MB, MQ FTE Agent start schedule when just 14MB was transfered to AGENT1OUT folder, and result is 14MB on AGENT2IN folder.
Maybe sssue is (my modest opinion) about locking file during FTP or MQ FTE AGENT is "much stronger" than FTP proces ?
(ftp can't lock the file during transfer in manner that MQ FTE Agent can't "use that file" before FTP end normaly... ? ) |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Feb 02, 2009 2:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you looked into ftp post processing?
FTP to a neutral directory and initiate a move to your input directory as post processing. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Nenad_SBS |
| Posted: Mon Feb 02, 2009 3:30 pm Post subject: |
|
|
Newbie
Joined: 02 Feb 2009
Posts: 8
|
Hello fjb_saper:
>Have you looked into ftp post processing?
>FTP to a neutral directory and initiate a move to your input directory as post >processing.
>_________________
>MQ & Broker admin
No, I didn't, because I looking for "full automated flow" using only MQ FTE "between":
-> end user's creation of the file on a given folder(AGENT1OUT) ->
and
->the very same file ended up at destination folder(AGENT2IN on another machine)<-
idea is:
- "anyone" should be able to FTP certain file(s) to a given folder(AGENT1OUT) without any further "actions"
- or more precisely, "anyone" should be able to put/create certain file(s) to a given folder(AGENT1OUT) using "different applications" ( FTP, or copy/paste, or "anything else"... including ".NET coded user's application" )
without any further actions (taken by, neither, end users and/or: MQ Admins,o r Storage Admins, or ... )
then:
MQ FTE take that file(or those files) and move it to AGENT2IN => end of job
And one more detail:
During copy/paste progress, file is "untouchable" for MQ FTE Agent process until copy/paste ends normaly ... |
|
| Back to top |
|
|
| RichCumbers |
| Posted: Tue Feb 03, 2009 2:14 am Post subject: |
|
|
Novice
Joined: 22 May 2006
Posts: 10
Location: Hursley, UK
|
| Nenad_SBS wrote: |
Maybe sssue is (my modest opinion) about locking file during FTP or MQ FTE AGENT is "much stronger" than FTP proces ?
(ftp can't lock the file during transfer in manner that MQ FTE Agent can't "use that file" before FTP end normaly... ? )
|
What you assume above is correct. Our application will try and get a lock on the file, and if it succeeds then it will start to transfer the file. The triggering mechanism is expected to work in this way as we have no way of knowing how big the file is going to be, and if we can get a lock on the file we assume we can transfer it.
Typically a trigger file that appears is one that does not get transferred. I would suggest sending the file you wish to transfer, followed by a very small trigger file which you set WMQFTE triggers on. Then the file will be complete and the transfer will move the whole file, rather then just part of it.
Rich
WMQFTE Development
(http://twitter.com/cumbers) |
|
| Back to top |
|
|
| Nenad_SBS |
| Posted: Tue Feb 03, 2009 6:18 am Post subject: |
|
|
Newbie
Joined: 02 Feb 2009
Posts: 8
|
| RichCumbers wrote: |
| Nenad_SBS wrote: |
Maybe sssue is (my modest opinion) about locking file during FTP or MQ FTE AGENT is "much stronger" than FTP proces ?
(ftp can't lock the file during transfer in manner that MQ FTE Agent can't "use that file" before FTP end normaly... ? )
|
What you assume above is correct. Our application will try and get a lock on the file, and if it succeeds then it will start to transfer the file.
...
... I would suggest sending the file you wish to transfer, followed by a very small trigger file which you set WMQFTE triggers on. Then the file will be complete and the transfer will move the whole file, rather then just part of it.
Rich
WMQFTE Development
(http://twitter.com/cumbers) |
Hello Rich,
Thank you very much for paying attention and for Your answers.
Yes, I already tryed with a "cheating aproach",
more preciosly: with a small file (named TRANSFER_IN_PROGRESS) before FTP and when FTP ends corectly, deleting that small file. At the same time, trigger is seted up with two conditions:
file=exist
file!=exist (TRANSFER_IN_PROGRESS)
and this method work just fine 
What is pretty much unbelievable to me is :
[my professional orientation is System Z and z/OS, z/VM]
how it is possible that FTP can not lock his own file ? in that way that "noone else" (except, maybe, few of very strong system procesess) can take control over that file (while FTP is still in progress ... ) ?
I assume that milliseconds are issue (FTP in progress <=>Trigger "jump in" ) even with very small files, so in real production, with DATA wich can be considered as important, behavior like this can be a problem ...
Best regards,
Nenad |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Feb 03, 2009 6:28 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| There's nothing in the FTP standard that dictates how FTP servers should handle the files that they create in response to receive operations. |
|
| Back to top |
|
|
| markt |
| Posted: Tue Feb 03, 2009 6:36 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
| Some implementations of ftp/ftpd certainly do lock the files they are creating. But with no standard to point to, you can't assume they all will. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Feb 03, 2009 6:46 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
The point of purchasing a managed file transfer product is to avoid the need for these FTP start/stop considerations.
The IBM WMQ FTE product should be enhanced to act as an FTP server so that it would know when the incoming file transfer ended.
To clarify, this would mean that it would accept incoming FTP client connections at the IP socket level and emulate an FTP server.
This is a suggestion, not a statement of fact or direction (since I don't work for IBM). |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Tue Feb 03, 2009 9:01 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
| zpat wrote: |
| The point of purchasing a managed file transfer product is to avoid the need for these FTP start/stop considerations. |
I thought the whole point of MQ FTE was to replace FTP completely, now you appear to be stuck "half way" enjoying worst of both worlds...
what is keeping you from completely going MQ FTE instead of keeping the FTP front-end?
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| RichCumbers |
| Posted: Tue Feb 03, 2009 9:36 am Post subject: |
|
|
Novice
Joined: 22 May 2006
Posts: 10
Location: Hursley, UK
|
| zpat wrote: |
The point of purchasing a managed file transfer product is to avoid the need for these FTP start/stop considerations.
The IBM WMQ FTE product should be enhanced to act as an FTP server so that it would know when the incoming file transfer ended.
To clarify, this would mean that it would accept incoming FTP client connections at the IP socket level and emulate an FTP server.
This is a suggestion, not a statement of fact or direction (since I don't work for IBM). |
I would not expect users to ftp a file from a remote location to a machine with an agent, and then for an agent to act on the trigger of this file and move it across.
An FTE Agent can run in client mode on a machine without MQ installed, which means that you could replace the FTP entirely with a remote agent on the machine that you are ftp'ing the file from. I see the original scenario as one hop too many, seems like it is going from Machine A -> Machine B via FTP and then from Machine B -> Machine C via FTE. Unless network visibility between Machine A and Machine C is a problem I would recommend going direct using FTE of course 
I would love to understand the need for this FTP part of the scenario. Our team are always looking for genuine Use Cases from users of our product.
Rich
WMQFTE Development
(http://twitter.com/cumbers) |
|
| Back to top |
|
|
| Nenad_SBS |
| Posted: Tue Feb 03, 2009 1:23 pm Post subject: |
|
|
Newbie
Joined: 02 Feb 2009
Posts: 8
|
| RichCumbers wrote: |
| zpat wrote: |
The point of purchasing a managed file transfer product is to avoid the need for these FTP start/stop considerations.
The IBM WMQ FTE product should be enhanced to act as an FTP server so that it would know when the incoming file transfer ended.
To clarify, this would mean that it would accept incoming FTP client connections at the IP socket level and emulate an FTP server.
This is a suggestion, not a statement of fact or direction (since I don't work for IBM). |
... I see the original scenario as one hop too many, seems like it is going from Machine A -> Machine B via FTP and then from Machine B -> Machine C via FTE. Unless network visibility between Machine A and Machine C is a problem I would recommend going direct using FTE of course
I would love to understand the need for this FTP part of the scenario. Our team are always looking for genuine Use Cases from users of our product.
Rich
WMQFTE Development
(http://twitter.com/cumbers) |
1. Yes, original scenario is:
[any machine out there] -> FTP or user's .NET application or whatever -> Machine B
then:
Machine B -> MQ FTE -> Machine C
1.a. yes, files MUST BE moved: Machine B -> Machine C (without corruption, even for one single bit) and yes, I do have respect for how MQ treat integrity-consistency of data (from point when MQ become "owner" of data/file).
1.b. Yes, network visibility between [any machine out there] and Machine C is a problem, realy big problem
1.c. yes, it will be very very nice to have full automated flow between Machine B and Machine C. (thousands files per day, small or big, random timestamp 0-24h)
2. It is impossible that [any machine out there] use MQ FTE Agent because:
not only matter of price [x.xxx,xxEUR],
also matter of logistic/location/organisation/ownership/so_many_end_users_"reasonable_reasons" 
2.a. And, after all, somehow, THE FILE must be created and start with his miserable life, on any folder somewhere, in one moment. If that moment and folder are the very same moment and folder when MQ FTE Agent start with an '"acitivity"[even on any machine out there], what result we can expect?
And I must say one more thing,
In my opinion, this issue is not addressed to MQ FTE's dissability or responsibility
real address is: FTP <=> TCP/IP <=> Windows (locking mechanisms, I would say, BASIC PREROGATIVE of any operating system or component of operating system or owner/creator of any file/data)
But, if MQ FTE developers can use "anything" and/or "everything" and solve "lack of basic file/data integrity-consistency", generated by WIN-TCP/IP-FTP, then they must do that because MQ FTE will be a serious piece of SW after solving this "little inconvenience".
This is just my opinion
thank You all again for contribution
Nenad |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Feb 03, 2009 1:36 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
One thing to take away from this is that your choice of FTP Server on Windows may impact the behavior in creating a file.
Some FTP servers, for example, will write the incoming file to a temporary location with a temporary name, and only when the file is complete will they move it to the final location with the final name.
Microsoft IIS FTP Server is not known to be friendly in this area. You may be able to script this behavior in ISS, or you may be able to use a different FTP server and have better results. |
|
| Back to top |
|
|
| Nenad_SBS |
| Posted: Tue Feb 03, 2009 1:55 pm Post subject: |
|
|
Newbie
Joined: 02 Feb 2009
Posts: 8
|
| mqjeff wrote: |
One thing to take away from this is that your choice of FTP Server on Windows may impact the behavior in creating a file.
Some FTP servers, for example, will write the incoming file to a temporary location with a temporary name, and only when the file is complete will they move it to the final location with the final name.
Microsoft IIS FTP Server is not known to be friendly in this area. You may be able to script this behavior in ISS, or you may be able to use a different FTP server and have better results. |
Hello mqjeff,
Yes, I'll try some ISV (independent software vendor) FTP service (FTP Server) instead IIS FTP Server, unfortunately my "regular working time" is 08-19h, and I don't have much time for trying "everything" 
and, I must say  , this is not "my choice" => FTP Server on Windos 
My choice is, as I said before, z/OS and z/VM on System Z 
my role-connection with this issue is: good will
Best regards,
Nenad |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
