| Author |
Message
|
| jharringa |
 Posted: Thu Feb 07, 2008 12:31 pm Post subject: FileInput 6.1 for grabbing files from the mainframe |
 |
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
I need to fetch files from the mainframe and I've been trying to set up my FTP directory but I have been unsuccessful so far. I've tried putting in 'MYMFDIR1' or MYMFDIR1 and even . but I seem to get the following in my system log (I suspect this is occurring only when I use . and that the other commands aren't throwing errors and I'm probably in an incorrect directory):
| Code: |
| [7197]BIP3380E: File node 'FileInput' in message flow 'MyFileInput' could not connect to remote FTP server 'ftpserver.mainframe.com'. Reason 'CWD .=>501 A qualifier in "." begins with an invalid character.' : UUID: /build/S610_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 711: com.ibm.broker.nodes.filenodes.ComIbmFileInputNode.getFtpFileManager: DynamicSubscriptionEngine: DynamicSubscriptionEngine |
Is there somewhere that I can look (assuming the ftp server doesn't log commands) to see what commands are being executed by the node? |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Feb 07, 2008 2:31 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
FTP tends to assume a path separator of "/", regardless of the remote OS.
That said, the FTP server side on the MF should be logging everything you're trying to do... if the MF admin configured it to do so, that is.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 08, 2008 1:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Speak to whoever's running the FTP on the mainframe side. Because mainframe files are not stored in any kind of directory structure you'll either need to pass over the DSN name or (more likely) find how the ftp is mapping onto the files.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqpaul |
| Posted: Fri Feb 08, 2008 7:21 am Post subject: Start with a workstation FTP client |
|
|
Acolyte
Joined: 14 Jan 2008
Posts: 66
Location: Hursley, UK
|
As you realize, the BIP3380 message insert
| Quote: |
| CWD .=>501 A qualifier in "." begins with an invalid character. |
is simply reporting a error return from the mainframe FTP server. It would probably help to use an FTP client on your workstation to see if you can understand what the server is expecting. That would get the Message Broker layer out of the way, and make it easier to understand what's happening.
You may get some help from the FTP PWD, CWD, CDUP and LIST commands, together with HELP, SITE and SYST to see what directory you start in, and where and how you can navigate elsewhere.
_________________
Paul |
|
| Back to top |
|
|
| jharringa |
| Posted: Fri Feb 08, 2008 8:20 am Post subject: |
|
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
OK. So this is where I stand right now:
I probably didn't need to give you that error code since I got past that error.
I talked to the FTP admins and it appears that it is getting to the correct "directory" and is getting the correct listing. It is essentially running these commands:
ftp mymainframe.com
userid
password
cd 'userid.'
quote pasv
dir
quit
I would guess that from the dir command it checks to see if it can find a file with the same "File name or pattern" as in the Basic tab in the properties of the FileInput.
I have executed the same exact commands on my local FTP to try and take broker out of the picture and I do see the file that I'm looking for in the listing. So now I think that this comes down to what Broker is searching for and how it is searching through the listing.
I'm searching for the following pattern: *RDPFILE* and my listing looks like this:
| Quote: |
ftp> dir
200 Port request OK.
125 List started OK
Volume Unit Referred Ext Used Recfm Lrecl BlkSz Dsorg Dsname
Migrated B12.D123.CDT.BANKO.JDDD.
ACTIVE
Migrated B12.D123.CDT.BANKO.JDDD.
TERM
S33089 3390 2008/02/07 1 1 VB 255 27998 PS JUNKRDP1
S33073 3390 2008/02/07 1 1 FB 80 3520 PS RDPFILE1
Migrated AA12345.CUST.ACCT.MSG
250 List completed successfully.
ftp: 450 bytes received in 0.03Seconds 15.00Kbytes/sec. |
I am continuing to try out different file patterns to select this. Any other ideas? The Record Detection is set up as Fixed Length 80 bytes and I do have this defined in a Message Set (and Input Message Parsing is pointed at that Message Set). |
|
| Back to top |
|
|
| JLRowe |
| Posted: Fri Feb 08, 2008 8:43 am Post subject: |
|
|
Yatiri
Joined: 25 May 2002
Posts: 664
Location: South East London
|
As mentioned earlier, the mainframe has a totally different file system and layout to (say) a unix type server. I can see from the directory listing that there are all kinds of mainframe specific columns, the ftp client code in the broker cannot interpret this format and is raising an error.
I have seen this before on the AS/400, which has 2 modes of ftp access, 'native' library style which most ftp clients baulk upon and a more friendly format that makes it look like any other unix ftp server. I seem to remember the command was called sitefmt, not sure if you can do this on the mainframe or not. |
|
| Back to top |
|
|
| jharringa |
| Posted: Fri Feb 08, 2008 8:46 am Post subject: |
|
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
| Very understandable (although somewhat funny given IBM's love for the mainframe). Should I see this error in the system logs or is this a silent error? I don't see any output the the system logs nor anything in my debug trace (which is understandable). |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Feb 11, 2008 12:19 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Mainframe (z/OS) FTP dataset access works in one of three modes.
1. HFS/ZFS - this is a true hierarchical file store like UNIX if you are using the Unix features of z/OS
2. Mapping directory names to dsname level qualifiers, this is where the CD commands traverse down the datasetname qualifiers and the DIR command lists datasets at each level. Also works at the member name level when the dataset is a PDS.
3. Fully qualified dataset name, this is where the dataset name is passed in single quotes by the FTP client and directly describes the target. This is my recommendation. eg
get 'aa.bb.cc.dd'
-----------------
You generally want to issue the ASCII FTP subcommand to enable EBCDIC-ASCII conversion during file transfer. There are many other FTP subcommands available and the IBM manuals are comprehensive. |
|
| Back to top |
|
|
| jharringa |
| Posted: Tue Feb 12, 2008 4:59 am Post subject: |
|
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
So I think we'll try to have the COBOL job write to the UNIX filesystem on the mainframe so that we can FTP in that way.
I have copied over one of the datasets and went in that way successfully but the only problem is that I think that the FileInput isn't too happy with the fact that it is EBCDIC. Is FileInput supposed to support EBCDIC files? I haven't found anything that seems to point me in either direction and I would think that it would work correctly.
I'm not really too concerned if we have to do an EBCDIC to ASCII conversion for this application. But I'm thinking that a translation may affect the way any signed packed decimal fields would turn out (this would be bad in some other projects). |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 12, 2008 5:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jharringa wrote: |
| I'm not really too concerned if we have to do an EBCDIC to ASCII conversion for this application. But I'm thinking that a translation may affect the way any signed packed decimal fields would turn out (this would be bad in some other projects). |
Most "standard" conversions are on a code page basis - this will turn any packed data into cream cheese. If you decide to RYO conversion be aware that the holding of packed data is about as platform specific as you get (the big endian / little endian debate is an obvious one). You might want to consider unpacking any numerics into character format then using a converter.
Depends on your data obviously.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Feb 12, 2008 5:28 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You should actually model your data in CWF or TDS, and use that with the FileInput node. This should allow you to deal with the file as EBCDIC.
Also, make sure that you do not set Transfer mode on the FTP tab of the FileInput node to "ASCII".
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| jharringa |
| Posted: Tue Feb 12, 2008 5:31 am Post subject: |
|
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
| OK. So it seems like there may be something wrong with the way that I'm modeling my message set then now that you say that. I'll have to investigate that further. I thought that I had imported this correctly but maybe I need to do something more after importing from the copybook. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Feb 12, 2008 5:36 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Did you set Message coded character set ID to the EBCDIC code page?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mqpaul |
| Posted: Tue Feb 12, 2008 5:49 am Post subject: ASCII and EBCDIC - FTP conversions |
|
|
Acolyte
Joined: 14 Jan 2008
Posts: 66
Location: Hursley, UK
|
It was good advice to use BINARY FTP transfer, and tell Broker to model the data in EBCDIC.
According to the FTP RFP, ASCII transfers should use US ASCII conversion. (I say "should" because it's up to the FTP server. Some servers ignore the ASCII flag, and some use it only to indicate the line end sequences should be standardized.)
You would get data loss on text coded in say ISO8859-1, the common European character set, since characters in the ISO set but not in US ASCII cannot be converted.
_________________
Paul |
|
| Back to top |
|
|
| jharringa |
| Posted: Tue Feb 12, 2008 9:42 am Post subject: |
|
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
Update:  FileInput now works great on a non-mainframe filesystem as long as I set the Message coded character set ID on the FileInput node (thanks jefflowrey!). I think that for some reason I took "Broker system default" to mean "Broker figures it out." Don't ask me why I did that...
So now my flow essentially takes the file, converts each line to CCSID of 1208 (setting OutputRoot.Properties and propagate to MQOutput) and outputs to the queue. Thank you all for your expertise on this. I have learned a great deal from you (and from other reading of course).
Now the only oddity is that transactionality doesn't seem to be working as I expect. If I set Transactions to "Yes" on the FileInput node shouldn't my MQ messages be in SyncPoint on the queue? In debug mode, I can watch my message go across with Root.Properties.Transactional=true but my next flow still picks up the message. My next flow is also set to Transaction mode of "Yes" on the MQInput. The other thing is that if the flow fails for some message down the line it seems to be alright with continuing its processing. I tested this by adding code to CAST an 'A' as an integer on the 3rd record (each record has a unique identifier so I compared with the identifier of the 3rd record). It processed records 1,2, and 4. Maybe I don't understand how Transactionality is implemented here but the docs say that everything else should be put in SyncPoint if possible.  |
|
| Back to top |
|
|
|
|
