| Author |
Message
|
| magvlvri |
 Posted: Mon Jul 27, 2015 6:37 am Post subject: IIB CDoutput Node vs Connect Direct scripts |
 |
|
Apprentice
Joined: 07 Nov 2014
Posts: 26
|
Hi,
I am trying to do a proof of concept on the CDOutput node for file-transfers , both inbound and outbound, as we are trying to move away from the native connect direct scripts, for better integration ith the IIB.
I am trying to send a file from IIB server to a OS390 box, with the right entries in place on Netmap and UserConfig files on both servers.
I have a message flow that is trying to send a file through with CDOutput node and the message flow is successfully sending the file the local connect direct server(which is running on the same machine as IIB).
However, i see the file seems to be stuck on the local connect direct server and is not making it to the target OS390 box.
I did a trace on the IIB server and below is what it shows..
I am seeing that IIB is handing off the file to the local connect direct server but the file is stuck in the transfers folder below.
No error reported by IIB and the file is stuck on connect direct folder, is now making me think if the CD nodes really have the same capability as the connect direct scripts. I am sure i might be missing an entry on netmap on the target server which might be causing this issue but issues like these are common and the cd node must be actually reporting back if the file is in transit or if it reached the target server.
In my experience, i have seen the connect direct scripts actually giving the accuarate return code that tells if the file reached the target server or not, which doesnt seem to the case with CD nodes.
Please share your comments.
6445 UserTrace BIP7954I: CDOutput node ''CD Output'' in message flow ''CDOutput'' has successfully sent a command to the IBM Sterling Connect:Direct server ''LOCALCDSERVER'' to start the transfer of the file to the remote system. Process script used: ''/PQ11171 PROCESS
SNODE=MAINFRAMENETMAP
PACCT="WebSphere Message Broker=LOCALIIBSERVER execution group=SAMPLEEG messageflow=CDOutput messageflow node=CD Output"
SACCT="WebSphere Message Broker=LOCALIIBSERVER, execution group=SAMPLEEG messageflow=CDOutput messageflow node=CD Output"
COPYFILE COPY
FROM (
FILE="/var/mqsi/common/CD/LOCALIIBSERVER/SAMPLEEG/Transfers/CDOutput/CD Output/MAINFRAMEDATASET"
SYSOPTS=":DATATYPE=text:STRIP.BLANKS=no:XLATE=yes:" )
TO (
FILE="MAINFRAMEDATASET"
DISP=NEW
SYSOPTS=":DATATYPE=text:STRIP.BLANKS=no:XLATE=yes:" )
CLEARUP if (COPYFILE LE 4 ) then
RUN TASK PNODE (PGM=UNIX)
sysopts="rm /var/mqsi/common/CD/LOCALIIBSERVER/SAMPLEEG/Transfers/CDOutput/CD Output/MAINFRAMEDATASET"
eif
PEND''.
None.
None.
6445 UserTrace BIP12057I: Sent the file ''MAINFRAMEDATASET''.
The CDOutput node ''CD Output'' has successfully sent a command to the primary IBM Sterling Connect:Direct server (PNODE) ''LOCALCDSERVER'' to transfer the file ''MAINFRAMEDATASET'' to the secondary IBM Sterling Connect:Direct server (SNODE) ''MAINFRAMENETMAP''. The correlating Connect:Direct process is ''PQ11171'', process number ''5''. You can use the Connect:Direct statistics to get further information about the process.
6445 UserTrace BIP11506I: Committed a local transaction.
A local transaction has been committed for work done on the message flow thread. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon Jul 27, 2015 6:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
CD is a hub/spoke archicture, as far as I know.
That means that once it gets to the CD Server, it stops being the responsibility of the sending application.
So the issue isn't "why is broker not sending the file to the mainframe", the issue is "why is CD not forwarding the file to the mainframe". Or "Why is the mainframe not pulling the file from the CD server". |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 27, 2015 6:49 am Post subject: Re: IIB CDoutput Node vs Connect Direct scripts |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| magvlvri wrote: |
| No error reported by IIB and the file is stuck on connect direct folder, is now making me think if the CD nodes really have the same capability as the connect direct scripts. I am sure i might be missing an entry on netmap on the target server which might be causing this issue but issues like these are common and the cd node must be actually reporting back if the file is in transit or if it reached the target server. |
Where in the IIB documentation does it say that the CDOutput node behaves like this? Why "must" it be reporting back?
| magvlvri wrote: |
| In my experience, i have seen the connect direct scripts actually giving the accuarate return code that tells if the file reached the target server or not, which doesnt seem to the case with CD nodes. |
Yes, the Connect Direct server (which the script is manipulating) does report errors.
| magvlvri wrote: |
| Please share your comments. |
See below:
| magvlvri wrote: |
6445 UserTrace BIP12057I: Sent the file ''MAINFRAMEDATASET''.
The CDOutput node ''CD Output'' has successfully sent a command to the primary IBM Sterling Connect:Direct server (PNODE) ''LOCALCDSERVER'' to transfer the file ''MAINFRAMEDATASET'' to the secondary IBM Sterling Connect:Direct server (SNODE) ''MAINFRAMENETMAP''. The correlating Connect:Direct process is ''PQ11171'', process number ''5''. You can use the Connect:Direct statistics to get further information about the process.
|
So IIB has successfully sent a command to Connect:Direct to initiate a managed file transfer, and now expects Connect:Direct to manage it. It doesn't monitor the transfer any more than it checks to see if an MQ message made it off a transmission queue.
So yes, if you run a script to perform an interactive transfer via Connect:Direct you get a return code. If you set up a managed transfer you don't, you have to look in the Connect:Direct client to see the status of the transfers.
Functioning as designed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| nelson |
| Posted: Mon Jul 27, 2015 7:10 am Post subject: |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
If the process was submitted correctly to the CD Server, it's OK for broker (see the out terminal description of CDOutput node):
| Quote: |
| The message received on the In terminal is propagated to this terminal if the record is written successfully. The message is unchanged except for status information in the Local Environment. |
And, as you can see, the only information that you get after that is:
| Quote: |
LocalEnvironment.WrittenDestination.CD:
ProcessName CHARACTER The name of the process that is sending the file.
ProcessNumber CHARACTER The number of the process that is sending the file.
Directory CHARACTER Absolute directory path of the output directory in the form that is used by the file system of the broker. For example, on Windows systems, this directory starts with the drive letter prefix (such as C: ).
Name CHARACTER File name of the output file.
PrimaryNodeName CHARACTER The name of the primary Connect:Direct server (PNODE)
PrimaryNodeOS CHARACTER The operating system of the primary Connect:Direct server
SecondaryNodeName CHARACTER The name of the secondary Connect:Direct server (SNODE)
SecondaryNodeOS CHARACTER The operating system of the secondary Connect:Direct server (this might not be the same as the IBM Integration Bus operating system) |
So, as Vitor pointed.. check the pnode/snode statistics to see what happened. |
|
| Back to top |
|
|
| magvlvri |
| Posted: Mon Jul 27, 2015 7:29 am Post subject: |
|
|
Apprentice
Joined: 07 Nov 2014
Posts: 26
|
Thanks for your inputs everyone.
I guess the basic premise that we could use IIB to manage the end-to-end file transfer using the CD nodes was wrong.
So, is it safe to say that the connect direct server still holds the control over whether the file reached the target server or not? or atleast the status of the transfer?
Is there a fool-proof way to have IIB monitor the file transfer process and say report back, once the file actually made it to the target server. There is actually a requirement to notify an application or group, once the file actually made it to the target server. I would like to know if it can be achieved using IIB.
We would want to move away from using scripts and have this control built into the IIB. If it cannot be achieved today, is there a RFE for something like this?
Do share your comments, if IIB is not the right place to have something like this. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 27, 2015 7:38 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| magvlvri wrote: |
| I guess the basic premise that we could use IIB to manage the end-to-end file transfer using the CD nodes was wrong. |
Yes it was.
| magvlvri wrote: |
| So, is it safe to say that the connect direct server still holds the control over whether the file reached the target server or not? or atleast the status of the transfer? |
As a piece of software that performs a managed file transfer, it's safe to say that the software holds control (or "manages") the transfer.
| magvlvri wrote: |
| Is there a fool-proof way to have IIB monitor the file transfer process and say report back, once the file actually made it to the target server. |
No.
| magvlvri wrote: |
| There is actually a requirement to notify an application or group, once the file actually made it to the target server. |
This is always a requirement. One of the aspects of management is to manage when it doesn't work.
| magvlvri wrote: |
| I would like to know if it can be achieved using IIB. |
It can't.
| magvlvri wrote: |
| We would want to move away from using scripts and have this control built into the IIB. |
Why? Or more specifically, why IIB? Why should a piece of transformation software be managing file transfers? Why not a piece of home brew that's using the Connect:Direct APIs if you don't like the native Connect:Direct client (which I admit is slightly naff)?
| magvlvri wrote: |
| If it cannot be achieved today, is there a RFE for something like this? |
Not to my knowledge, if there was one I wouldn't vote for it and I doubt any such RFE would ever be taken up. Because:
| magvlvri wrote: |
| Do share your comments, if IIB is not the right place to have something like this. |
IBM think Stirling File Gateway is the best place to hold the management function you're describing and market it as such. I don't see IBM building this function into IIB when they have a competing product.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jul 27, 2015 7:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If CD can emit a message of some kind - an http post, an mq message, etc... - when it delivers the file. Then yes, you can code IIB to process that and notify people.
Not the best place to do it, however. Unless the notification has to go to a business process, not a set of people. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 27, 2015 7:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Vitor wrote: |
| I don't see IBM building this function into IIB when they have a competing product. |
Ok, let me recant slightly. They might build it as a taster, in the same way that a shaved down eXtreme Scale is built into IIB; the expectation is that you'll outgrow it and buy the full version.
But eXtreme Scale adds to the value of IIB. I still doubt IBM would want IIB to turn into a management tool.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| magvlvri |
| Posted: Mon Jul 27, 2015 8:03 am Post subject: |
|
|
Apprentice
Joined: 07 Nov 2014
Posts: 26
|
I agree that connect direct is the actual place where the transfers are managed, however be it IIB or some other product, they would like to know the status on the file transfer.
When we have cdinput and cdoutput nodes that help IIB to pickup and push files to Connect direct, i am thinking, one should also provide a way for IIB to receive statistics from the connect direct server.
Asking for a mq message or http post could be a little too much, but atleast there should be some way to track the file transfer using the processid or statistics.
Anyway, i got an idea on the current capability, thanks to everyone. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 27, 2015 8:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| magvlvri wrote: |
| atleast there should be some way to track the file transfer using the processid or statistics. |
There is; via Stirling File Gateway or the Connect:Direct client.
Why should IIB track a file transfer it's handed off to another piece of software? It doesn't track MQ messages (to repeat my example of above).
Also, why is this your (in an IIB sense) problem? Why is the management of Connect:Direct transfers not the problem of whoever owns the Connect:Direct software? When I said:
| Vitor wrote: |
| magvlvri wrote: |
| There is actually a requirement to notify an application or group, once the file actually made it to the target server. |
This is always a requirement. One of the aspects of management is to manage when it doesn't work. |
I meant that there is always a requirement to notify an application or group with the expectation that they will then do something about the problem.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 27, 2015 8:25 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| magvlvri wrote: |
| they would like to know the status on the file transfer. |
Tell them they shouldn't write tightly coupled applications 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| magvlvri |
| Posted: Mon Jul 27, 2015 9:01 am Post subject: |
|
|
Apprentice
Joined: 07 Nov 2014
Posts: 26
|
Sterling control center seems to have the capability to monitor the activity on the connect direct server and emits jms events which could be potentially be consumed by IIB and report to the appropriate application.
Even though its a long shot, atleast there seems to be a way for emitting events that represent the status of file transfers.
The SCC product is already in use by the b2b commerce group in the enterprise, is this something that can be worth exploring? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 27, 2015 9:28 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| magvlvri wrote: |
| Sterling control center seems to have the capability to monitor the activity on the connect direct server and emits jms events which could be potentially be consumed by IIB and report to the appropriate application. |
I still question the utility of a design where individual applications are either dependent on knowing if a file has been transferred and/or doing something about it when it isn't.
| magvlvri wrote: |
| The SCC product is already in use by the b2b commerce group in the enterprise, is this something that can be worth exploring? |
Yes:
| Quote: |
| IBM® Control Center tracks the critical events across your B2B and managed file transfer (MFT) software for improved operations, customer service and B2B governance |
(the name changed from Stirling Control Center to IBM Control Center from v6 for the same reason we're now talking about IBM Integration Bus not Websphere Message Broker)
If anything this has rather more capabilities than you describe in your requirements. We (this site) uses SFG to manage our Connect:Direct very successfully (and it's cheaper).
This is of course not an issue for you as this b2b commerce group has already paid. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| nelson |
| Posted: Mon Jul 27, 2015 1:01 pm Post subject: Re: IIB CDoutput Node vs Connect Direct scripts |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
| magvlvri wrote: |
| ..we are trying to move away from the native connect direct scripts, for better integration ith the IIB.. |
From this we can assume that none transformation/enrichment is taking place within IIB. So... a couple of questions:
How do you actually trigger the file transfers? Why not to use a Connect:Direct client like IBM Sterling Connect:Direct File Agent for these purposes? |
|
| Back to top |
|
|
|
|
