| Author |
Message
|
| WBIMaestro |
 Posted: Thu Oct 27, 2011 6:14 am Post subject: Post Processing after batching records with FileOutput Node |
 |
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
Hi,
I currently have a simple msg flow to batch MQ records into a file and FTP out. I am using a FTPOutput node for this. I also need to call a system encryption routine for this (gpg). How do I Incorporate this into the flow afte the file has been created, given that the FileOutput Node is an endpoint in itself. Would I need to create another flow?
Thx |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Oct 27, 2011 6:27 am Post subject: Re: Post Processing after batching records with FileOutput N |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| WBIMaestro wrote: |
| How do I Incorporate this into the flow afte the file has been created, given that the FileOutput Node is an endpoint in itself. |
If your FileOutput node doesn't have any output terminals then your installation has a problem. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 27, 2011 6:35 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
How can you encrypt a file that you have already FTPed out?
If you want to encrypt the contents then do so before using the fileoutput node.
You would need to accumulate the MQ messages (these are not records) into memory for this. Or produce one file and process it again as a single lump before the final fileoutput (and FTP). |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 27, 2011 6:46 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| I also need to call a system encryption routine for this (gpg) |
For what? for the ftp transmission i.e. sftp, or for the file content? (it will arrive in encrypted format at the destination)? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 27, 2011 6:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| How can you encrypt a file that you have already FTPed out? |
Valid point. You could call the routine and encrypt a file that the FileOutput has written locally but if the unecrypted version has already been ftp'd what can any piece of software do?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| WBIMaestro |
| Posted: Thu Oct 27, 2011 8:58 am Post subject: |
|
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
| maybe I wasnt clear enough. I understand that I cant encrypt it after its been ftp'ed out. I have a requirement to use pgp to encrypt the file irrespective of wether its sent out by ftp or sftp. Im thinking I would need to use either the job exceution node(not sure) or via a system call in a JCN. my conundrum was that since the records are being accumulated inthe File output node and 'batched' there, I would need to use some other mechanism to do the batching before the file output node. How do I go about that? |
|
| Back to top |
|
|
| WBIMaestro |
| Posted: Thu Oct 27, 2011 9:00 am Post subject: |
|
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
| or just use two flows - one to batch the mq messages and write to a local file and another to read the file, encrypt and (s)ftp out. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Thu Oct 27, 2011 9:03 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
You may need two flows.
Flow 1 will batch up your MQ records into a file and then move the file to a holding directory.
Flow 2 reads files from your holding directory and uses pgp on the file and then (s)ftp's the file out to the final destination.
NOTE: You posted this same thing as I was writing mine.
This would also decouple the batching process from the encryption process, may be important in the future if other places need different encryption. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 27, 2011 9:57 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| joebuckeye wrote: |
You may need two flows.
Flow 1 will batch up your MQ records into a file and then move the file to a holding directory.
Flow 2 reads files from your holding directory and uses pgp on the file and then (s)ftp's the file out to the final destination.
NOTE: You posted this same thing as I was writing mine.
This would also decouple the batching process from the encryption process, may be important in the future if other places need different encryption. |
This may not be feasible. The OP did not specify whether or not he/she was dealing with PCI. It may well be that nothing is allowed to go to the file without being encrypted first...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| WBIMaestro |
| Posted: Thu Oct 27, 2011 10:20 am Post subject: |
|
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
| no, I could write things unencrypted to local file system before encryption. |
|
| Back to top |
|
|
| Bravo |
| Posted: Fri Oct 28, 2011 5:54 am Post subject: |
|
|
Centurion
Joined: 03 Oct 2005
Posts: 146
|
1 flow: connect the File O/p node End of data terminal to the compute node and create a generic XML message to trigger 2 flow
This generic XML message contains source, destination directory etc
2 flow: Use JCN node to use system commands to do encrpytion and sftp/ftp.
I am not sure this is a best design.
_________________
Bravo |
|
| Back to top |
|
|
| WBIMaestro |
| Posted: Fri Oct 28, 2011 9:15 am Post subject: |
|
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
sounds like a design that'll work nevertheless  so let me give it a try |
|
| Back to top |
|
|
| WBIMaestro |
| Posted: Mon Oct 31, 2011 11:56 am Post subject: |
|
|
Acolyte
Joined: 18 Feb 2005
Posts: 53
|
| what gets Propogated to the file outut node end of data terminal? Does the End-of_data Trigger message get propogated there? Looking for ways to use attributes in the XML message content to use for post processing the files output in the file output node.. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 31, 2011 12:01 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| WBIMaestro wrote: |
| what gets Propogated to the file outut node end of data terminal? Does the End-of_data Trigger message get propogated there? |
As indicated here in the description of the end of data terminal:
| Quote: |
| The message received on the Finish File terminal is propagated to this terminal if the file is processed successfully |
 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
