| Author |
Message
|
| pauline_coffey |
 Posted: Mon Jan 15, 2018 9:06 am Post subject: Integrating IIB with AWS S3 |
 |
|
Novice
Joined: 05 Dec 2002
Posts: 15
Location: Dublin, Ireland
|
Hi,
Can anyone point me to resources to integrate from IIB to AWS S3? I just need to upload data from IIB to an S3 bucket. Is it natively supported in IIB or do I need a separate standalone client?
Thanks |
|
| Back to top |
|
 |
| abhi_thri |
| Posted: Tue Jan 16, 2018 1:18 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
Hi Pauline...don't think IIB provides a specific AWS node as such, you will have rely on say HttpRequest to send requests to AWS S3 as per their REST API.
PS: A quick googling suggested that as most AWS services uses SNI (Server Name Indication) the IIB version need to be at 9006/10006 level, see Apar IT14330
http://www-01.ibm.com/support/docview.wss?uid=swg1IT14330 |
|
| Back to top |
|
|
| pauline_coffey |
| Posted: Tue Jan 16, 2018 3:07 am Post subject: |
|
|
Novice
Joined: 05 Dec 2002
Posts: 15
Location: Dublin, Ireland
|
Thanks abhi - I had come to the same conclusion since  |
|
| Back to top |
|
|
| souciance |
| Posted: Wed Jan 24, 2018 2:19 am Post subject: Re: Integrating IIB with AWS S3 |
|
|
Disciple
Joined: 29 Jun 2010
Posts: 169
|
| pauline_coffey wrote: |
Hi,
Can anyone point me to resources to integrate from IIB to AWS S3? I just need to upload data from IIB to an S3 bucket. Is it natively supported in IIB or do I need a separate standalone client?
Thanks |
I basically wrote the S3 integration in a java compute node. There is no support for this natively. The tricky part is to do the AWS authentication otherwise its pretty ok. |
|
| Back to top |
|
|
| pauline_coffey |
| Posted: Thu Jan 25, 2018 1:33 am Post subject: |
|
|
Novice
Joined: 05 Dec 2002
Posts: 15
Location: Dublin, Ireland
|
| Thanks - I have since implemented the integration form a Java node using the AWS SDK. Do you have any tips on caching the authentication details securely for usage in the compute node? I also have a concern that the upload action could cause a bottleneck in my broker processing if I have thousands of files to be uploaded and so wondered if I should instead do the upload once the broker process has completed (e.g. write IIB processed data to filesystem and then perform the AWS upload outside of the IIB transaction). |
|
| Back to top |
|
|
| souciance |
| Posted: Thu Jan 25, 2018 10:54 am Post subject: |
|
|
Disciple
Joined: 29 Jun 2010
Posts: 169
|
Well it kind of depends on how secure you want your details to be.
You could go for the easy option and put the keys and secrets as user defined properties and then override them at deploy time for your different environments. I would think this is ok. There is a global cache in IIB but I am not sure how flexible or "user friendly" it is to work with. I would probably stick with a user defined property.
As for sending lots of files, why don't you consume them from a queue. That way one messages are only consumed when the previous one is dealt with. That should take the load of your broker and if something goes wrong you can backout the message to a backout queue. |
|
| Back to top |
|
|
| pauline_coffey |
| Posted: Fri Jan 26, 2018 1:52 am Post subject: |
|
|
Novice
Joined: 05 Dec 2002
Posts: 15
Location: Dublin, Ireland
|
| Thanks again for the reply. My IIB input is a singe large message from a JMS bridge which I need to subdivide into (potentially) thousands of discrete files for upload to AWS. While the broker can parse the input and chunk the data for output without issue, the 1 to Many nature of the IIB flow means that broker throughput is wholly dependant on the AWS upload performance. Hence my thinking that I should output the flies to file system first for an async upload. |
|
| Back to top |
|
|
| souciance |
| Posted: Fri Jan 26, 2018 4:14 am Post subject: |
|
|
Disciple
Joined: 29 Jun 2010
Posts: 169
|
| pauline_coffey wrote: |
| Thanks again for the reply. My IIB input is a singe large message from a JMS bridge which I need to subdivide into (potentially) thousands of discrete files for upload to AWS. While the broker can parse the input and chunk the data for output without issue, the 1 to Many nature of the IIB flow means that broker throughput is wholly dependant on the AWS upload performance. Hence my thinking that I should output the flies to file system first for an async upload. |
Perhaps you could look into saving the file directly from the jms queue to a filesystem without parsing it. Then if the records are structured you can split the file during read in your FileInput to reduce parsing and performance load? |
|
| Back to top |
|
|
| pauline_coffey |
| Posted: Fri Jan 26, 2018 4:33 am Post subject: |
|
|
Novice
Joined: 05 Dec 2002
Posts: 15
Location: Dublin, Ireland
|
| Yes, there's merit in this idea - thanks again |
|
| Back to top |
|
|
| Rajasekhar143 |
| Posted: Mon Aug 02, 2021 12:29 am Post subject: |
|
|
Newbie
Joined: 30 Jul 2021
Posts: 7
|
Hi , I have been provided below details and asked to upload the files to AWS S3 bucket using IIB.
"S3BucketName":
"S3AccessKey":
"S3SecretKey":
"CloudFrontUrl":
"S3Region":
Can somone help me here the best way of implementing this in IIB? Is there any bult-in feature to connect AWS S3 via CloudFront or can we use HTTPRequest node ? |
|
| Back to top |
|
|
|
|
