ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportIntegrating IIB with AWS S3

Post new topicReply to topic
Integrating IIB with AWS S3 View previous topic :: View next topic
Author Message
pauline_coffey
PostPosted: Mon Jan 15, 2018 9:06 am Post subject: Integrating IIB with AWS S3 Reply with quote

Newbie

Joined: 05 Dec 2002
Posts: 8
Location: Dublin, Ireland

Hi,
Can anyone point me to resources to integrate from IIB to AWS S3? I just need to upload data from IIB to an S3 bucket. Is it natively supported in IIB or do I need a separate standalone client?

Thanks
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Tue Jan 16, 2018 1:18 am Post subject: Reply with quote

Disciple

Joined: 17 Jul 2017
Posts: 154
Location: UK

Hi Pauline...don't think IIB provides a specific AWS node as such, you will have rely on say HttpRequest to send requests to AWS S3 as per their REST API.

PS: A quick googling suggested that as most AWS services uses SNI (Server Name Indication) the IIB version need to be at 9006/10006 level, see Apar IT14330

http://www-01.ibm.com/support/docview.wss?uid=swg1IT14330
Back to top
View user's profile Send private message
pauline_coffey
PostPosted: Tue Jan 16, 2018 3:07 am Post subject: Reply with quote

Newbie

Joined: 05 Dec 2002
Posts: 8
Location: Dublin, Ireland

Thanks abhi - I had come to the same conclusion since
Back to top
View user's profile Send private message
souciance
PostPosted: Wed Jan 24, 2018 2:19 am Post subject: Re: Integrating IIB with AWS S3 Reply with quote

Disciple

Joined: 29 Jun 2010
Posts: 152

pauline_coffey wrote:
Hi,
Can anyone point me to resources to integrate from IIB to AWS S3? I just need to upload data from IIB to an S3 bucket. Is it natively supported in IIB or do I need a separate standalone client?

Thanks


I basically wrote the S3 integration in a java compute node. There is no support for this natively. The tricky part is to do the AWS authentication otherwise its pretty ok.
Back to top
View user's profile Send private message
pauline_coffey
PostPosted: Thu Jan 25, 2018 1:33 am Post subject: Reply with quote

Newbie

Joined: 05 Dec 2002
Posts: 8
Location: Dublin, Ireland

Thanks - I have since implemented the integration form a Java node using the AWS SDK. Do you have any tips on caching the authentication details securely for usage in the compute node? I also have a concern that the upload action could cause a bottleneck in my broker processing if I have thousands of files to be uploaded and so wondered if I should instead do the upload once the broker process has completed (e.g. write IIB processed data to filesystem and then perform the AWS upload outside of the IIB transaction).
Back to top
View user's profile Send private message
souciance
PostPosted: Thu Jan 25, 2018 10:54 am Post subject: Reply with quote

Disciple

Joined: 29 Jun 2010
Posts: 152

Well it kind of depends on how secure you want your details to be.

You could go for the easy option and put the keys and secrets as user defined properties and then override them at deploy time for your different environments. I would think this is ok. There is a global cache in IIB but I am not sure how flexible or "user friendly" it is to work with. I would probably stick with a user defined property.

As for sending lots of files, why don't you consume them from a queue. That way one messages are only consumed when the previous one is dealt with. That should take the load of your broker and if something goes wrong you can backout the message to a backout queue.
Back to top
View user's profile Send private message
pauline_coffey
PostPosted: Fri Jan 26, 2018 1:52 am Post subject: Reply with quote

Newbie

Joined: 05 Dec 2002
Posts: 8
Location: Dublin, Ireland

Thanks again for the reply. My IIB input is a singe large message from a JMS bridge which I need to subdivide into (potentially) thousands of discrete files for upload to AWS. While the broker can parse the input and chunk the data for output without issue, the 1 to Many nature of the IIB flow means that broker throughput is wholly dependant on the AWS upload performance. Hence my thinking that I should output the flies to file system first for an async upload.
Back to top
View user's profile Send private message
souciance
PostPosted: Fri Jan 26, 2018 4:14 am Post subject: Reply with quote

Disciple

Joined: 29 Jun 2010
Posts: 152

pauline_coffey wrote:
Thanks again for the reply. My IIB input is a singe large message from a JMS bridge which I need to subdivide into (potentially) thousands of discrete files for upload to AWS. While the broker can parse the input and chunk the data for output without issue, the 1 to Many nature of the IIB flow means that broker throughput is wholly dependant on the AWS upload performance. Hence my thinking that I should output the flies to file system first for an async upload.


Perhaps you could look into saving the file directly from the jms queue to a filesystem without parsing it. Then if the records are structured you can split the file during read in your FileInput to reduce parsing and performance load?
Back to top
View user's profile Send private message
pauline_coffey
PostPosted: Fri Jan 26, 2018 4:33 am Post subject: Reply with quote

Newbie

Joined: 05 Dec 2002
Posts: 8
Location: Dublin, Ireland

Yes, there's merit in this idea - thanks again
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportIntegrating IIB with AWS S3
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.