ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportFeedback on handling large files via HTTP

Post new topicReply to topic
Feedback on handling large files via HTTP View previous topic :: View next topic
Author Message
souciance
PostPosted: Mon Oct 02, 2017 10:47 am Post subject: Feedback on handling large files via HTTP Reply with quote

Voyager

Joined: 29 Jun 2010
Posts: 83

Hello,

I wanted to get some feedback on the following requirements.

1. We are required to download some files via some urls so it will be HTTP GET calls.

2. These files can vary from a couple of MB to say 500 MB or even 1 GB in size.

3. We are not required to parse them. There is no mapping involved.

4. We are required to store them on disk.

My initial thought is to do a HTTP GET and ask for chunked encoding to get some form of streaming going. This is then connected to a File outbound node.

The main concern I have is whether the streaming will go all the way from the HTTP GET to the file writer or will there be some buffering some where along the way?

Do you foresee any other risks to take into consideration?

Please note, I do realize handling large files over HTTP is bad practise, but we are constraints by the requirements in the project and there is no away around it for now.
Back to top
View user's profile Send private message
souciance
PostPosted: Mon Oct 02, 2017 10:58 am Post subject: Reply with quote

Voyager

Joined: 29 Jun 2010
Posts: 83

By the way this thread http://www.mqseries.net/phpBB2/viewtopic.php?t=72164&sid=48a7fa849fa0749fbeb9ec694c43044f says that the HTTP request nodes do no support streaming. So, I am guessing that solution is already heading for the dustbin?
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Oct 02, 2017 11:02 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24648
Location: Ohio, USA

souciance wrote:
By the way this thread http://www.mqseries.net/phpBB2/viewtopic.php?t=72164&sid=48a7fa849fa0749fbeb9ec694c43044f says that the HTTP request nodes do no support streaming. So, I am guessing that solution is already heading for the dustbin?



_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
souciance
PostPosted: Mon Oct 02, 2017 12:45 pm Post subject: Reply with quote

Voyager

Joined: 29 Jun 2010
Posts: 83

Well a workaround for us will most likely be lightweight Apache Camel app using Apache's http async client which is more streamlined for streaming downloads. We'll then somehow trigger the Camel app via IIB.
Back to top
View user's profile Send private message
zpat
PostPosted: Tue Oct 03, 2017 1:09 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5576
Location: UK

I am frequently amazed by the (cultural?) reluctance of some people to push back on a unrealistic "requirement".

Quite often there is an alternative but no-one has bothered to ask for it.

If you can't do it elegantly with the many built-in functions of IIB, then raise a RFE to get the feature added, in the meantime use something else outside of IIB.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
souciance
PostPosted: Tue Oct 03, 2017 5:07 am Post subject: Reply with quote

Voyager

Joined: 29 Jun 2010
Posts: 83

zpat wrote:
I am frequently amazed by the (cultural?) reluctance of some people to push back on a unrealistic "requirement".

Quite often there is an alternative but no-one has bothered to ask for it.

If you can't do it elegantly with the many built-in functions of IIB, then raise a RFE to get the feature added, in the meantime use something else outside of IIB.


I agree completely, but a lot of the times in projects at large enterprises, developers are likely to be the last person to find out about high level requirements. By that time its almost too late to push back as a lot of the decisions have been made.

In this case, this is a new requirement being added in a project where pushing back hasn't resulted in much.

I will have to add an RFE then.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportFeedback on handling large files via HTTP
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.