| Author |
Message
|
| souciance |
 Posted: Mon Oct 02, 2017 10:47 am Post subject: Feedback on handling large files via HTTP |
 |
|
Disciple
Joined: 29 Jun 2010
Posts: 169
|
Hello,
I wanted to get some feedback on the following requirements.
1. We are required to download some files via some urls so it will be HTTP GET calls.
2. These files can vary from a couple of MB to say 500 MB or even 1 GB in size.
3. We are not required to parse them. There is no mapping involved.
4. We are required to store them on disk.
My initial thought is to do a HTTP GET and ask for chunked encoding to get some form of streaming going. This is then connected to a File outbound node.
The main concern I have is whether the streaming will go all the way from the HTTP GET to the file writer or will there be some buffering some where along the way?
Do you foresee any other risks to take into consideration?
Please note, I do realize handling large files over HTTP is bad practise, but we are constraints by the requirements in the project and there is no away around it for now. |
|
| Back to top |
|
 |
| souciance |
| Posted: Mon Oct 02, 2017 10:58 am Post subject: |
|
|
Disciple
Joined: 29 Jun 2010
Posts: 169
|
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 02, 2017 11:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| souciance |
| Posted: Mon Oct 02, 2017 12:45 pm Post subject: |
|
|
Disciple
Joined: 29 Jun 2010
Posts: 169
|
| Well a workaround for us will most likely be lightweight Apache Camel app using Apache's http async client which is more streamlined for streaming downloads. We'll then somehow trigger the Camel app via IIB. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Oct 03, 2017 1:09 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I am frequently amazed by the (cultural?) reluctance of some people to push back on a unrealistic "requirement".
Quite often there is an alternative but no-one has bothered to ask for it.
If you can't do it elegantly with the many built-in functions of IIB, then raise a RFE to get the feature added, in the meantime use something else outside of IIB.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| souciance |
| Posted: Tue Oct 03, 2017 5:07 am Post subject: |
|
|
Disciple
Joined: 29 Jun 2010
Posts: 169
|
| zpat wrote: |
I am frequently amazed by the (cultural?) reluctance of some people to push back on a unrealistic "requirement".
Quite often there is an alternative but no-one has bothered to ask for it.
If you can't do it elegantly with the many built-in functions of IIB, then raise a RFE to get the feature added, in the meantime use something else outside of IIB. |
I agree completely, but a lot of the times in projects at large enterprises, developers are likely to be the last person to find out about high level requirements. By that time its almost too late to push back as a lot of the decisions have been made.
In this case, this is a new requirement being added in a project where pushing back hasn't resulted in much.
I will have to add an RFE then. |
|
| Back to top |
|
|
|
|
