| Author |
Message
|
| StrQST400 |
 Posted: Thu Sep 14, 2017 11:34 am Post subject: [ SOLVED ] IIB Large Message Handling |
 |
|
Novice
Joined: 24 Apr 2014
Posts: 12
|
Version: '10.0.0.9'
Product: 'IBM Integration Bus'
Build Number: '344'
IE02 level: 'ie02-L20140415-1143'
IB Level: 'ib1000-L170601.344_P'
Server level: 'S1000-L170601.10408'
Following the developerworks article by David Crichton https://www.ibm.com/developerworks/websphere/library/techarticles/1601_crighton-trs/1601_crighton.html ( Sorry unable to post a direct link to the article)
Using an embedded HTTP listener, we are able to process messages as large as 999MB, as a mediator for FileNet. At 1G this process breaks.
We have a PMR open for the problem and while IBM is looking at the issue, I was wondering if there is a setting somewhere that would allow us to process > 1G message sizes. maxPostSize seems to be ignored while doing chunked HTTP transfers, according to my reading of the Request.java in Tomcat
Any other suggestions of places that I can look at to see why this breaks?
Last edited by StrQST400 on Fri Sep 29, 2017 5:44 am; edited 1 time in total |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Sep 14, 2017 11:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Why exactly are you piping all that data over HTTP?
Or, if you prefer, what exactly do you mean by "mediator"? You may need to get fairly technical.
We use IIB to control file movement into and out of FileNet without any of the files going anywhere near IIB. The request messages (which are SOAP web services and more recently REST APIs) are fairly standard things of a fairly standard size.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| StrQST400 |
| Posted: Thu Sep 14, 2017 11:59 am Post subject: |
|
|
Novice
Joined: 24 Apr 2014
Posts: 12
|
Thank you for your response.
The producing application is in a protected network tier and FileNet sits in an unprotected Tier. The only options they have is to traverse an integration tier.
Mediator: An EAI Pattern where an Integration Solution sits between the source and destination and provides no further value-add other than a session break.
At this point, I am only looking to see if there is a setting that I may have missed anywhere that could cause the blocking of a large message. I have already looked at maxPostSize and my reading of Request.java shows that it is not used when the client is using Transfer-encoding:chunked. A google search does not produce any relevant hits at this message size.
Other than the Knowledge base and this site, any other places that I should go hunting to see if I can find why this breaks? |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 14, 2017 12:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| StrQST400 wrote: |
| The producing application is in a protected network tier and FileNet sits in an unprotected Tier. The only options they have is to traverse an integration tier. |
In your topology possibly. That's not true generally.
| StrQST400 wrote: |
| Mediator: An EAI Pattern where an Integration Solution sits between the source and destination and provides no further value-add other than a session break. |
Thank you.
| StrQST400 wrote: |
At this point, I am only looking to see if there is a setting that I may have missed anywhere that could cause the blocking of a large message.
...
Other than the Knowledge base and this site, any other places that I should go hunting to see if I can find why this breaks? |
I'm not aware of anything, but then 1 Gb is a lot larger than the HTTP traffic I've ever used (and would be blocked by a number of sites, including this one, as a DoS attack!).
Your PMR is the way to go. I'm interested in how this comes out, and hope you post a solution when you find one.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| StrQST400 |
| Posted: Thu Sep 14, 2017 1:33 pm Post subject: |
|
|
Novice
Joined: 24 Apr 2014
Posts: 12
|
| Quote: |
| In your topology possibly. That's not true generally. |
Of course.
I have never attempted this large a message transfer in anything other than a file. But this is an interesting challenge and it gives me an opportunity to peel back the onion, so to speak.
I will definitely bring back whatever I have learnt from the PMR and this exercise.
Thank you! |
|
| Back to top |
|
|
| StrQST400 |
| Posted: Fri Sep 15, 2017 5:54 am Post subject: |
|
|
Novice
Joined: 24 Apr 2014
Posts: 12
|
Update:
Found the issue inside the Tomcat code that IIB uses. There is a method call
| Code: |
| org.apache.tomcat.util.buf.ByteChunk.makeSpace( int count ) |
. This call does this piece
| Code: |
// grow in larger chunks
if( desiredSize < 2 * buff.length ) {
newSize= buff.length * 2;
if( limit >0 &&
newSize > limit ) newSize=limit;
tmp=new byte[newSize];
} else {
newSize= buff.length * 2 + count ;
if( limit > 0 &&
newSize > limit ) newSize=limit;
tmp=new byte[newSize];
}
|
When the size of the request is 1G, this will return 2G which cannot fit in an int. Which means this class will throw a java.lang.NegativeArraySizeException.
I will continue to see if there is a change I can make a change to this class to see if I can fix this. |
|
| Back to top |
|
|
| StrQST400 |
| Posted: Sat Sep 16, 2017 3:01 am Post subject: |
|
|
Novice
Joined: 24 Apr 2014
Posts: 12
|
SOLVED: I am not sure if there is way to mark this thread solved. IBM agrees that there is a problem with that section of Tomcat code and is giving us a patch.
Thank you! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Sep 18, 2017 3:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| StrQST400 wrote: |
| SOLVED: I am not sure if there is way to mark this thread solved. |
The typical way is to edit the subject with something like [ SOLVED ]
It's not a real standard and it's not followed very much.
And it probably doesn't help people coming back in three years asking how you solved it.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
