| Author |
Message
|
| marko.pitkanen |
 Posted: Sun Mar 06, 2011 11:59 pm Post subject: HTTPRequest -node max message size? |
 |
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi,
I'm sorry if I'm asking this again. I did a quick search but didn't find the answer.
I know that there are perhaps clever ways to design this but I have found an integration where broker provides WS for the portal and ask some data from the legacy system through http -request. With some requests the amount of the returned data from the legacy seems to become quite a large.
It seems that there is 100 Mb size limit for the responses for the HTTPRequest, because broker uses internally WMQ queue while processing.
Is there any possibilities to configure broker to use segmentation or similar to handle larger responses for the HTTPRequests?
--
Marko |
|
| Back to top |
|
 |
| Esa |
| Posted: Mon Mar 07, 2011 12:30 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Moi Marko,
I think sending such large messages over http does not sound like a good idea. Neither does assemling segmented over 100 MB messages together in MB for processing...
You should try to figure out if you can process the messages in smaller chunks, the same way you can do with FileInput nodes.
If the messages can be processed in chunks and you can change the http client side of the application, you could segment the messages on the client side and add a couple of yor own HTTP headers like GroupID and LastMsgOfTheGroup...
Just a wild idea, I havent tested if it works.
Esa |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 07, 2011 3:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm not sure that HTTPRequest node uses a queue internally.
Have you demonstrated a hard limit for 100MB? |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Mar 07, 2011 4:13 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
I'm not sure that HTTPRequest node uses a queue internally.
|
I agree, there should not be any reason for HttpRequest to use MQ queues.
It seems I did not read the question carefully enough. My reply above applies to HttpInput node, not HttpReply. I got distracted by the reference to internal queues. |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Mon Mar 07, 2011 4:29 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi,
Flow works as following: HTTPInput->Compute->HTTPRequest-->Compute-->HTTPReply
97 Mb response message comes through broker to the portal, but for example 122 Mb response from legacy won't come at least to the portal and from system log can be found error number BIP3117 according to SYSTEM.BROKER.WS queues.
So it can also be that HTTPReply can't communicate with http -listener (put huge message to the internal queue).
I'll try to reprocess the error and found more detailed description what happened.
--
Marko |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Mar 07, 2011 4:40 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| marko.pitkanen wrote: |
Hi,
Flow works as following: HTTPInput->Compute->HTTPRequest-->Compute-->HTTPReply
97 Mb response message comes through broker to the portal, but for example 122 Mb response from legacy won't come at least to the portal and from system log can be found error number BIP3117 according to SYSTEM.BROKER.WS queues.
So it can also be that HTTPReply can't communicate with http -listener (put huge message to the internal queue).
I'll try to reprocess the error and found more detailed description what happened.
--
Marko |
Marko,
Yes, the problem is that HttpReply cannot put the oversize message into SYSTEM.BROKER.WS.REPLY
HttpReply is just a specialized MQOutput. You could try to use MQOutput to put a similar message to SYSTEM.BROKER.WS.REPLY with message flag MQMF_SEGMENTATION_ALLOWED. Will you be successfull, it depends on the way the httpListener reads messages from the reply queue. |
|
| Back to top |
|
|
| paintpot |
| Posted: Mon Mar 07, 2011 5:10 am Post subject: |
|
|
Centurion
Joined: 19 Sep 2005
Posts: 112
Location: UK
|
Are you using HTTP compression? This might alter the limits quite a bit, if you aren't  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 07, 2011 5:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| In Broker v7, you can configure the HTTPInput/HTTPReply node to NOT use the bipHTTPListener, and instead use the SOAP listener, which runs inside the EG process and does NOT use MQ queues... |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Mon Mar 07, 2011 6:08 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Thanks for all help.
I managed to reprocess the exception and got better error message from user trace.
| Code: |
The HTTP reply node 'XX.HTTP ReplyXX' attempted to put a message to the specified queue ''SYSTEM.BROKER.WS.REPLY'' connected to queue manager ''XXX''. The MQCC was 2 and the MQRC was 2031.
A message flow node failed to write to the WebSphere MQ queue 'SYSTEM.BROKER.WS.REPLY' owned by queue manager 'XXX' with the resulting WebSphere MQ completion and reason codes of 2 and 2031. This queue is used for internal communication between the listener process and the message flow. |
We will try mqjeff's advice after flow has been migrated to the v7 broker env.
--
Marko |
|
| Back to top |
|
|
| WMBEAI |
| Posted: Tue Aug 23, 2011 1:38 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2009
Posts: 66
|
Hi.
I see that my HTTP Response i get out of the HTTP Request node itself is truncated.
will it be a similar problem like MAX message size?
Can you tell me where and how did you got the trace mentioned above? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Aug 23, 2011 2:00 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
marko stated that it was taken
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| prasadpav |
| Posted: Thu Mar 15, 2012 6:04 am Post subject: |
|
|
Centurion
Joined: 03 Oct 2004
Posts: 142
|
mqjeff wrote:
| Quote: |
| In Broker v7, you can configure the HTTPInput/HTTPReply node to NOT use the bipHTTPListener, and instead use the SOAP listener, which runs inside the EG process and does NOT use MQ queues... |
May be I'm asking the obvious question, are there any message size limitations when SOAP listeners embedded within execution group is used in both sending & receiving responses? I don't see any such parameters (configurable), hence, I assume there is no such limitation. I understand there are other things to consider like JVMheapsize and others, just want to check only on the message size.
Also, do anyone have strong opinions to use SwA for large messages instead of just SOAP message? Relative merits/demerits..etc.
Thanks in advance. |
|
| Back to top |
|
|
| bloomy |
| Posted: Thu Mar 15, 2012 1:28 pm Post subject: |
|
|
Acolyte
Joined: 11 Feb 2009
Posts: 61
Location: Bloomington, IL USA
|
mqjeff wrote:
| Quote: |
| In Broker v7, you can configure the HTTPInput/HTTPReply node to NOT use the bipHTTPListener, and instead use the SOAP listener, which runs inside the EG process and does NOT use MQ queues... |
Can you please confirm on this, I read and was told by Hursely during the beta testing for V8 that it is a feature in WMB V8 and not in V7. In V7 HTTPInput/HTTPReply still use the httplistener.
_________________
“Design is not just what it looks like and feels like. Design is how it works.”
~ Steve Jobs |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Mar 16, 2012 5:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bloomy wrote: |
mqjeff wrote:
| Quote: |
| In Broker v7, you can configure the HTTPInput/HTTPReply node to NOT use the bipHTTPListener, and instead use the SOAP listener, which runs inside the EG process and does NOT use MQ queues... |
Can you please confirm on this, I read and was told by Hursely during the beta testing for V8 that it is a feature in WMB V8 and not in V7. In V7 HTTPInput/HTTPReply still use the httplistener. |
Many different teams of people work at the IBM labs in Hursley. Were you told by someone from the Broker team? If so, you might ask them to explain this page. http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fbc43700_.htm
And then, too, remember that many of the people on the Broker team in Hursley do not actually do anything with the HTTP components, and so they might not be aware of which features were part of what release. |
|
| Back to top |
|
|
| bloomy |
| Posted: Fri Mar 16, 2012 12:51 pm Post subject: |
|
|
Acolyte
Joined: 11 Feb 2009
Posts: 61
Location: Bloomington, IL USA
|
mqjeff:
Thank you for the response and the link. This is news for me, was waiting for WMB V8 implementation at our place to implement this. Will try it out on WMB 7. Thanks!
_________________
“Design is not just what it looks like and feels like. Design is how it works.”
~ Steve Jobs |
|
| Back to top |
|
|
|
|
