| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| WMB file processing and SOAP connection errors... |
« View previous topic :: View next topic » |
| Author |
Message
|
| Allan Mackay |
 Posted: Fri Sep 07, 2012 2:18 am Post subject: WMB file processing and SOAP connection errors... |
 |
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
Hi,
I'm seeking advice on an issue I have with a message flow...
The message flow, basically, has a FileInput Node, for a file with 1000s of records. I'm processing 1 record at a time, transforming the content and invoking a web service via a SOAP Request node/SOAP Extract node pair (so a synchronous invocation) before sending the results on via an MQOuput node.
The problem lies with the web service invocation. After a number of records are process (circa 1300), I start getting errors which are, ultimately, "java.net.SocketException: Too many open files". This occurs for a block of records (150-200 or so) then, magically, clears and processing carries on as normal.
Whilst I understand that this particular error is to do with the number of 'open connections' to the web service, I don't believe I have any control on the opening and closing... do I? And given that, what can I do to circumvent this issue?
Any advice, derogatory or otherwise greatly received on this one 
Many thanks in advance,
-Allan. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Sep 07, 2012 2:37 am Post subject: Re: WMB file processing and SOAP connection errors... |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Allan Mackay wrote: |
Hi,
I'm seeking advice on an issue I have with a message flow...
The message flow, basically, has a FileInput Node, for a file with 1000s of records. I'm processing 1 record at a time, transforming the content and invoking a web service via a SOAP Request node/SOAP Extract node pair (so a synchronous invocation) before sending the results on via an MQOuput node.
The problem lies with the web service invocation. After a number of records are process (circa 1300), I start getting errors which are, ultimately, "java.net.SocketException: Too many open files". This occurs for a block of records (150-200 or so) then, magically, clears and processing carries on as normal.
Whilst I understand that this particular error is to do with the number of 'open connections' to the web service, I don't believe I have any control on the opening and closing... do I? And given that, what can I do to circumvent this issue?
Any advice, derogatory or otherwise greatly received on this one
Many thanks in advance,
-Allan. |
Have you tried splitting this into 2 flows
Flow 1 = file input to Queue
Flow 2 = Queue to SOAP call to Queue
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Sep 07, 2012 4:06 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Is the FileInput node processing the Whole File and then you are splitting it into records? Or is the FileInput splitting it into records?
You might get a different picture if you let the FileInput node split the file into records and then use multiple instances of the flow to process several records at the same time. |
|
| Back to top |
|
|
| Allan Mackay |
| Posted: Fri Sep 07, 2012 5:29 am Post subject: |
|
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
The FileInput node is splitting the file into records, the flow is dealing with one record at a time as a single 'messsage'.
Sorry if I seem a little dense here (it's Friday afternoon after one heck of a week - that's my excuse and I'm sticking  )... I think you're suggesting that, using multiple additional instances, several threads will be running, each with their own effective connection pool limit... right? If so, that's certainly food for thought - something I hadn't considered for what I see (from standard flat file processing on the mainframe) as serial processing... |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Sep 07, 2012 5:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's never a good idea to build a single threaded process, unless you are forced to.
If you are already processing records individually using the FileInput node to split records, then I suspect that adding additional instances will make whatever problem is occurring worse ,rather than better.
I'm not convinced that the open files really refers to connections on the web service call. I would expect that you would only ever have one - but presumably you've actually taken steps to verify that?
It's definitely worth looking at the connections from the web service side of things, though, if you haven't already. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
