ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportMessage Model Integration Bus taking too much time

Post new topicReply to topic Goto page 1, 2  Next
Message Model Integration Bus taking too much time View previous topic :: View next topic
Author Message
mhasan23
PostPosted: Thu May 03, 2018 11:40 pm Post subject: Message Model Integration Bus taking too much time Reply with quote

Newbie

Joined: 12 Oct 2017
Posts: 9

Hello all,

I am facing issue in IBM Integration (Integration Service [web service]) bus during parsing a big size data string (delimited file) on FileRead node. I have set my Message Model on File Read node but when I provide small chunk or data like 2 - 3 Mb it works fine but if data is over 4 Mb it stuck and error occured of "Java Heap Space" on integration toolkit. I have tested the Message Model in Test Parse Model, it takes time too much and in the end toolkit gives "java.exception".

I have tried to get this from MQGet Node but still same issue there (both cases BLOB or if I use my Message model in Input Message Parsing option).

Moreover when I am testing this Integration Service getting error ;
"Error getting response; java.net.SocketException: Connection reset"

Kindly help me out.
Back to top
View user's profile Send private message
timber
PostPosted: Fri May 04, 2018 3:54 am Post subject: Reply with quote

Shaman

Joined: 25 Aug 2015
Posts: 773

It's not a good idea to parse megabytes of data in a single chunk. The FileInput node has facilities for processing a file one record at a time. I recommend that you investigate those options.
I do not recommend using MQGet unless you know exactly what you are doing and why. It is often used incorrectly by inexperienced developers. In any case, the parser code used by an MQGet node is identical to the code executed by any other node.
Back to top
View user's profile Send private message
mhasan23
PostPosted: Fri May 04, 2018 4:16 am Post subject: Reply with quote

Newbie

Joined: 12 Oct 2017
Posts: 9

timber,

We need to process large size file using integration service, I have tried both ways MQ node as well as File node.

My question is that why integration bus taking too much time for parsing file which is in size of megabytes not in gigabytes and what could I if it is not working, what will be the alternate solution?

Regards,
Mhasan
Back to top
View user's profile Send private message
timber
PostPosted: Fri May 04, 2018 5:54 am Post subject: Reply with quote

Shaman

Joined: 25 Aug 2015
Posts: 773

Quote:
We need to process large size file using integration service
Please go ahead. Nobody said that you cannot do that. The IIB Knowledge Center has whole sections dedicated to the processing of large files/messages.
Quote:
I have tried both ways MQ node as well as File node.
Please explain where the message flow is spending most of its time. If the parsing is taking most of the time then I don't see why you would bother to experiment with using MQ instead of a file.
Quote:
My question is that why integration bus taking too much time for parsing file which is in size of megabytes not in gigabytes
Yes. Good question. IIB is usually faster than alternative technologies, so I don't blame you being surprised. If you explain your file format and your message model, and give us some details about the design of your message flow then we might be able to help you.
Back to top
View user's profile Send private message
mhasan23
PostPosted: Fri May 04, 2018 6:18 am Post subject: Reply with quote

Newbie

Joined: 12 Oct 2017
Posts: 9

Thank you for your reply, I would like to share information with you about my message flow and message model. As for your second question; I have created a message model to parse CSV file ('|' delimited). Now if I used this message model on File Read node it will take time as well as if I go with BLOB format on File Read node that would also took time (as I have already mentioned that file is greater than 3 Mb).

For MQ Node, I have just tried because if I find the solution with that I am ok but issue still same like in File Node that is why I mentioned both.

File format is simple CSV, format shared below;

Code:
COL1|COL2|COL3|COL4
ROW1a|ROW1b|ROW1c|ROW1d
ROW2a|ROW2b|ROW2c|ROW2d


Message flow design;

Code:
[SOAP Input] -> [Compute to set File Name] -> [File Read Node] -> [Mapping Node for SOAP Reply] -> [SOAP Output]


In above message flow [File Read Node] have the Message Model set in DFDL domain and at this point it took time to parse the file.

Currently I set the "Record Detection" property of [file Read Node] on "Whole File".

Regards,
Mhasan
Back to top
View user's profile Send private message
timber
PostPosted: Tue May 08, 2018 12:40 am Post subject: Reply with quote

Shaman

Joined: 25 Aug 2015
Posts: 773

Quote:
if I used this message model on File Read node it will take time as well
What do you define as 'take time'. Tell us how long it takes - we cannot see your screen.
Quote:
it will take time as well as if I go with BLOB format on File Read node that would also took time
Again, please provide actual numbers. It would also help if you tell us what the non-functional requirements (SLAs) for this message flow are.
Quote:
Currently I set the "Record Detection" property of [file Read Node] on "Whole File".
Why did you choose that setting? Do you need the entire file to be treated as a single atomic transaction?
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue May 08, 2018 6:10 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

mhasan23 wrote:
(as I have already mentioned that file is greater than 3 Mb).


Okay, that's not a "large" file IMHO. A large file is something when you need to check how much memory the JVM has (50Mb +). It may run your workstation out of memory but that means you have a tiny laptop that's probably below spec for IIB.

mhasan23 wrote:
Message flow design;

Code:
[SOAP Input] -> [Compute to set File Name] -> [File Read Node] -> [Mapping Node for SOAP Reply] -> [SOAP Output]


In above message flow [File Read Node] have the Message Model set in DFDL domain and at this point it took time to parse the file.

Currently I set the "Record Detection" property of [file Read Node] on "Whole File".


So stop me when I go wrong:

- The flow gets a SOAP request with the file name
- The flow reads this 3Mb file and reformats it to XML
- The flow sends the result to the original requester as a SOAP response
- The flow takes a long time

So where does it take the time? Are you sure it's in the parsing not elsewhere?

Why (as my associate says) are you reading as Whole File for a record based structure? Are you simply trying to get all the records in a single SOAP reply and if so, how did you pick this method?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mhasan23
PostPosted: Wed May 09, 2018 11:07 am Post subject: Reply with quote

Newbie

Joined: 12 Oct 2017
Posts: 9

Hello Vitor, thanks for replying,

First thing is that I am not using laptop or any small computer. I have windows server machine on which I have setup IBM Integration Bus version 10.0.0.4 with 16Gb of RAM.

Quote:
So stop me when I go wrong:

- The flow gets a SOAP request with the file name
- The flow reads this 3Mb file and reformats it to XML
- The flow sends the result to the original requester as a SOAP response
- The flow takes a long time


For above quote; I have a problem at second point which is [File Read Node], in this node I have used input message parsing where I select my Message Model. And this node is set to "Read Whole File" option.

If I would say about minimum number of records in my file; then I would say its around 100,000 Lines (delimited file).

What my target is to get the file name from SOAP Request, then read the specific file, parse it to XML format and reply it to client.

While I run this in debugging, IBM Integration Toolkit throw an exception of java; I have attached the link below for error screen

Link:
https://drive.google.com/file/d/0B_H1J15MTuLGVlBUOVBHeEZsVkFVTlMwSnA0VjRsMFJVdzc0/view
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed May 09, 2018 11:44 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

mhasan23 wrote:
in this node I have used input message parsing where I select my Message Model. And this node is set to "Read Whole File" option.


Two of us have now asked you why you're using this option to read a delimited file. We're less inclined to answer your questions when you don't answer ours.

mhasan23 wrote:
If I would say about minimum number of records in my file; then I would say its around 100,000 Lines (delimited file).


That's not "large" by any reasonable measure.

mhasan23 wrote:
What my target is to get the file name from SOAP Request, then read the specific file, parse it to XML format and reply it to client.


Thank you for confirming my understanding.

mhasan23 wrote:
While I run this in debugging, IBM Integration Toolkit throw an exception of java; I have attached the link below for error screen

Link:
https://drive.google.com/file/d/0B_H1J15MTuLGVlBUOVBHeEZsVkFVTlMwSnA0VjRsMFJVdzc0/view


Not only is that site inaccessible to me (the financial group I work for doesn't like the use of personable cloud storage from the office - something about uploading data from the internal systems) but the debugger is the wrong tool to use to solve a parsing problem, as you've discovered.

Take a user trace, see where the time is going and explain what part of the requirements drive you to use whole file on a delimited record structure.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed May 09, 2018 11:48 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

It would also help if you gave some indication of the logic you're using (having parsed the file) to convert it into XML.

If it's anything like:

Code:
 SET OutputRoot.XMLNSC.OutputMessage.Segment[x] = InputRoot.DFDL.InputMessage.Record[x];
SET x = x + 1;


then I've found your problem.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mhasan23
PostPosted: Wed May 09, 2018 10:55 pm Post subject: Reply with quote

Newbie

Joined: 12 Oct 2017
Posts: 9

Quote:
Two of us have now asked you why you're using this option to read a delimited file. We're less inclined to answer your questions when you don't answer ours.


If I know why I am getting problem with these options then I don't post question like this here. I have tried with small number of records file while developing message flow and it works with these options.

Quote:
That's not "large" by any reasonable measure.


I am not saying that; it is large or small, I am just telling the minimum number of records.

Quote:
Take a user trace, see where the time is going and explain what part of the requirements drive you to use whole file on a delimited record structure.


My answer is same as above; For User Trace I have set the Trace node for that noticed that dump error file write on location ;

"C:\ProgramData\IBM\MQSI\common\errors"

I have seen Windows Event Logs found that;

Code:
The description for Event ID 0 from source IBM Java cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

JVMDUMP032I JVM requested Java dump using 'C:\ProgramData\IBM\MQSI\common\errors\javacore.20180510.114754.13804.0009.txt' in response to an event



Code:
( MyBorkerNode.TestNode ) Java exception: ''java.lang.OutOfMemoryError''; thrown from class name: ''org.eclipse.paho.client.mqttv3.MqttAsyncClient'', method name: ''disconnect'', file: ''MqttAsyncClient.java'', line: '529'   

The message contains that data associated with a Java exception.   

No user action required. 
Back to top
View user's profile Send private message
timber
PostPosted: Thu May 10, 2018 12:24 am Post subject: Reply with quote

Shaman

Joined: 25 Aug 2015
Posts: 773

I suggest that you switch to using 'Delimited' instead of 'Whole file' (or else explain why you cannot)
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu May 10, 2018 5:39 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

mhasan23 wrote:
Quote:
Two of us have now asked you why you're using this option to read a delimited file. We're less inclined to answer your questions when you don't answer ours.


If I know why I am getting problem with these options then I don't post question like this here. I have tried with small number of records file while developing message flow and it works with these options.


It may "work" but that's not any reasonable design decision. How did you decide to do this, did you simply leave the node at the default and why (given that you're now having problems) are you sticking with it?

That's why we're asking the question. It's not the obvious choice for a delimited file and often causes problems like the ones you're seeing.

But if you're happy with the principle that anything which works on test data during development must be suitable for prod then have at it and you don't need any more advice from us.

mhasan23 wrote:
[
Quote:
That's not "large" by any reasonable measure.


I am not saying that; it is large or small, I am just telling the minimum number of records.


And that's my point - you're not attempting anything that should be difficult.

mhasan23 wrote:
[
Quote:
Take a user trace, see where the time is going and explain what part of the requirements drive you to use whole file on a delimited record structure.


My answer is same as above; For User Trace I have set the Trace node for that noticed that dump error file write on location


I didn't say use a Trace node - I said take a user trace using the broker's inbuilt tracing facility as documented.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mhasan23
PostPosted: Fri May 11, 2018 12:18 am Post subject: Reply with quote

Newbie

Joined: 12 Oct 2017
Posts: 9

Thanks all for suggestions and guidelines. I have resolve this issue, as thanks @vitor for UserTrace point. I have found the JAVA Heap Memory exception in that. As I increase the Heap Memory it is working now.

Thanks all again.
Back to top
View user's profile Send private message
timber
PostPosted: Fri May 11, 2018 2:46 am Post subject: Reply with quote

Shaman

Joined: 25 Aug 2015
Posts: 773

So the only problem was in the IIB Toolkit? No problems with performance of the deployed message flow?
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexWebSphere Message Broker SupportMessage Model Integration Bus taking too much time
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.