| Author |
Message
|
| kiran_mvr |
 Posted: Fri Jan 13, 2006 12:09 pm Post subject: Handling Large files using Custom Node |
 |
|
Apprentice
Joined: 12 Dec 2004
Posts: 35
|
hi,
We are trying to handle large files(>50MB) from file system. For this we have designed a Java custom node. we increased the Heap size of a particular execution group using mqsichangeproperties command, to eleminate "Out of Memory errors".
My questions are
1. when a flow starts processing files, It uses all the memory assigned to its execution group. Does it releases the memory after processing a file or holds?
2. If the memory is not released, what should be done to release the memory?
3. Are there any considerations while handling large files using Custom Nodes?
Thanks,
Kiran |
|
| Back to top |
|
 |
| Tibor |
| Posted: Sat Jan 14, 2006 8:12 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Kiran,
This "out-of-memory" may be a platform-dependent behavior. Please send some information about os and broker versions.
Other question: did you release all the allocated area after the propagating? (e.g. clearMessage() for messages)
Tibor |
|
| Back to top |
|
|
| kiran_mvr |
| Posted: Tue Jan 17, 2006 7:02 am Post subject: Handling large files using Java custom Node |
|
|
Apprentice
Joined: 12 Dec 2004
Posts: 35
|
hi,
thanks for your replay, I am implementing it as you said.
I have one more question.
1. If we are using same Plug-In node for many flows how to implement synchronisation between the flows?
2. Is it automatically done or do we need to implement it explicitly in the code?
3. If many flows are trying to access the same Plug-In at a time how does the Plug-In node acts?
_________________
Kiran_manny |
|
| Back to top |
|
|
| Tibor |
| Posted: Tue Jan 17, 2006 7:50 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
As far as I know, the flows are running in separated threads and memory areas. For shared data access, I recommend you the IA91 support pack.
Tibor |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jan 17, 2006 8:50 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
|
| Back to top |
|
|
| kiran_mvr |
| Posted: Fri Jan 20, 2006 8:18 am Post subject: |
|
|
Apprentice
Joined: 12 Dec 2004
Posts: 35
|
hi,
I impemented the Java plugin as I have to handle a large EDI-X12 message of size >50MB. I am using Bufferedreader to handle the entire file. I am clearing the Bufferedreader after processing the message.
I am clearing the mesage ... mbmessage.clearMessage() after passing the message to the 'out' teriminal. I am also calling the method onDelete() and closing the database statements and connections. I hope I was following the basic convenctions in implementing a Java Plug-In.
While testing the Plug-In, I found it was using large amount of Memory and CPU resources. How can I overcome this?
Need your advice.
_________________
Kiran_manny |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jan 20, 2006 3:09 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| kiran_mvr wrote: |
hi,
I impemented the Java plugin as I have to handle a large EDI-X12 message of size >50MB. I am using Bufferedreader to handle the entire file. I am clearing the Bufferedreader after processing the message.
I am clearing the mesage ... mbmessage.clearMessage() after passing the message to the 'out' teriminal. I am also calling the method onDelete() and closing the database statements and connections. I hope I was following the basic convenctions in implementing a Java Plug-In.
While testing the Plug-In, I found it was using large amount of Memory and CPU resources. How can I overcome this?
Need your advice. |
I would be very surprised if what that file contained would be a single EDI message. My question/challenge to you would be to create a reader that would recognize the start / end of each individual message on the file and put each individual message on the queue. Of course you would need to keep a count somewhere so as to be able to restart in case of loss of power etc... + a bad message queue to avoid poison messages....
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
