| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Execution Group Memory Usage Problem |
« View previous topic :: View next topic » |
| Author |
Message
|
| kayhansefat |
 Posted: Tue Sep 11, 2007 6:36 am Post subject: Execution Group Memory Usage Problem |
 |
|
Acolyte
Joined: 18 Oct 2006
Posts: 65
|
I have a large file containing approx 480,000 records from which I need to create a message for each. I therefore have a JCN which reads the file and for each line, creates a new message and propagates it to the output terminal, which then links to an MQ Output Node.
My Java code:
| Code: |
File sourceDirectory = new File("/home/wbiadmin/LOAD");
File[] sourceDirectory = sourceDirectory.listFiles();
BufferedReader bf = null;
while (getNextFileIndex(sourceDirObject))
{
String patientDetails = "";
try
{
// Create the Buffered Reader with the correct Character Set.
bf = new BufferedReader(new InputStreamReader(new FileInputStream(sourceDirObject[fileIndex]),"ISO8859_1"));
while ((patientDetails = bf.readLine()) != null)
{
//Do some translation here with the patientDetails variable
try
{
copyMessageHeaders(inMessage, outMessage);
addBody(outMessage, bodyData);
out.propagate(outAssembly);
}
finally
{
// clear the outMessage even if there's an exception
outMessage.clearMessage();
}
}
catch (FileNotFoundException e)
{
addToLog(sourceDirObject[fileIndex].getName(),"File not found");
}
catch (IOException e)
{
addToLog(sourceDirObject[fileIndex].getName(),"File not readable: " + e.getMessage() + " " + e.toString());
}
finally
{
if (bf != null)
{
try
{
bf.close();
}
catch (Exception e){}
}
}
}
//I then move the file to another directory
|
The getNextFileIndex() method just moves the pointer onto the next file in the directory to load all records in each of the files (if there is more than one file).
The code seems to work fine and propagates the messages onto the MQ Queue, however the memory usage continually rises all the way up to 97%, and the mem usage for the execution group that the flow is in is very high (67.8%):
| Code: |
| wbiadmin 18923 53.9 67.8 3085472 1394332 ? S 14:12 27:05 DataFlowEngine WBRK6_DEFAULT_BROKER 5f8fb2f4-1401-0000-0080-af4eb6a690ef InitialLoad 0 |
Now the messages are continually populated UNTIL approx 51,000. Then I get the following error (see below) and the EG restarts, restoring the memory usage to what it should do.
| Code: |
Sep 11 10:02:48 localhost out of memory [19063]
Sep 11 10:02:50 localhost WebSphere Broker v6004[27474]: (WBRK6_DEFAULT_BROKER)[231025584]BIP2060W: The broker has detected that the Execution Group InitialLoad, process ID 19063, has shutdown. : WBRK6_DEFAULT_BROKER.agent: /build/S600_P/src/AdminAgent/ImbAdminAgent.cpp: 4215: ImbAdminAgent::startAndMonitorADataFlowEngine: :
Sep 11 10:02:51 localhost WebSphere Broker v6004[4133]: (WBRK6_DEFAULT_BROKER.InitialLoad)[-1218584448]BIP2201I: Execution Group started: process '4133'; thread '-1218584448'; additional information: brokerName 'WBRK6_DEFAULT_BROKER'; executionGroupUUID 'b87a83da-1401-0000-0080-dd6718a729d1'; executionGroupLabel 'InitialLoad'; defaultExecutionGroup 'false'; queueManagerName 'WBRK6_DEFAULT_QUEUE_MANAGER'; trusted 'false'; dataSourceName 'DEFBKDB6'; userId 'wbiadmin'; migrationNeeded 'false'; brokerUUID '33e06a12-1101-0000-0080-8bf3e44c5c5c'; filePath '/opt/ibm/mqsi/6.0'; workPath '/var/mqsi'; ICU Converter Path ''. : WBRK6_DEFAULT_BROKER.b87a83da-1401-0000-0080-dd6718a729d1: /build/S600_P/src/DataFlowEngine/ImbMain.cpp: 351: main: :
Sep 11 10:02:57 localhost WebSphere Broker v6004[4133]: (WBRK6_DEFAULT_BROKER.InitialLoad)[-1218584448]BIP7407I: Publish/Subscribe subscriptions recovery started. Found '53' subscriptions. : WBRK6_DEFAULT_BROKER.b87a83da-1401-0000-0080-dd6718a729d1: /build/S600_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 792: SubscriptionService.readSubscriptions: DynamicSubscriptionEngine: DynamicSubscriptionEngine
Sep 11 10:02:57 localhost WebSphere Broker v6004[4133]: (WBRK6_DEFAULT_BROKER.InitialLoad)[-1218584448]BIP7410I: Publish/Subscribe subscriptions recovery. Recovered '2500' subscriptions. : WBRK6_DEFAULT_BROKER.b87a83da-1401-0000-0080-dd6718a729d1: /build/S600_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 792: SubscriptionService.readSubscriptions: DynamicSubscriptionEngine: DynamicSubscriptionEngine
Sep 11 10:02:57 localhost WebSphere Broker v6004[4133]: (WBRK6_DEFAULT_BROKER.InitialLoad)[-1218584448]BIP7033I: Publish/Subscribe state recovery completed. : WBRK6_DEFAULT_BROKER.b87a83da-1401-0000-0080-dd6718a729d1: /build/S600_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 667: SubscriptionService.readSubscriptions: DynamicSubscriptionEngine: DynamicSubscriptionEngine |
I have tried to get around this by splitting the file into chunks, each containing 50,000 lines. The same memory problem applies and even after the first file is completed, the memory usage still remains high. And if there is only one 50,000 in the directory, after it is finished the memory is not freed.
I am not sure if I should be freeing certain memory after each file, or if I am using the BufferedReader correctly. But surely when I close the bf memory should be freed?
Any help will be appreciated, thanks in advance.
Oh, Broker v6.0.0.4, MQ v6.0.2.1 |
|
| Back to top |
|
 |
| PieterV |
| Posted: Wed Sep 12, 2007 5:23 am Post subject: |
|
|
Disciple
Joined: 04 Jan 2006
Posts: 164
Location: Belgium
|
you can increase the heap size for that specific EG.
See infocenter. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
