| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Flow performs slow - is there a better way to handle this? |
« View previous topic :: View next topic » |
| Author |
Message
|
| fjb_saper |
 Posted: Thu Nov 15, 2007 8:54 am Post subject: |
 |
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
as well check out Integer.decode("0xFF"); 
_________________
MQ & Broker admin |
|
| Back to top |
|
 |
| murdeep |
| Posted: Tue Nov 27, 2007 9:27 am Post subject: |
|
|
Master
Joined: 03 Nov 2004
Posts: 211
|
Ok, I built and tested a JCN to do this.
Here's the code:
| Code: |
import java.util.ListResourceBundle;
import com.ibm.broker.javacompute.MbJavaComputeNode;
import com.ibm.broker.plugin.*;
public class AddDelimiter extends MbJavaComputeNode {
private final static String CRLF_CHARACTERS = "crlfCharacters";
private final static String RECORD_LENGTH = "recordLength";
public void evaluate(MbMessageAssembly contact admin) throws MbException {
MbElement msgBody;
byte[] msgByteIn = null;
//get UDP's required to process message
String crlfCharacters = (String)getUserDefinedAttribute(CRLF_CHARACTERS);
if (crlfCharacters == null) {
MbService.logWarning(this,
"UDP_Undefined",
ErrorMessages.MESSAGE_SOURCE,
ErrorMessages.NO_CRLF_UDP,
"",
null);
}
Integer recordLength = (Integer)getUserDefinedAttribute(RECORD_LENGTH);
if (recordLength == null) {
MbService.logWarning(this,
"UDP_Undefined",
ErrorMessages.MESSAGE_SOURCE,
ErrorMessages.NO_RECORD_LENGTH_UDP,
"",
null);
}
// convert delimiter characters to byte[]
byte[] crlfBytes = new byte[crlfCharacters.length() / 2];
for (int i = 0; i < crlfBytes.length; i++) {
crlfBytes[i] = (byte) Integer.parseInt(crlfCharacters.substring(2*i, 2*i+2), 16);
}
//get node terminals
MbOutputTerminal out = getOutputTerminal("out");
MbOutputTerminal alt = getOutputTerminal("alternate");
//get input message assembly
MbMessage inMessage = contact admin.getMessage();
// create new message
MbMessage outMessage = new MbMessage();
MbMessageAssembly outAssembly = new MbMessageAssembly(contact admin,
outMessage);
// copy headers from the input message
copyMessageHeaders(inMessage, outMessage);
//get message body from input
msgBody = inMessage.getRootElement().getLastChild();
msgByteIn = msgBody.toBitstream("","","",0,0,0);
//calculate total records in message (note flow must validate message before calling this JCN
//flow must ensure that message size is even multiple of record length
int msgLength = msgByteIn.length;
int totalRecords = msgLength/recordLength.intValue();
//allocate the new message buffer
byte[] msgByteOut = new byte[msgLength + (totalRecords*crlfBytes.length)];
//populate the new message buffer copying each record from input message and appending record delimiter
int currentRecord = 0;
while (currentRecord < totalRecords) {
System.arraycopy(msgByteIn,currentRecord*recordLength.intValue(),msgByteOut,(currentRecord*(recordLength.intValue()+crlfBytes.length)),recordLength.intValue());
System.arraycopy(crlfBytes,0,msgByteOut,(currentRecord*(recordLength.intValue()+crlfBytes.length))+recordLength.intValue(),crlfBytes.length);
currentRecord = currentRecord + 1;
}
//get output message root element
MbElement outRoot = outMessage.getRootElement();
//Create the output Blob Parser element
MbElement outParser = outRoot.createElementAsLastChild(MbBLOB.PARSER_NAME);
MbElement outBodyBLOB = outParser.createElementAsLastChild(MbElement.TYPE_NAME_VALUE, "BLOB", msgByteOut);
try {
// The following should only be changed
// if not propagating message to the 'out' terminal
out.propagate(outAssembly);
} finally {
// clear the outMessage
outMessage.clearMessage();
}
}
public void copyMessageHeaders(MbMessage inMessage, MbMessage outMessage) throws MbException
{
MbElement outRoot = outMessage.getRootElement();
MbElement header = inMessage.getRootElement().getFirstChild();
while(header != null && header.getNextSibling() != null)
{
outRoot.addAsLastChild(header.copy());
header = header.getNextSibling();
}
}
public static class ErrorMessages extends ListResourceBundle
{
public static final String MESSAGE_SOURCE = ErrorMessages.class.getName();
public final static String NO_CRLF_UDP = "No crlfCharacters UDP";
public final static String NO_RECORD_LENGTH_UDP = "No recordLength UDP";
private Object[][] messages =
{
{NO_CRLF_UDP,
"No UDP for: " + CRLF_CHARACTERS
},
{NO_RECORD_LENGTH_UDP,
"No UDP for: " + RECORD_LENGTH
}
};
public Object[][] getContents()
{
return messages;
}
}
}
|
Don't know if this is efficient Java but it sure beats the snot out of ESQL. This implementation can dequeue 1.4 4M msgs/second. So the entire transfer detailed above takes about 210 seconds versus ~22000 when using ESQL.
Comments welcome. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Nov 27, 2007 3:23 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Clarification: murdeep was encountering a performance defect in ESQL. Not very surprising that Java beats ESQL for his defect scenario. Normally, ESQL and Java have very similar performance, and it is a matter of preference which one you choose. |
|
| Back to top |
|
|
| Ian |
| Posted: Wed Nov 28, 2007 9:49 am Post subject: |
|
|
Disciple
Joined: 22 Nov 2002
Posts: 152
Location: London, UK
|
I have not had a chance to take this ESQL and try it out for myself but I would be interested to know the amount of memory being used by the DataFlowEngine process whilst using the ESQL method ?
I suspect that the memory may be increasing and from a quick look at the ESQL BLOB processing this could be because the heap is being fragmented and it is the cost of the increasing number of new memory allocations that are taking up the processing time.
As my colleague has said, the cost of ESQL and JCN processing are generally comparable or within a percentage of each other. However, in this (specific) case the ESQL processing may be causing fragmentation of the heap and this is what is affecting the processing time.
In this case the solution may be the use of a JCN. If you are interested in understanding the reason why the ESQL method is causing a problem then let me know what the DFE memory usage start and end points are.
_________________
Regards, Ian |
|
| Back to top |
|
|
| murdeep |
| Posted: Wed Nov 28, 2007 10:40 am Post subject: |
|
|
Master
Joined: 03 Nov 2004
Posts: 211
|
| Using the ESQL version the DFE memory usage is 76876K at DFE start. When the flow processes a message it ends up at 90056K. As soon as I put another 4M message on the queue it drops to 87000K and then ramps up to ~90056K with peaks at around 92000K. This cycle repeats each time I drop a message on the flow input queue. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
