| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Large messages & speed |
« View previous topic :: View next topic » |
| Author |
Message
|
| shogan2003 |
 Posted: Thu May 20, 2004 1:29 am Post subject: Large messages & speed |
 |
|
Centurion
Joined: 03 Jul 2003
Posts: 133
Location: London
|
Hi
I am running a Java program sitting on the server running in binding mode which simply puts to a queue. The put rate was laughable; 9 mins for a 2Mb file and 25 mins for a 4Mb file.
The sys admin staff will be looking at the Solaris kernel settings.
What sort of message rates could we expect for messages in the 1Mb - 10Mb range once the settings are optimised ? What should those settings be ?
_________________
MQSI2 certified specialist
MQSeries certified specialist |
|
| Back to top |
|
 |
| shogan2003 |
| Posted: Thu May 20, 2004 4:26 am Post subject: |
|
|
Centurion
Joined: 03 Jul 2003
Posts: 133
Location: London
|
The question is actually : What's the fastest way of reading a file in Java so as to pas on the text contents to the put call ?
There is no text manipulation so I am hoping there is a Java object which can read the file in one go (rather than a line at a time) and that object has a mthod to pass the contents as one (potentially enormous) string to the put call.
_________________
MQSI2 certified specialist
MQSeries certified specialist |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Thu May 20, 2004 8:10 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Don't use String but use StringBuffer and allocate the StringBuffer ahead of time.
| Code: |
RandomAccessFile in = new RandomAccessFile(fileName, "r");
StringBuffer sb = new StringBuffer(in.length());
|
Or read the file as bytes.
| Code: |
RandomAccessFile in = new RandomAccessFile(fileName, "r");
byte[] byteData = new byte[in.length()]; |
Or you can read it in chucks: i.e. 1,000,000 bytes at a time.
Regards,
Roger Lacroix
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| shogan2003 |
| Posted: Fri May 21, 2004 5:45 am Post subject: |
|
|
Centurion
Joined: 03 Jul 2003
Posts: 133
Location: London
|
Mr. Lacroix, thanks/merci. I had some local help who suggested BufferedReader. I could then use the object with the writeTo method.
Now MQSeries is now whizz-bang fast for put...must now do the same for Get
===============================================
msg_txt = "";
File f1;
// Dummy initialise
f1 = new File("x");
f1 = new File(full_file_name);
System.out.println("File : " + f1.getName());
System.out.println("File length = " + f1.length());
//************************************************
// Build up stream from input file
//************************************************
// Calculate number of buffer blocks
long remainder = f1.length() % buffLen;
long numBuffBlocks = f1.length() / buffLen;
if (remainder > 0 ) {
numBuffBlocks++;
}
System.out.println("Number of buffer blocks = " + numBuffBlocks);
if ((numBuffBlocks == 1) && (remainder > 0 )) {
System.out.println("Size of first block is " + remainder);
}
if (remainder == 0 ) {
System.out.println("Size of first block is " + buffLen);
}
if ((numBuffBlocks > 1) && (remainder == 0 )) {
System.out.println("Size of last block is " + buffLen);
}
if ((numBuffBlocks > 1) && (remainder > 0 )) {
System.out.println("Size of last block is " + remainder);
}
int lastBuffLen;
if (remainder > 0) {
lastBuffLen = (int)remainder;
}
else {
lastBuffLen = buffLen;
}
char[] inBuffer = new char[buffLen];
char[] inlastBuffer = new char[lastBuffLen];
BufferedReader buff = new BufferedReader(new FileReader(f1));
System.out.print("READING BLOCKS ");
try {
//while (buff.read(inBuffer, 0, buffLen) > 0) {
for (int i=0; i < numBuffBlocks - 1; i++) {
buff.read(inBuffer, 0, buffLen);
if (debug_mode.equals("DEBUG=YES")) {
System.out.print(".");
}
msg_txt += String.valueOf(inBuffer);
}
// Populate last block
buff.read(inlastBuffer, 0, lastBuffLen);
// Populate last block
buff.read(inlastBuffer, 0, lastBuffLen);
System.out.println("READ LAST BLOCK");
msg_txt += String.valueOf(inlastBuffer);
} catch (Exception e) {
System.out.println("Buffer error : " + e);
}
// write msg to queue
//************************************************
//Populate some fields in MQseries message header
pmo.options = MQC.MQPMO_SET_ALL_CONTEXT ;
msg_put.userId = user_id;
//msg_put.userId = MQEnvironment.userID;
msg_put.writeString(msg_txt);
local_queue.put(msg_put,pmo);
System.out.println("Message put on queue");
_________________
MQSI2 certified specialist
MQSeries certified specialist |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Fri May 21, 2004 9:23 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
You don't explicitly state it but it appears that msg_txt is a String. Therefore, all of these lines are killing you:
The JVM has to completely re-allocate / re-build the string object for msg_txt each time these lines are executed. Very, very, very, very slow!!!!
Use StringBuffer object for msg_txt and then use the append() method. Your code will be much faster.
You might want to search the Sun Java forums or get a hints and tricks Java book for more Java coding tips.
Regards,
Roger Lacroix
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| shogan2003 |
| Posted: Fri May 21, 2004 9:59 am Post subject: |
|
|
Centurion
Joined: 03 Jul 2003
Posts: 133
Location: London
|
Well spotted !
Funnily enough, the msg_txt+= .. statement is not slow if BufferedReader is used, but we are consuming LARGE AMOUNTS OF MEMORY !
Roughly 12 times as muuch memory as the message size !
Think what that means for a 100Mb message!
_________________
MQSI2 certified specialist
MQSeries certified specialist |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
