| Author |
Message
|
| zombie99 |
 Posted: Wed Oct 31, 2007 2:04 am Post subject: Java plug-in trace node hangs execution group |
 |
|
Newbie
Joined: 31 Oct 2007
Posts: 5
|
Hi all,
I'm new to this list, but I think it looks like a nice place where I can get some help.
We are currently using a custom java plug-in for the MQ message broker. The plugin is a Jar-file which is installed under the directory IBM\WebSphere MQ Integrator 2.1\jplugin . We use the plug-in to trace to log files what is happening in the message flows. We don't use the built-in trace node because it doesn't rotate the log files and it locks them up so we cannot delete them when they have grown to large.
The broker we are using is v5.3 and it runs Java 1.3.1 for the java plug-ins. The problem we are experiencing is that when there is a heavy throughput of messages in a message flow that splits (with a compute node) one message into (at most) 6 messages, which are then propagated through the custom trace node it sometimes causes the entire execution group to hang and consume 100% cpu. Sometimes it doesn't happen for a couple of thousand messages and sometimes it happens only after a couple of hundred messages, so the problem is not depending on a certain kind of message or something like that.
We don't know why it behaves like this, but it seems that when we are writing to the log file (using a PrintWriter and a FileOutputStream in a try-catch-finally block where finally closes the stream) very frequently it causes the execution group to hang. Hope someone find this interesting and can help me figure this out.
Regards,
Mattew |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Oct 31, 2007 3:50 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
2.1?
2.1?
You have significantly bigger problems than an occasionally hanging custom plugin.
Start planning a migration to v6.0. It's the last version you'll have a supported migration path to. And you're so far out of support, it's ridiculous. As well, 2.1 is painful to use.
Your custom plugin problem is almost certainly an issue with the Java code. Likely something is causing it to get stuck in a loop.
Btw - "v5.3" is the version of MQ you are using, not the version of Broker.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| zombie99 |
| Posted: Wed Oct 31, 2007 4:06 am Post subject: |
|
|
Newbie
Joined: 31 Oct 2007
Posts: 5
|
| jefflowrey wrote: |
2.1?
2.1?
You have significantly bigger problems than an occasionally hanging custom plugin.
Start planning a migration to v6.0. It's the last version you'll have a supported migration path to. And you're so far out of support, it's ridiculous. As well, 2.1 is painful to use.
Your custom plugin problem is almost certainly an issue with the Java code. Likely something is causing it to get stuck in a loop.
Btw - "v5.3" is the version of MQ you are using, not the version of Broker. |
Yes, I know the broker is old and we have been planning to migrate to v6.0 for some time. Hopefully it will happen soon...
Regarding the Java code, it is currently only about 20 lines and there is nothing going on in that code which makes it stick in a loop. I can prove it by writing a Java 6.0 application that does the same thing. If it has something to do with Java, it must either be that Java 1.3.1 has some bug or a bug in the Java library which is provided by IBM, that is jplugin.jar which contains classes like com.ibm.broker.plugin.MbNode.
/Mattew |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 31, 2007 4:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zombie99 wrote: |
a bug in the Java library which is provided by IBM, that is jplugin.jar which contains classes like com.ibm.broker.plugin.MbNode.
|
Given you're using v2.1 then that has to be a possible scenario. Given also I know nothing about Java and I'm going purely on general v2.1 principles (i.e. it's painful, bug ridden and just contact admin).
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 31, 2007 4:24 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You haven't really proven anything by writing a Java 6 program.
You can try migrating the file manipulation code into a standalone java program, and running that against the same jvm. So a JDK 1.3 java application, not a JDK1.6 application.
Does your Java code do any looping at all? Or does it only write out a fixed set of fields from the Input message?
Is your node in a flow that is using multiple instances? Are you sure your code is written to handle that? Is your node used by more than one flow in the same EG?
If there is actually a bug either in the JVM or in the broker plugin infrastructure - then there's absolutely nothing you can do about it. And likely, the only people who can tell you for sure that this is so are IBM support - and you can't talk to them because you're out of support.
You can spend some time searching through IBM's website for APARs and bug fixes on Java and JVMs in Broker 2.1. And you can make sure you're at the last released fixpack of 2.1 runtime. If you've done both of these, and you firmly believe that there's nothing wrong with your code... then you have to limp along with this problem until you migrate to v6.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| zombie99 |
| Posted: Wed Oct 31, 2007 5:33 am Post subject: |
|
|
Newbie
Joined: 31 Oct 2007
Posts: 5
|
| jefflowrey wrote: |
You haven't really proven anything by writing a Java 6 program.
You can try migrating the file manipulation code into a standalone java program, and running that against the same jvm. So a JDK 1.3 java application, not a JDK1.6 application. |
Well, I have proven that my code isn't causing loops in Java 6. If I compile a JDK 1.3 application and that fails, I will prove that Sun has a bug.
| jefflowrey wrote: |
Does your Java code do any looping at all? Or does it only write out a fixed set of fields from the Input message? |
No loops at all. I only print out about 300 lines of text every time. Okay, that is done in a loop, but it will always terminate after 30 loops.
| jefflowrey wrote: |
Is your node in a flow that is using multiple instances? Are you sure your code is written to handle that? Is your node used by more than one flow in the same EG?
|
The node is used in more flows, but in the tests I've run, the only exectuting flow has been the one that has crashed. I have also tried with various locking-techniques but they have all caused the same problem. The EG hangs both if I have locking and if I don't. In the test cases where I know only one flow is writing to one unique log file at a time I have not used locking, but it hangs anyway.
| jefflowrey wrote: |
If there is actually a bug either in the JVM or in the broker plugin infrastructure - then there's absolutely nothing you can do about it. And likely, the only people who can tell you for sure that this is so are IBM support - and you can't talk to them because you're out of support. |
Too bad - I guess we're in desperate need of an upgrade 
| jefflowrey wrote: |
You can spend some time searching through IBM's website for APARs and bug fixes on Java and JVMs in Broker 2.1. And you can make sure you're at the last released fixpack of 2.1 runtime. If you've done both of these, and you firmly believe that there's nothing wrong with your code... then you have to limp along with this problem until you migrate to v6. |
Okay. Thanks for your input. Much appreciated.
/Mattew |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 31, 2007 5:37 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
So you have the same code that works fine in many flows?
In one flow, with one particular kind of message (although not necessarily any particular message itself), then the flow will lock up under heavy volume and tank the EG...?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 31, 2007 5:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
If you're still using v2.1 your need for an upgrade is now past desperate.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zombie99 |
| Posted: Wed Oct 31, 2007 6:55 am Post subject: |
|
|
Newbie
Joined: 31 Oct 2007
Posts: 5
|
| jefflowrey wrote: |
So you have the same code that works fine in many flows?
In one flow, with one particular kind of message (although not necessarily any particular message itself), then the flow will lock up under heavy volume and tank the EG...? |
Yes, it works fine in all other flows. Many flows uses the custom trace node, but in this particular case, where one message is split into at most 6 different messages, the messages seem to arrive exactly at the same time to the node.
However, even if I do synchronization (tried many techniques) things lock up. And synchronization issues often don't cause processes to use 100% cpu. Also, the particular flow that is hanging always receives hundreds of messages at a time.
/Mattew |
|
| Back to top |
|
|
| zombie99 |
| Posted: Thu Nov 01, 2007 6:16 am Post subject: Broker version |
|
|
Newbie
Joined: 31 Oct 2007
Posts: 5
|
By the way, we are using v5.0 of the broker. The 2.1 in the installation directory is not the version of the broker.
BIPv500 en US
ucnv Console CCSID 1208 dft ucnv CCSID 1208
OS ISO8859-1 ICUW ISO-8859-1 ICUA ISO-8859-1
BIP8996I: Version: 500
BIP8997I: Product: WebSphere MQ Integrator
BIP8998I: CMVC Level: S500-L40713
BIP8999I: Build Type: Production
/Mattew |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Nov 01, 2007 6:34 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Aha. Well try upgrading it to the latest FP of v5, then.
And you're still in support on v5 - for a bit yet - and can open a PMR.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
