| Author |
Message
|
| hikari_sukunami |
 Posted: Tue Oct 29, 2013 1:32 am Post subject: Broker did not provide a response within the specified time |
 |
|
Newbie
Joined: 26 Sep 2013
Posts: 6
|
Hi,
I'm doing a load testing of our applications using jmeter. With some due reasons that I don't know, every time I run 90+ transactions but in one thread only, it produces this message in the view results tree of jmeter: Timeout. Broker did not provide a response within the specified time interval. The message reached the main flow, but timed out during processing.
This is the log:
EXCEPTION LIST
===========
( ['MQROOT' : 0x2aaad468b7d0]
(0x01000000:Name):RecoverableException = (
(0x03000000:NameValue):File = '/build/slot1/S800_P/src/DataFlowEngine/ImbDataFlowNode.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 1154 (INTEGER)
(0x03000000:NameValue):Function = 'ImbDataFlowNode::createExceptionList' (CHARACTER)
(0x03000000:NameValue):Type = 'ComIbmJavaComputeNode' (CHARACTER)
(0x03000000:NameValue):Name = 'DynamicOrchestrationControllerMain#FCMComposite_1_15' (CHARACTER)
(0x03000000:NameValue):Label = 'DynamicOrchestrationControllerMain.DOC' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPmsgs' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 2230 (INTEGER)
(0x03000000:NameValue):Text = 'Node throwing exception' (CHARACTER)
(0x01000000:Name ):RecoverableException = (
(0x03000000:NameValue):File = '/build/slot1/S800_P/src/DataFlowEngine/PluginInterface/com_ibm_broker_plugin_CMbService.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 1871 (INTEGER)
(0x03000000:NameValue):Function = 'ImbJavaExceptionUtils::throwableToNativeException' (CHARACTER)
(0x03000000:NameValue):Type = '' (CHARACTER)
(0x03000000:NameValue):Name = '' (CHARACTER)
(0x03000000:NameValue):Label = '' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPmsgs' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 4367 (INTEGER)
(0x03000000:NameValue):Text = 'Unhandled exception in plugin method.' (CHARACTER)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'evaluate' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'DOC' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'java.lang.NullPointerException' (CHARACTER)
)
(0x01000000:Name ):RecoverableException = (
(0x03000000:NameValue):File = '/build/slot1/S800_P/src/DataFlowEngine/PluginInterface/com_ibm_broker_plugin_CMbService.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 1889 (INTEGER)
(0x03000000:NameValue):Function = 'ImbJavaExceptionUtils::throwableToNativeException' (CHARACTER)
(0x03000000:NameValue):Type = '' (CHARACTER)
(0x03000000:NameValue):Name = '' (CHARACTER)
(0x03000000:NameValue):Label = '' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPmsgs' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 4395 (INTEGER)
(0x03000000:NameValue):Text = 'Unhandled exception in plugin method' (CHARACTER)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'java.lang.NullPointerException' (CHARACTER)
I don't know why it has a null pointer exception in the 90th transaction and so on. But it is working fine before it reach the 90th transaction. I'm using jdbc and memcached in the apps.
Can someone please help me to know what is the cause or what is the workaround to this problem?
Thanks! |
|
| Back to top |
|
 |
| Esa |
| Posted: Tue Oct 29, 2013 2:14 am Post subject: Re: Broker did not provide a response within the specified t |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| hikari_sukunami wrote: |
I'm using jdbc and memcached in the apps.
|
That is where you will have to look. You need to add some exception handling in your JCN code to capture and throw the stack trace.
What is your reason for using memcached and not the built-in global cache? |
|
| Back to top |
|
|
| MikeOliverAZ |
| Posted: Wed Oct 30, 2013 10:06 pm Post subject: same exact thing 90 and null pointer |
|
|
Apprentice
Joined: 10 Jul 2013
Posts: 47
|
Ok we used log4jnode and set some DEBUG logging around where we figured it was.
| Code: |
2013-10-30 19:14:03,872 DEBUG [DOCUTILS] Memcached:Too many open files
2013-10-30 19:14:03,874 DEBUG [DOCUTILS] ruleDefinitions:null
|
On linux the memcached server runs out over the open files limit with the default of 1024 at around 90 +- 3 for us on load testing.
That number can be increased but you will run into it eventually.
I have read that the answer is to put the client in a singleton so the multiple threads use the same client and that is thread safe, but just now looking into that.
As for why use Memcached instead of the global cache...for us it was because we wanted to store the java objects and not database records.
With couchbase we get both a persistent cache and an active distributed cache with a memcached interface. So any changes to the objects gets distributed automatically where the global cache is not active to be kind. |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Oct 30, 2013 11:36 pm Post subject: Re: same exact thing 90 and null pointer |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| MikeOliverAZ wrote: |
As for why use Memcached instead of the global cache...for us it was because we wanted to store the java objects and not database records.
With couchbase we get both a persistent cache and an active distributed cache with a memcached interface. So any changes to the objects gets distributed automatically where the global cache is not active to be kind. |
Yes, Global Cache is quite limited, it only supports caching java primitive types and Strings, not objects (unless you serialize them). And you cannot update the values.
As you have found out the hard way, standard exception handling in a JCN strips off java stack trace.
It would be a good idea to make the JCN throw a new MBRecoverableException with a serialized stack trace
when it encounters an exception that is not an instance of MbBrokerException. For example. |
|
| Back to top |
|
|
| Simbu |
| Posted: Wed Oct 30, 2013 11:59 pm Post subject: Re: same exact thing 90 and null pointer |
|
|
Master
Joined: 17 Jun 2011
Posts: 289
Location: Tamil Nadu, India
|
| Esa wrote: |
Yes, Global Cache is quite limited, it only supports caching java primitive types and Strings, not objects (unless you serialize them). And you cannot update the values. |
| Quote: |
| IBM Integration Bus Version 9.0 introduces Java POJO support for the Global Cache, allowing message flow developers to store complex Java business objects within the cache with ease. |
|
|
| Back to top |
|
|
| Esa |
| Posted: Thu Oct 31, 2013 12:52 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Thank you for the information, Simbu.
| V8 InfoCenter wrote: |
| For keys and values, Java primitive types and strings are supported. |
| V9 InfoCenter wrote: |
| For keys and values, Java primitive types and strings are supported. Java objects are supported as values. |
|
|
| Back to top |
|
|
|
|
