| Author |
Message
|
| riteshprasad1 |
 Posted: Mon Jun 05, 2017 10:09 am Post subject: Multi-Threaqding |
 |
|
Apprentice
Joined: 31 Jul 2014
Posts: 25
|
Hi,
I am using the add additional instance feature of MQ Input node. I am running a test bed of 100 messages and expect 400 valid responses.
Using order by to anything other than default I am getting valid responses with unacceptable performance throughput.
When I use the order by feature to default I am getting invalid responses 20 threads running at run time with very fast performance time.
Can someone please suggest a solution to this problem? Any help will me much appreciated.
Thanks |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Jun 05, 2017 10:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Additional instances don't have a "order by" feature, so I assume you're referring to some other setting and/or code construct. Perhaps you'd care to comment on this assumption, or do you prefer us to just guess?
Based on that assumption, I infer that:
- you have some kind of affinity issue and/or your code is not thread safe
- you need around 20 threads to meet your SLA
But I've no clue what you do about it. Because I don't know what you're doing already.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| riteshprasad1 |
| Posted: Mon Jun 05, 2017 10:19 am Post subject: |
|
|
Apprentice
Joined: 31 Jul 2014
Posts: 25
|
| Order mode is another property of the MQ input node. Add additional instance is a separate feature. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jun 05, 2017 10:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So you send 100 messages and expect back 400.
Is that 4 per message sent?
And you need to get those 4 response messages "in order"?
You should probably consider using Aggregation and Resequence.
And then reconsider the flow design compared to the requirements.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 05, 2017 10:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| riteshprasad1 wrote: |
| Order mode is another property of the MQ input node. |
Thank you for dripping that tiny piece of information about your solution out, and not spoiling the fun of guessing about the rest of your processing. 
So if you try and retrieve the messages by user id, queue order or some field in the record, your processing works but runs slowly with a number of threads that you don't mention? But if you just let them come off the queue the flow runs up to 20 threads (which may or may not correlate to the number of additional instances you're set) and doesn't work.
Sounds like a message affinity problem connected to the reason you've not mentioned why 100 messages are supposed to give 400 responses.
Or something.
Drip some more information and see if that helps.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| riteshprasad1 |
| Posted: Mon Jun 05, 2017 10:35 am Post subject: |
|
|
Apprentice
Joined: 31 Jul 2014
Posts: 25
|
yes correctly put up thanks. I am using 20 instances in all cases.
Order by - default gives me the best throughput but invalid responses.
Order by - User ID or queue order or by user defined xpath order I get valid responses on 20 threads but very poor performance. |
|
| Back to top |
|
|
| riteshprasad1 |
| Posted: Mon Jun 05, 2017 10:41 am Post subject: |
|
|
Apprentice
Joined: 31 Jul 2014
Posts: 25
|
There are different kinds of messages in the test bed. One message can give one response or More than one response based on a slash delimiter in the data field.
There is an xsl node + compute node and an mq mapping node being used in the message flow. |
|
| Back to top |
|
|
| riteshprasad1 |
| Posted: Mon Jun 05, 2017 10:50 am Post subject: |
|
|
Apprentice
Joined: 31 Jul 2014
Posts: 25
|
| Please post examples or esql scenarios of making the code thread safe... |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jun 05, 2017 11:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So you are receiving *one* response message with more than one record in it?
Or more than one response message, with one or more records in each?
Threading only affects performance in IIB the same way it affects performance elsewhere.
Also, if you are using an XSLTransform node, that's probably a big part of your performance issue. Don't use that.
_________________
chmod -R ugo-wx /
Last edited by mqjeff on Mon Jun 05, 2017 11:02 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 05, 2017 11:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| riteshprasad1 wrote: |
| There is an xsl node |
Seriously? Why?
| riteshprasad1 wrote: |
| compute node and an mq mapping node being used in the message flow. |
Is that a mapping node? Or is there a specific MQ Mapping node I'm not aware of.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 05, 2017 11:05 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| riteshprasad1 wrote: |
| Please post examples or esql scenarios of making the code thread safe... |
I said you had a message affinity problem and/or a thread safety issue.
In light of the additional information you've dripped, I don't think you have a threading problem.
I agree with my most worthy associate - you have an affinity issue and a bad way of handling it.
You also have an XSLT node which is doing your performance (and indeed your maintainability and code quality) no favors at all.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| riteshprasad1 |
| Posted: Mon Jun 05, 2017 11:44 am Post subject: |
|
|
Apprentice
Joined: 31 Jul 2014
Posts: 25
|
I tried out a couple of more things.
Made a test bed of one message expecting one response. Pushed 200 same messages to a temp queue and then moved all the messages using rfhutil to the input queue of the flow.
Order mode set to Default - I got valid 200 responses.
Add Additional instance set to 20.
Result was valid 200 responses and fast performance.
Made a test bed of one message expecting 10 responses. Pushed 200 same messages to a temp queue and then moved all the messages using rfhutil to the input queue of the flow.
Order mode set to Default - I got valid 2000 responses.
Add Additional instance set to 20.
Result was valid 2000 responses and fast performance.
When I am trying messages with a random combination of both types of messages with Order mode as default I get invalid responses.
Did not ponder on the xslTransform node because it gave me valid responses with single thread or multithread with order mode as user id or queue order or user defined. Will it be worth a try to replace xsl node to ESQL? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 05, 2017 11:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| riteshprasad1 wrote: |
| Will it be worth a try to replace xsl node to ESQL? |
For so many good reasons, not the least of which is it will be a good deal easier to diagnose your problem in ESQL (which you can trace) than an XSLT (which you can't).
You may also find a single thread starts giving you acceptable performance.....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| riteshprasad1 |
| Posted: Sat Jun 10, 2017 8:31 pm Post subject: |
|
|
Apprentice
Joined: 31 Jul 2014
Posts: 25
|
| Vitor wrote: |
| riteshprasad1 wrote: |
| Will it be worth a try to replace xsl node to ESQL? |
For so many good reasons, not the least of which is it will be a good deal easier to diagnose your problem in ESQL (which you can trace) than an XSLT (which you can't).
You may also find a single thread starts giving you acceptable performance..... |
Slogged for continuous hours to convert the xsl to esql. No respite. facing the same issue..... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jun 11, 2017 7:36 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| riteshprasad1 wrote: |
| Vitor wrote: |
| riteshprasad1 wrote: |
| Will it be worth a try to replace xsl node to ESQL? |
For so many good reasons, not the least of which is it will be a good deal easier to diagnose your problem in ESQL (which you can trace) than an XSLT (which you can't).
You may also find a single thread starts giving you acceptable performance..... |
Slogged for continuous hours to convert the xsl to esql. No respite. facing the same issue..... |
May I suggest following: it looks like you have 2 different type of messages. Why not create a dispatcher flow that will separate the messages per type into 2 different queues. Then you can have 2 different instances of the flow possibly in separate egs handle each only one type of message. Looks like you had no problem doing that... Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
