| Author |
Message
|
| sselva86 |
 Posted: Tue Jul 12, 2016 1:38 am Post subject: Broker - Inputroot and Outputroot throuhput |
 |
|
Novice
Joined: 25 Oct 2010
Posts: 17
|
Hello all ,
I have small problem with inputroot and outputroot in compute node .
Have 3 compute nodes each have 6k inputroot and outputroot assignments. Each time when i run EG with 2 mb of data its taking 300 + milliseconds .
so totaly its taking .9 second to process single message .Is there any property which can improve the performance . Hope 18k of assignments is not that much big for broker in single run .
Have tried to increase the heapsize for broker and execution group .
for broker : 16mb min and 1gb as max
for execution group : 16mb as min and 60mb as max . |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Tue Jul 12, 2016 3:02 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Heapsize has zero effect if the compute node is an ESQL one.
You could copy the InputRoot into the Environment. That way no copying is done
But honestly and I am sure that others will agree, there is something else wrong with your system. Many of us run systems that process hundreds of messages a second.
Are you saying that your ESQL is 6K long? What do you mean by 'assignments'?
Can you show us some of your code?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| sselva86 |
| Posted: Tue Jul 12, 2016 4:19 am Post subject: |
|
|
Novice
Joined: 25 Oct 2010
Posts: 17
|
Hi Jedi ,
Yes ESQL is having 6k linkes
sample :
CREATE COMPUTE MODULE LOADTEST_Compute
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
-- CALL CopyMessageHeaders();
-- CALL CopyEntireMessage();
SET OutputRoot.XMLNSC.cust.KEY = InputRoot.XMLNSC.CACust.KEY;
SET OutputRoot.XMLNSC.Cust.REC0001 = InputRoot.XMLNSC.CACust.REC0001;
SET OutputRoot.XMLNSC.Cust.REC0002 = InputRoot.XMLNSC.CACust.REC0002;
SET OutputRoot.XMLNSC.Cust.REC0003 = InputRoot.XMLNSC.CACust.REC0003;
.
.
SET OutputRoot.XMLNSC.Cust.REC6000 = InputRoot.XMLNSC.CACust.REC6000;
RETURN TRUE;
END;
assignments i am about to tell assigning values to outputroot from inputroot .
3 compute nodes are having this kind of ESQL code . when message process through its taking 300 miliseconds in each compute node .
we are trying to fine tune this , hope broker will not take 300 milisecond to process just 6k lines .
your help is greatly appreciated guys .  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 12, 2016 4:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Use reference variables.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jul 12, 2016 4:36 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| sselva86 wrote: |
Hi Jedi ,
Yes ESQL is having 6k linkes
sample :
| Code: |
CREATE COMPUTE MODULE LOADTEST_Compute
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
-- CALL CopyMessageHeaders();
-- CALL CopyEntireMessage();
SET OutputRoot.XMLNSC.cust.KEY = InputRoot.XMLNSC.CACust.KEY;
SET OutputRoot.XMLNSC.Cust.REC0001 = InputRoot.XMLNSC.CACust.REC0001;
SET OutputRoot.XMLNSC.Cust.REC0002 = InputRoot.XMLNSC.CACust.REC0002;
SET OutputRoot.XMLNSC.Cust.REC0003 = InputRoot.XMLNSC.CACust.REC0003;
.
.
SET OutputRoot.XMLNSC.Cust.REC6000 = InputRoot.XMLNSC.CACust.REC6000;
RETURN TRUE;
END; |
your help is greatly appreciated guys . |
Looks messy. Wouldn't a one liner do the deed here?
| Code: |
| SET OutputRoot.XMLNSC.Cust = InputRoot.XMLNSC.CACust; |
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sselva86 |
| Posted: Tue Jul 12, 2016 8:30 pm Post subject: |
|
|
Novice
Joined: 25 Oct 2010
Posts: 17
|
Thanks Jeff ,
@fjp_saper : yes that can do but what we are trying to accomplish is to understand how much broker will take to process 6k lines . In future we are planning to do some mapping for 6k lines , each line will have some mapping like adding some decimal values or concatenation of some values etc.,
If broker took 300+ miliseconds for 6k lines then it may take more time when we implement future enhancements . So we are trying to fine tune this.
your expertise is needed . |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 12, 2016 9:50 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| sselva86 wrote: |
If broker took 300+ miliseconds for 6k lines then it may take more time when we implement future enhancements . So we are trying to fine tune this.
your expertise is needed . |
As has been said, you will certainly improve performance if you use references.
Go on, give it a try. You may be surprised by what to see.
I'm just guessing here but you are trying some worst case scenarios. But you really need someone with years of experience in this product to worl alongside you in order to get the best throughput.
There are all sorts of tricks that can be used to speed up processing but you need to know what you are doing with the product. Again just guessing, I assume that you don't have this experience.
Without us knowing the message structure in detail and what fields you need to update it is hard to give you anything more than general hints.
Also, just timing one message or even a 100 messages won't give you a true picture of the performance. you need to process thousands of messages and apply some form of statistical normalisation to the results in order to get a true feel for what is going on. This all takes time, lots of time to do it properly.
There was one system I worked on and tested to the 'Nth' degree in order to reach the contracted performance levels. We looked at everything. not just the message flows but the networking, the storage and ... well everything. It took two of us three months but we got there in the end. We went from 16 messages/sec to 56/sec.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| sselva86 |
| Posted: Wed Jul 13, 2016 12:26 am Post subject: |
|
|
Novice
Joined: 25 Oct 2010
Posts: 17
|
Hi smdavies99 ,
yes , i am trying worst case scenario , this is our first try with IIB .
message format which we are trying here is XML , normal xml format .
<cust>
<KEY>1</KEY>
<DESC>AUTO</DESC>
<REC0001>AAAAAAAAAAAAAAAAAAAAA</REC0001>
<REC0002>AAAAAAAAAAAAAAAAAAAAA</REC0002>
.
.
.
<REC6000>AAAAAAAAAAAAAAAAAAAAA</REC6000>
</cust>
and trying to update all fields ( 6k fields ) . one to one assignment .
I am trying to use references ,will share the results to you . if you have some more information please share and help |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jul 13, 2016 3:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| sselva86 wrote: |
| if you have some more information please share and help |
Use a reference to point to locations in the Input tree as well as in the output tree.
Using [n] causes the ESQL parser to count from FIRSTCHILD all the way up to N. If you then ask for [n+1], it starts over and counts again.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| mgk |
| Posted: Wed Jul 13, 2016 4:17 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1638
|
| Quote: |
| and trying to update all fields ( 6k fields ) . one to one assignment |
If this is all you are doing, a SELECT may well be the fastest way to achieve this.
Kind regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| sselva86 |
| Posted: Wed Jul 13, 2016 8:51 pm Post subject: |
|
|
Novice
Joined: 25 Oct 2010
Posts: 17
|
Hi All ,
References made the difference . now time taken is 160 miliseconds . thanks all for your help guys .
@mgk : will try to use select to check the fast .
will update you guys |
|
| Back to top |
|
|
| sselva86 |
| Posted: Sun Jul 17, 2016 8:53 pm Post subject: |
|
|
Novice
Joined: 25 Oct 2010
Posts: 17
|
| Seems references working is giving good results .it reduces 50% of the time . thanks for your help experts. |
|
| Back to top |
|
|
|
|
