| Author |
Message
|
| sebastia |
 Posted: Fri Mar 14, 2014 2:08 am Post subject: message count when multiple instances |
 |
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Hello.
We have a flow and want to count the number of input messages it has managed.
So, we need a counter in global memory, right ?
Second, we shall run lots of instances (more than 50) of that flow, so we do not want colisions on that variable.
In a "classical" environment, I would have an "index" somehow related to each instance and then increment "input_count[my_index]" on every instance.
How can we do this in MB flow ?
Sebastian. |
|
| Back to top |
|
 |
| zpat |
| Posted: Fri Mar 14, 2014 2:43 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You can use a shared variable (type of external).
To update it without collisions, code a BEGIN ATOMIC around (just) the update.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| sebastia |
| Posted: Fri Mar 14, 2014 2:47 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Hi, Zpat.
I said "global var" and you use "shared var" but it is the same thing.
We do NOT want to use ATOMIC because 50 or 80 flows increasing the same variable shall have too many collisions and delays.
Any other solution ? |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Mar 14, 2014 3:04 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
The time taken to update a shared (i.e. EXTERNAL in ESQL terms) variable will be trivial. I don't think you would see any measureable delays. Have you tried it?
Shared variables are EG specific and the overhead to update it is extremely low.
If you mean something more global with significant amounts of product code to provide global serialisation, then it might affect performance.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| sebastia |
| Posted: Fri Mar 14, 2014 3:11 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
yes, we both talk about the same ESQL "EXTERNAL" var ...
Lets have some figures
70 instances
7.500 messages/second
all of them doing "set num = num + 1"
I shall have to test, but dont like it
Tx |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Mar 14, 2014 3:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| sebastia wrote: |
yes, we both talk about the same ESQL "EXTERNAL" var ...
Lets have some figures
70 instances
7.500 messages/second
all of them doing "set num = num + 1"
I shall have to test, but dont like it
Tx |
May I remind you that dot and comma have different meaning in different language?
To be clear, are we talking about seven and a half messages per second or seven thousand five hundred? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| stoney |
| Posted: Fri Mar 14, 2014 3:21 am Post subject: |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
| If you don't mind using Java, you could use a static ThreadLocal object to provide a per-thread counter. |
|
| Back to top |
|
|
| sebastia |
| Posted: Fri Mar 14, 2014 3:38 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Tx both
mr Saper - we are talking about 7 thousand 5 hundred messages per second, each being about 5 Kbytes. My fault, sorry
mr Stoney - I try avoiding Java as a rule, but almost never succeeding ... jejeje |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Mar 14, 2014 7:11 am Post subject: Re: message count when multiple instances |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| sebastia wrote: |
We have a flow and want to count the number of input messages it has managed.
So, we need a counter in global memory, right ? |
From developer's point of view, perhaps.
But what is the exact requirement? Does the flow itself, while processsing messages, need to know exactly how many messages it has processed so far?
Or do people need to know how many messages the flow has processed -- today, for example?
Have you considered using broker statistics or monitoring events?
In-memory counters are reset when you restart the broker or eg or when you redeploy the flow ... or when the eg crashes. Is that acceptable? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 14, 2014 7:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
If you just need to know the number of input transactions, why not obtain that from the accounting and statistics functions of WMB rather than an in-code solution?
Before someone says that this data is needed for something other than the MB Explorer, I'm suggesting you take the data as XML, process it and stick it wherever your final destination is.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Mar 14, 2014 7:27 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
If you just need to know the number of input transactions, why not obtain that from the accounting and statistics functions of WMB rather than an in-code solution?
Before someone says that this data is needed for something other than the MB Explorer, I'm suggesting you take the data as XML, process it and stick it wherever your final destination is. |
it's also published, not distributed. So you can subscribe as many times as you want from as many things as you want. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 14, 2014 8:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Vitor wrote: |
If you just need to know the number of input transactions, why not obtain that from the accounting and statistics functions of WMB rather than an in-code solution?
Before someone says that this data is needed for something other than the MB Explorer, I'm suggesting you take the data as XML, process it and stick it wherever your final destination is. |
it's also published, not distributed. So you can subscribe as many times as you want from as many things as you want. |
 Quite.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sebastia |
| Posted: Sun Mar 16, 2014 12:56 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Thanks for your opinions, coleagues !
If it were to be just input/output message count, then we could think on monitor statistics, of course. We use them to verify the EG is "alive and running".
But we want "application events" accounting.
As an example, we want to know how many times we have received this kind of message and it has ended up in triggering that subflow or event.
We hope the EG to run almost forever, so losing the counters when the EG is stopped shall be acceptable, mr ESA. |
|
| Back to top |
|
|
| Esa |
| Posted: Sun Mar 16, 2014 11:29 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| sebastia wrote: |
But we want "application events" accounting.
As an example, we want to know how many times we have received this kind of message and it has ended up in triggering that subflow or event.
|
Sounds like a business case for monitoring events.
Or rather, because of the heavy load of messages, a combination of shared variables and monitoring events. Like propagating to a passthrough node each time a shared variable starts a new hundred and monitoring the passthough node. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 17, 2014 4:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sebastia wrote: |
But we want "application events" accounting.
As an example, we want to know how many times we have received this kind of message and it has ended up in triggering that subflow or event. |
So use monitoring events rather than accounting stats.
Same advantages apply.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
