| Author |
Message
|
| dprogwmb |
 Posted: Wed Sep 14, 2011 7:04 am Post subject: Question About Loggin Statistics of Message Flows |
 |
|
Voyager
Joined: 19 Jul 2011
Posts: 96
|
Hi guys,
In my job, I was asked to develop a kind of General Framework on Websphere Message Broker to Log certains aspects of the execution of the message flows...I'm planning to use the log4J Pluggin node for that.
But I was asked in particular to devolop the following functionality:
One metrics log (oriented to gerencial people): with the name of the interface, Start and End timestamps of execution, Success /Failures (of the whole process), quantity of times executes per day (of the whole process), processing media time, etc...
Definitions:
*A whole process is a list of activities ... a whole process can be for example : insert a tax payer (there you have to insert the person on SAP, and then on RH system, etc... this would be the activities)
*An activity: Is for example make an soap request to one application
How Can I develop it? Any ideas or starting points? Any links?
Is it possible to develop a generic flow that collects and makes some statical processing for all of the other flows of one Message Broker in particular?
I hope anyone can help me!!!
REGARDSSSS |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Sep 14, 2011 7:06 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Abandon all notions of using Log4J and of embedding this inside your message flows.
Review the information in the Information center on Montioring events, and enable those in your flows instead.
And then think outside of your tiny little "everything must be java, so I'll use log4j" world, and consider purchasing a real monitoring tool, like Tivoli, to capture and store and react to this information.
You are going down the wrong path if you try and build this yourself. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Sep 14, 2011 7:11 am Post subject: Re: Question About Loggin Statistics of Message Flows |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| dprogwmb wrote: |
| I'm planning to use the log4J Pluggin node for that. |
Yucky pooey.
If you use log4j, use it directly, don't use the fix pack.
| mqjeff wrote: |
| consider purchasing a real monitoring tool, like Tivoli, to capture and store and react to this information. |
Good advice. I agree.
| mqjeff wrote: |
| Abandon all notions of using Log4J |
Bad advice. I disagree. Log4j has a place in message broker development; however the OP chose the wrong place.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 14, 2011 7:14 am Post subject: Re: Question About Loggin Statistics of Message Flows |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| mqjeff wrote: |
| Abandon all notions of using Log4J |
Bad advice. I disagree. Log4j has a place in message broker development; however the OP chose the wrong place. |
Log4j only has a place in message broker development where there is any existing investment in using Java in the first place.
It is entirely the wrong idea to introduce Java simply for the purposes of using Log4j - and bad advice to suggest it without knowing before hand if such investment has been made.
This also holds true of Singletons. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Sep 14, 2011 9:00 am Post subject: Re: Question About Loggin Statistics of Message Flows |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mqjeff wrote: |
| lancelotlinc wrote: |
| mqjeff wrote: |
| Abandon all notions of using Log4J |
Bad advice. I disagree. Log4j has a place in message broker development; however the OP chose the wrong place. |
Log4j only has a place in message broker development where there is any existing investment in using Java in the first place.
It is entirely the wrong idea to introduce Java simply for the purposes of using Log4j - and bad advice to suggest it without knowing before hand if such investment has been made.
This also holds true of Singletons. |
A programming language is simply a means to an end, it is not an end. You can have a Singleton programmed in ESQL and you can have an ESQL Compute node issue calls to the log4j jar directly. WMB pallette contains many options for flow construction. None of the nodes are mutually exclusive of the other.
Even COBOL is still a useful language today.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| dprogwmb |
| Posted: Wed Sep 14, 2011 9:25 am Post subject: No money for Tivoli :( |
|
|
Voyager
Joined: 19 Jul 2011
Posts: 96
|
| mqjeff wrote: |
Abandon all notions of using Log4J and of embedding this inside your message flows.
Review the information in the Information center on Montioring events, and enable those in your flows instead.
And then think outside of your tiny little "everything must be java, so I'll use log4j" world, and consider purchasing a real monitoring tool, like Tivoli, to capture and store and react to this information.
You are going down the wrong path if you try and build this yourself. |
Ok, thanks for the answer... the use of log4J, I'm planning to make it using log4J1_1 function, that come with the pluggin node...
On the other hand, the "client" organization doesn't have anymore money to buy anymore licenses (for example Tivoli ITCam for SOA) , and I was asked to fix this problem, without using any tool that represent spending money... any other idea? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Sep 14, 2011 9:40 am Post subject: Re: No money for Tivoli :( |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| dprogwmb wrote: |
On the other hand, the "client" organization doesn't have anymore money to buy anymore licenses (for example Tivoli ITCam for SOA) , and I was asked to fix this problem, without using any tool that represent spending money... any other idea? |
Time to look for a new role then. As has been stated there is an awful lot more to this than you realise. To give the client a proper solution will cost them a lot of your time and there is even a risk that you might not get it working properly in the time allowed. Some of the data you have stated that must be 'recorded' is well outside most normal solutions. You may well have to write code and make changes to the code to enable this.
But I'd look at the possibilities that Terminal Events might give you for starters.
As you are coming at this cold, you need to experiment. Try things out. Don't go with the first solution you dream up. It might not work in reality.
On the other hand, get reviving your CV.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| dprogwmb |
| Posted: Wed Sep 14, 2011 10:54 am Post subject: Re: No money for Tivoli :( |
|
|
Voyager
Joined: 19 Jul 2011
Posts: 96
|
| smdavies99 wrote: |
| dprogwmb wrote: |
On the other hand, the "client" organization doesn't have anymore money to buy anymore licenses (for example Tivoli ITCam for SOA) , and I was asked to fix this problem, without using any tool that represent spending money... any other idea? |
Time to look for a new role then. As has been stated there is an awful lot more to this than you realise. To give the client a proper solution will cost them a lot of your time and there is even a risk that you might not get it working properly in the time allowed. Some of the data you have stated that must be 'recorded' is well outside most normal solutions. You may well have to write code and make changes to the code to enable this.
But I'd look at the possibilities that Terminal Events might give you for starters.
As you are coming at this cold, you need to experiment. Try things out. Don't go with the first solution you dream up. It might not work in reality.
On the other hand, get reviving your CV. |
OK... so the option is to use automatized options ...
I will read a bit about the monitored events and statistics on message broker... the problem is that the information that is genereted of those statistics (for example how many times a flow was executed in a day) will be lost from the analitics view... (if I can't use any kind of automatize analitic solution...  .... any other idea or tip? (other than get a new job or kill myself =) ) |
|
| Back to top |
|
|
| goffinf |
| Posted: Wed Sep 14, 2011 12:10 pm Post subject: |
|
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
We use the monitoring events which are part of the standard product and therefore quick and easy to implement, for basic usage statistics (e.g. Number of invocations per flow per execution group per hour, and also basic thruput info like average time spent in flow). To do that we build a relatively simple set of flows that subscribe to the monitoring events and (using Collector nodes) create aggregated information which is eventually written to a database. The database supports simple analytics and reporting (dashboard'esk). In the future we may use this data for other purposes like a charging model.
Pretty straight-forward development for a competent MB Dev.
I appreciate I've skirted over the detail, but it might give you an idea of what is possible using existing and supported capabilities of the product.
HTH
Feasted |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Sep 14, 2011 12:19 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
If you use FlowMonitoring events to obtain performance data then you will get incorrect results.
The End Transaction or Rollback event messages do indeed have a timestamp but this may not actually be the time when the thread ended. It can be sometime later.
You are going to have to use a combination of events and other things in order to assemble all the data you have specified. You are in effect re-inventing tools like Tivoli. This is no small task.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| dprogwmb |
| Posted: Wed Sep 14, 2011 12:20 pm Post subject: |
|
|
Voyager
Joined: 19 Jul 2011
Posts: 96
|
| goffinf wrote: |
We use the monitoring events which are part of the standard product and therefore quick and easy to implement, for basic usage statistics (e.g. Number of invocations per flow per execution group per hour, and also basic thruput info like average time spent in flow). To do that we build a relatively simple set of flows that subscribe to the monitoring events and (using Collector nodes) create aggregated information which is eventually written to a database. The database supports simple analytics and reporting (dashboard'esk). In the future we may use this data for other purposes like a charging model.
Pretty straight-forward development for a competent MB Dev.
I appreciate I've skirted over the detail, but it might give you an idea of what is possible using existing and supported capabilities of the product.
HTH
Feasted |
Ok, very good answer... I didn't understand this part: "(using Collector nodes) create aggregated information which is eventually written to a database" can you expain it to me a bit more, please... Do you have any link or suggestion to start with this matter? Or any good example in which you based to develop your "monitoring system"? REGARDSSSS |
|
| Back to top |
|
|
| dprogwmb |
| Posted: Wed Sep 14, 2011 12:24 pm Post subject: |
|
|
Voyager
Joined: 19 Jul 2011
Posts: 96
|
| smdavies99 wrote: |
If you use FlowMonitoring events to obtain performance data then you will get incorrect results.
The End Transaction or Rollback event messages do indeed have a timestamp but this may not actually be the time when the thread ended. It can be sometime later.
You are going to have to use a combination of events and other things in order to assemble all the data you have specified. You are in effect re-inventing tools like Tivoli. This is no small task. |
Which Tivolli tool would you recommend? (Tivoli has a very big set of tools - IT CAM for SOA , IT CAM for Transactions, etc....) |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Sep 14, 2011 12:37 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dprogwmb wrote: |
| can you expain it to me a bit more, please... |
I don't know if this is what goffinf is doing, but one of the monitoring functions here is timings for a flow, principally how much of the time a flow is taking is spent waiting for a response from an external web service. The flow uses a Collector node to assemble all the events from a flow by event id, and takes the timings from the creationTime of the event. It's not especially accurate, but the requirements don't call for it.
We also used to count how many times a given flow was executed in a given business day & email the result to the department head. I was never sure why and after a few months he decided he didn't need to know any more.
We also keep 30 days worth of event data in a database (inserted by a flow) and anyone with curisority imports it into a Excel spreadsheet in whatever way meets their needs.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Sep 15, 2011 4:28 am Post subject: Re: No money for Tivoli :( |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| dprogwmb wrote: |
| mqjeff wrote: |
Abandon all notions of using Log4J and of embedding this inside your message flows.
Review the information in the Information center on Montioring events, and enable those in your flows instead.
And then think outside of your tiny little "everything must be java, so I'll use log4j" world, and consider purchasing a real monitoring tool, like Tivoli, to capture and store and react to this information.
You are going down the wrong path if you try and build this yourself. |
Ok, thanks for the answer... the use of log4J, I'm planning to make it using log4J1_1 function, that come with the pluggin node...
On the other hand, the "client" organization doesn't have anymore money to buy anymore licenses (for example Tivoli ITCam for SOA) , and I was asked to fix this problem, without using any tool that represent spending money... any other idea? |
Don't use the plugin node. Use log4j directly. Or, had you missed my earlier point?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| goffinf |
| Posted: Fri Sep 16, 2011 12:34 am Post subject: |
|
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
| Vitor wrote: |
| I don't know if this is what goffinf is doing ... |
Yeah pretty much. The first flow subscribes to the monitoring events and uses a Collector to relate TransactionStat/End/Rollback pairs based on the local transaction id. This is the 'hottest' of the flows since it is dealing with all the events. It outputs a single message per pair which contains the time taken for the transaction. The second flow takes the messages from flow 1 and using another Collector relates them by local transaction id and calculates the total invocation count, failure count and time taken for each flow for each day/hour. This Collector creates an output message based on the quantity property (I think its set to something like 500). The thrird collector batches message using a timer so that we can control how often the database is updated. The final flow inserts into the database. |
|
| Back to top |
|
|
|
|
