| Author |
Message
|
| Partha.Baidya |
 Posted: Mon Feb 22, 2016 10:48 am Post subject: IIB Logging Framework: log4j vs. IIB Event based logging |
 |
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
I want building a new integration solution for a client using IIBv9, I want to build the logging framework using IIB events based framework instead of log4j utility. I am planning to capture the IIB events and put them in Database instead of using lag4j utilty and creating logging file?
Could you please let me know if there is a disadvantage of using event based logging instead of log4j? |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon Feb 22, 2016 11:07 am Post subject: Re: IIB Logging Framework: log4j vs. IIB Event based logging |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Partha.Baidya wrote: |
| Could you please let me know if there is a disadvantage of using event based logging instead of log4j? |
Event based logging is faster, asynchronous, much more flexible, is pub/sub based and so can be received by multiple consumers for multiple types of responses, can even in theory (unless I remember wrong) be stored using the Record/Reply features, can even be processed by IIB since it's MQ based...
Oh. Wait. You said disadvantage... 
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Partha.Baidya |
| Posted: Mon Feb 22, 2016 3:00 pm Post subject: |
|
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
Thanks mqjeff, for quick reply.
Is there any other way to implement IIB logging framework, apart from log4j? |
|
| Back to top |
|
|
| Partha.Baidya |
| Posted: Mon Feb 22, 2016 6:03 pm Post subject: |
|
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
IIB/WMB support some basic logging mechanism which is not sufficeint in a large organisation.
I can think about the below solution for logging framework
1. using log4j java classes to log message flow activities to files?
2. using IIB event monitoring feature and write the events to Database.
3. putting messages using WMQ queues and write these messages to Database using some other programs.
Could you please suggest if any other mechanism is available and best approach for implementation a logging framework?
Regards,
Partha
Integration/ESB Architect |
|
| Back to top |
|
|
| timber |
| Posted: Tue Feb 23, 2016 1:30 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| IIB/WMB support some basic logging mechanism which is not sufficeint in a large organisation. |
More details, please. What is not sufficient, and why? |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue Feb 23, 2016 2:43 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
Heavy user of monitoring (also know as record and replay) here 
We have a standard in our company. Every message flow developed must comply with the policies on this standard.
One of the policies defines which nodes in the message flow should have event monitoring enabled , which events are triggered and which information should be recorded.
Example: this policies defines that every soap request, soap reply, soap input, endpoint lookup should have monitoring enabled
Also it defined that for every soap reply node, the event "terminal in", "failure" and "terminal out" should be recorded. This policie defines that the event "terminal in" from every soap reply node should record the payload in base64 format.
In the end of the day we end up with a database of events we can analyze, tha advantages we recognize:
- Quick and flexible log analyze (just use SQL, searching in files is so ... vintage)
- We can enable and disable monitoring on the fly (throug mqsi commands, no restarts needed, no redeploys needed)
- No perfomance problems on using monitoring. The monitoring "infra-structure" opearates in an Async fashion, so you don't have to wait for the database to complete the insert nor you get application errors when database is out of space.
- You can have multiple consumers of this info (excel reports, devops monitoring plataforms), and you can access it via sql or connect directly to MQ (just subscribe a topic - pub/sub)
Caution: if the monitoring information is not enough to understand/trace problems do not instruct monitoring to gather more and more events. In this cases what you are looking for is a service trace. Keep the focus, this is auditing/high level log information , it should not be used for complete debug (ok , you can do it with monitoring but there are better ways)
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| Partha.Baidya |
| Posted: Tue Feb 23, 2016 10:34 am Post subject: |
|
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
| timber wrote: |
| Quote: |
| IIB/WMB support some basic logging mechanism which is not sufficeint in a large organisation. |
More details, please. What is not sufficient, and why? |
IIB has it s own logging capabilities like writing logs to Operating System event logs/syslog where the error or exception messages get logged. Even we can use trace logs to some files.
But this level of logging details does not give a end to end view of what is going on with a message coming to IIB. We can not use this logs to monitor SLAs and performance of the message flow. It can not be used to create a Dashboard of system monitoring.
We are planning to build a logging infrastructure with end to end visibility of the entire message flow. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Feb 23, 2016 11:12 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Partha.Baidya wrote: |
| timber wrote: |
| Quote: |
| IIB/WMB support some basic logging mechanism which is not sufficeint in a large organisation. |
More details, please. What is not sufficient, and why? |
IIB has it s own logging capabilities like writing logs to Operating System event logs/syslog where the error or exception messages get logged. Even we can use trace logs to some files.
But this level of logging details does not give a end to end view of what is going on with a message coming to IIB. We can not use this logs to monitor SLAs and performance of the message flow. It can not be used to create a Dashboard of system monitoring.
We are planning to build a logging infrastructure with end to end visibility of the entire message flow. |
Monitoring should not be the primary objective of such a system. I see it more as audit.
For monitoring you do have activity statistics that are published by the IIB node and server about the flows and can be turned on / off at a flow level.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 23, 2016 11:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Partha.Baidya wrote: |
| But this level of logging details does not give a end to end view of what is going on with a message coming to IIB. We can not use this logs to monitor SLAs and performance of the message flow. It can not be used to create a Dashboard of system monitoring. |
That's not logging - that's business activity monitoring.
| Partha.Baidya wrote: |
| We are planning to build a logging infrastructure with end to end visibility of the entire message flow. |
I'm sure you'll build a very nice wheel.
Before you go too far, you might want to investigate some of the commercial products available that do exactly this. Especially if you want end to end visibility of your entire transactional flow, not just the part that runs through IIB, which these products provide.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| inMo |
| Posted: Tue Feb 23, 2016 11:52 am Post subject: |
|
|
Master
Joined: 27 Jun 2009
Posts: 216
Location: NY
|
| Vitor wrote: |
I'm sure you'll build a very nice wheel.
|
That's funny |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 23, 2016 12:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| inMo wrote: |
| Vitor wrote: |
I'm sure you'll build a very nice wheel.
|
That's funny |
I'm a funny guy.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Partha.Baidya |
| Posted: Tue Feb 23, 2016 12:34 pm Post subject: |
|
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
You are correct, There is Websphere Business Monitor tool available which can be integrated for monitoring those activities.
If you take WMB itself it is logging to event logs. If I want to customize the logging then I have to use trace nodes or log4j utility and write some codes to log the information.
Do you we have any other option to create a custom logging framework?
n?
Last edited by Partha.Baidya on Tue Feb 23, 2016 12:38 pm; edited 1 time in total |
|
| Back to top |
|
|
| Partha.Baidya |
| Posted: Tue Feb 23, 2016 12:37 pm Post subject: |
|
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
| Some customer does not want to use BAM products as well. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 23, 2016 12:41 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Partha.Baidya wrote: |
| If I want to customize the logging then I have to use trace nodes or log4j utility and write some codes to log the information. |
Don't use Trace nodes. Aside from the lack of flexibility, the memory cost of these nodes with production size volumes is prohibitive.
| Partha.Baidya wrote: |
| Do you we have any other option to create a custom logging framework? |
Aside from studding your code with log4j calls or intercepting event messages? Not really.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 23, 2016 12:45 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Partha.Baidya wrote: |
| Some customer does not want to use BAM products as well. |
It never ceases to amaze me (and it's been amazing me for some years now) how some customers insist on building their own solutions to problems that have a perfectly acceptable commercial solution. You try and explain "total cost of ownership" and it's like you're speaking Martian.
Often of course, the person making the choice to build in house (typically cheaper) isn't the same guy saddled with the support costs over the years to come (typically more expensive).
And while IBM BAM is such a solution, it's not the only available solution.
Enjoy inventing your wheel.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
