| Author |
Message
|
| sleepyjamie |
 Posted: Thu Sep 10, 2015 6:34 am Post subject: Log4j2 and IIB9 |
 |
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
I'm not sure where you guys post code examples. I've created some sample code on how to use Log4j2 and IIBv9.
https://github.com/jamie3/iib-log4j2
I will be creating this as a plugin shortly. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Sep 10, 2015 6:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What are the strong advantages of using Log4J (2 or not) over using java.util.logging?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Thu Sep 10, 2015 6:49 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| mqjeff wrote: |
| What are the strong advantages of using Log4J (2 or not) over using java.util.logging? |
For our system we want to have a lot more logging enabled in production operations. Log4j2 will provides much better performance for us.
Here is from log4j2 website.
Log4j 2 contains next-generation Asynchronous Loggers based on the LMAX Disruptor library. In multi-threaded scenarios Asynchronous Loggers have 18 times higher throughput and orders of magnitude lower latency than Log4j 1.x and Logback. See Asynchronous Logging Performance for details. Otherwise, Log4j 2 performs faster than Log4j 1.x in critical areas and similarly to Logback under most circumstances. See Performance for more information. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 10, 2015 6:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Those only talk about previous versions of Log4j and Logback.
Not the native java.util.logging.
If you've already got tools and infrastructure to manage Log4j, then there probably isn't a compelling reason to switch.
If you don't, then it's significantly more work to add to IIB - you have to maintain jar files in the right classpath locations at the right levels and maintain the properties files and and and.
If you're using a UDN, then you also need to maintain the UDN at the right level.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Thu Sep 10, 2015 7:01 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
In the Maven/Java world its easy as adding a dependency into a pom.xml. This is where I find IIB lacks the power to easily extend it with 3rd party APIs. This is something IBM says they are addressing in IIBv10.
Also here are some benchmarks between Log4j2 and JUL. Log4
http://blog.takipi.com/the-logging-olympics-a-race-between-todays-top-5-logging-frameworks/
Our requirement is logging close to a GB of log messages a day in production. We can use JUL, but a 10-15% performance impact is significant. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 10, 2015 7:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I hope that a GIG of log messages is the minimum data necessary for business integrity, and not an ongoing stream of debugging information.
I'd also be really surprised if all of that Gig was coming from one EG/integration server.
Spread out over more than one EG, the performance impact should be less significant.
The big thing to remember about IIB is that it's not just a java runtime, and to expect it to act exactly like any other java runtime is a bit out of sync with the facts.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 10, 2015 7:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sleepyjamie wrote: |
| Our requirement is logging close to a GB of log messages a day in production. |
If I had close to a Gb of log messages a day in production, I'd wonder why and I'd certainly look for something other than a plain text file to keep all that data in. Something with (for example) good search facilities.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Thu Sep 10, 2015 7:23 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| mqjeff wrote: |
I hope that a GIG of log messages is the minimum data necessary for business integrity, and not an ongoing stream of debugging information.
I'd also be really surprised if all of that Gig was coming from one EG/integration server.
Spread out over more than one EG, the performance impact should be less significant.
The big thing to remember about IIB is that it's not just a java runtime, and to expect it to act exactly like any other java runtime is a bit out of sync with the facts. |
True, I know IIB is not a Java runtime, however IMO the toolkit can be improved significantly to enable more rapid development for working with 3rd party APIs. Originally Java was not friendly either, and hence Maven was created. And Maven is no silver bullet, but it has dramatically made Java more friendly when you want to go beyond just the simple JDK APIs. I would like to see IIB have something similar where you can easily fetch plugins / subflows nodes etc.. from a repository such as maven central or something internal.
And yes this is not debugging information. The logging data will contain Application Performance Monitoring, KPIs, Business Analytics, measuring SLAs, etc.. |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Thu Sep 10, 2015 7:26 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| Vitor wrote: |
| sleepyjamie wrote: |
| Our requirement is logging close to a GB of log messages a day in production. |
If I had close to a Gb of log messages a day in production, I'd wonder why and I'd certainly look for something other than a plain text file to keep all that data in. Something with (for example) good search facilities. |
We are sending all of our logging data from all of our systems to Splunk/Logstash and Graphana/InfluxDB.
Our logging data include, MQ, WebSphere ODM, IIB, WAS, etc...
These systems also have alerting built in so when there are patterns detected in the logs DevOps is notified of any potential issues. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 10, 2015 7:34 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I might still look at IIB monitoring events and stats messages to gather the log data you need for IIB apps.
You can then use a logstash input to pull the messages from a queue, which again will happen asynchronously and could happen on an entirely separate machine. And use a standard syslog logstash filter to process IIB system events.
Again, avoids supporting third party extensions in IIB.
But, as I said. If you already have infrastructure and perf testing for your log4j solution, then there's no reason I can think of not to use it.
Except, as mentioned, the performance of writing a gig of data to a file.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Thu Sep 10, 2015 7:42 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| mqjeff wrote: |
I might still look at IIB monitoring events and stats messages to gather the log data you need for IIB apps.
You can then use a logstash input to pull the messages from a queue, which again will happen asynchronously and could happen on an entirely separate machine. And use a standard syslog logstash filter to process IIB system events.
Again, avoids supporting third party extensions in IIB.
But, as I said. If you already have infrastructure and perf testing for your log4j solution, then there's no reason I can think of not to use it.
Except, as mentioned, the performance of writing a gig of data to a file. |
True, our log files are rolling, so they typically get purged locally after 100mb, but this data is piped and collected by logstash/splunk in real time where there is tens of Gigs of data.
Yeah we are also looking at IIB events/stats. Also splunk can encode log4j message patterns natively which is a very nice feature for us. Log4j also allows us to easily customize the log format to work natively with InfluxDB. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Sep 10, 2015 9:55 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
| sleepyjamie wrote: |
| Our requirement is logging close to a GB of log messages a day in production. |
If I had close to a Gb of log messages a day in production, I'd wonder why and I'd certainly look for something other than a plain text file to keep all that data in. Something with (for example) good search facilities. |
  
Personally, I think that places that rely on flat text files for this sort of stuff are living in the 1970's and not the 2010's.
To the person who logs the GB of data daily.
When was the last time you have to search through it all to find a problem?
How long did it take you?
Now think about what could be done to make this just so much easier?
Answers to your management...
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Fri Sep 11, 2015 4:12 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
Agreed, however I think you guys are missing the point I was trying to make for logging. The purpose of these log files is for the business and the DevOps team. They are not intended for debugging problems, per say. It allows these teams to gain real-time insight into what is happening.
In our case we have this data being fed to BI, Marketing, Finance in real-time. A lot of the logs are used for KPIs, Analytics, Predictive Analysis, Sensing, Pattern Recognition, etc...
So they are not your traditional developer logs where you do a simple text search to find problems. These are used to fuel visualization and analytics platforms. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Sep 11, 2015 4:27 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| sleepyjamie wrote: |
Agreed, however I think you guys are missing the point I was trying to make for logging. The purpose of these log files is for the business and the DevOps team. They are not intended for debugging problems, per say. It allows these teams to gain real-time insight into what is happening.
In our case we have this data being fed to BI, Marketing, Finance in real-time. A lot of the logs are used for KPIs, Analytics, Predictive Analysis, Sensing, Pattern Recognition, etc...
So they are not your traditional developer logs where you do a simple text search to find problems. These are used to fuel visualization and analytics platforms. |
Even more reason for the data to be fed to a queue and have a process offload it from there. That offloading process may have nothing to do with IIB...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sleepyjamie |
| Posted: Wed Sep 16, 2015 6:52 am Post subject: |
|
|
Centurion
Joined: 29 Apr 2015
Posts: 135
|
| That could be done, but setting up infrastructure to deal with queues is more complex. |
|
| Back to top |
|
|
|
|
