| Author |
Message
|
| smdavies99 |
 Posted: Fri Mar 29, 2013 7:26 am Post subject: |
 |
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
My 2p worth is that for high volumes, the last thing you really want to do is slow down the processing of the message/file/whatever because of latencies in the logging/audit subsystem.
This is why I prefer to off load the logging to a separate flow via a queue. Then at times of high volume the log messages can build up on the queue with much less effect on the overall transaction throughput.
simply put, the logging is done asynchronously to the main job of the flows which is to process the message in as optimal manner as possible.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Mar 29, 2013 7:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nanotech wrote: |
| @Vitor and @fjb_saper: Do you think if we move over to java.util.logging , this will fix the issue. Currently we are using sl4j wrapper with log4j |
I'm told by Java people that java.logging is "better" than log4j and I'll believe them. Will it fix this issue? Maybe, maybe not.
My personal view is if you're produceing enough output via log4j to cause this problem, log4j is not what you should be using to produce this output. In one of the cases where I see it, the 2 complaints of the team in question is that the file doesn't roll over & when they're trying to find details of a particular transaction it's very cumbersome to search the log4j text for the details. I pointed out there are these things called "databases" where you can store lots of data & search for it easily; they gave me a pitying look and pointed out the business requirement was for an audit log.
I also agree with @smdavies99; if you're offloadeding a lot of data (no matter where you're putting it) then using a queue & flow solution as he describes is often a good idea.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| nanotech |
| Posted: Fri Mar 29, 2013 8:20 am Post subject: |
|
|
Novice
Joined: 29 May 2009
Posts: 15
|
We have on average 3 JCNs per Message flow and we log just minimal information about the message going out(just before propagating to out terminal) from each JCN. The logging level is set to INFO, and is just one line per message. So its more to determine how message is flowing through the system. We do not log the whole message / message body in the logs...we do that only if Exception occurs.
We saw no impact on message flow performance after we put log4j in .
But this issue of file handle staying open even after file rolls over is annoying everyone( especially, people in operations ), who have to monitor the files for ERRORs and Exceptions.
We can consider writing to MQ and then to a file asynchronously if this problem does not get resolved.
But I wonder if IBM would be able to help us in resolving this( or at least to find out why EG classloader is not releasing the file handle ) |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Mar 29, 2013 8:22 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
[quote="Vitor"]
| nanotech wrote: |
| I pointed out there are these things called "databases" where you can store lots of data & search for it easily; they gave me a pitying look and pointed out the business requirement was for an audit log. |
My current project logs 450,000+ entries a day. Now try searching for that obscure once a week problem if you are logging to flat files!
As Vitor says, there are these things called Databases. Makes linking the inputs and outputs very easy (with proper design).
If someone needs the logs as a flat file, that isn't a problem. Select, save as csv etc.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 29, 2013 9:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nanotech wrote: |
| But I wonder if IBM would be able to help us in resolving this( or at least to find out why EG classloader is not releasing the file handle ) |
Only IBM can answer that.
My take is that IAM3 (the support pac IBM supplies for log4j in WMB) is a Cat 2 (i.e. unsupported) pac. If you can demonstrate it's the EG classloader that's at fault then clearly you'll get support on that but I've always believed this behaviour is the log4j code rather than the infrastructure.
Regular readers will of course know how much credibility to give any opinion I state on Java matters. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 29, 2013 9:12 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nanotech wrote: |
| So its more to determine how message is flowing through the system. |
Don't forget you can do that with the inbuilt WMB monitoring facilities.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| nanotech |
| Posted: Fri Mar 29, 2013 9:13 am Post subject: |
|
|
Novice
Joined: 29 May 2009
Posts: 15
|
Yes we could use a database , but its more of an organizational question whether or not they want to invest in Oracle license for this for each environment (QA / UAT / Prod/ Performance Test ) . Plus then they will have to think about giving access to people for these environments. Flat files (such as produced by log4j ) are monitored by automated agents and e-mail is sent out when certain keywords are found in those files.
All I am saying is that its not my decision whether or not logging should be done to DB.
However I can pitch for asynchronous logging and log to MQ and then to file (if possible and makes sense )..
@Vitor: We are not using the IBM support pac, we are just using log4j + slf4j jars |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 29, 2013 9:26 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nanotech wrote: |
| Yes we could use a database , but its more of an organizational question whether or not they want to invest in Oracle license for this for each environment (QA / UAT / Prod/ Performance Test ) |
There are other, cheaper, db platforms than Oracle that can be used for this kind of non-business critical data.
| nanotech wrote: |
| Plus then they will have to think about giving access to people for these environments. |
Most such platforms come with such controls.
| nanotech wrote: |
| Flat files (such as produced by log4j ) are monitored by automated agents and e-mail is sent out when certain keywords are found in those files. |
Such automated agents can also query databases. We use such a tool here.
| nanotech wrote: |
| All I am saying is that its not my decision whether or not logging should be done to DB. |
Quite, but you can express informed technical opinion to the decision makers.
| nanotech wrote: |
| @Vitor: We are not using the IBM support pac, we are just using log4j + slf4j jars |
My point was that IBM don't support the log4j solution they provide. It's my assertion that they will be as wary of your solution, as they are for all user code related support.
But I'm not (currently) affiliated with IBM and it certainly won't hurt to ask them.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| goffinf |
| Posted: Sat Mar 30, 2013 3:01 am Post subject: |
|
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
| nanotech wrote: |
We have on average 3 JCNs per Message flow and we log just minimal information about the message going out(just before propagating to out terminal) from each JCN. The logging level is set to INFO, and is just one line per message. So its more to determine how message is flowing through the system. We do not log the whole message / message body in the logs...we do that only if Exception occurs.
We saw no impact on message flow performance after we put log4j in .
But this issue of file handle staying open even after file rolls over is annoying everyone( especially, people in operations ), who have to monitor the files for ERRORs and Exceptions.
We can consider writing to MQ and then to a file asynchronously if this problem does not get resolved.
But I wonder if IBM would be able to help us in resolving this( or at least to find out why EG classloader is not releasing the file handle ) |
I suppose you could re-design that slighty to only output one message to Log4j which contains all of the individual parts that you want to record. In your example that would reduce the number by by 66% and make correlation easier (possibly even redundant).
We emit a single message per flow to provide a basic elapsed time metric and we also use[d] Log4j. The appender writes asynchronously to the actual file and the overhead of the handoff to Log4j is certainly no more than it would be to MQ and probably significantly less. That said, we have switched to using MQ more recently mostly for the reasons raised here.
We started out using monitoring events emitted at various points thru a flow and then used a separate subscriber flow to correlate them thru a Collector. But frankly the Collector node wasn't up to it, so we switched back to something simpler.
Sometimes we use an appender configured to forward messages via UDP, partly because using the file system has both the problems you mention and because it can be a 'file system full' problem waiting to happen if your housekeeping isn't sufficiently robust (I certainly don't want my companies core business applications to be unavailable just because our solution for counting how many transactions we could have done if only it didn't crash ... has crashed).
Of course some log entries are more critical than others and in and of themselves are important business audits than need to be treated as such (sometimes for legal or regulatory reasons), YMMV.
As far as retrospective analysis or even 'near real time' alerting of significant business or technical events, there are lots of possible solution designs. Using a database 'mart' is favoured by lots of shops, and underpin most, even if not explicitly. Clearly searching thru large logs and trying to correlate messages in a high concurrency environment is near impossible at least from the perspective of 'timeliness'.
Here we make use of a product called Splunk and find it to be excellent in making sense even out of relatively unstructured data from files and many other sources. Of course we try and implement standards around structured content, and particularly correlation, but it's not always possible for every application you have hanging off your ESB. Please note I'm NOT an advocate for this product and dont't have any involvement with the vendor. It isn't free and does require it's own infrastructure. That said our experience thus far has been very good and so I would go so far as to recommend it as something that can be used for creating dashboards for many types of audience (business and technical) as well as supporting near real time alerting and 'ad hoc' investigation.
HTHs
Fraser. |
|
| Back to top |
|
|
|
|
