| Author |
Message
|
| deepak_paul |
 Posted: Fri Feb 12, 2010 9:31 am Post subject: File Lock with Log4j |
 |
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
All,
We are planning to use Log4j in the message flows which have multiple instances. We believe that there will be a lock on the log file while log4j is writing into log file. So other process(instance) has to wait for the first process to finish writing into the log file. This scenerio may reduce the performance of having multiple instances.
Is there any way to avoid this sequence execution of flows?
Thanks
Paul |
|
| Back to top |
|
 |
| zpat |
| Posted: Sat Feb 13, 2010 12:14 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Don't use log4j, it's not a standard WMB feature and shows signs of treating WMB as a Java application platform - which it is not.
Use event messages if your flow needs tracing. |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Tue Feb 16, 2010 7:27 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
But this is the recommended approach from our Technical Architects for logging.
Did anyone face this issue before and have some solution? |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Tue Feb 16, 2010 8:10 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
see by implementing log4j architecture it shouldn't stop your business flow.
and you should try to avoid writing a lot in the log files..secondly you should use it only for exceptional/error handling side.
Log4j experts can give much more idea on this...i can also do some tests to see this.  
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die
Last edited by Gaya3 on Tue Feb 16, 2010 8:23 am; edited 1 time in total |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Feb 16, 2010 8:19 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| deepak_paul wrote: |
But this is the recommended approach from our Technical Architects for logging.
Did anyone face this issue before and have some solution? |
May I humbly suggest that you ask your technical architects if they are experts in the way Message Broker works.
Their approach might be ok for apps totally written in Java but may not be appropriate for Broker based solutions.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Tue Feb 16, 2010 9:53 pm Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
i tested...Log4j framework will take care this issue of concurrency.
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Wed Feb 17, 2010 5:56 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
Thanks for your time, Gayathri.
Can you please elaborate how LOG4j framework is taking care of concurrent writing(No file lock?)?
Can you also let us know whether writing to the log file by the first process would make other jobs wait for completion(though milli seconds)?
Thanks
Paul |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Wed Feb 17, 2010 6:22 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
|
| Back to top |
|
|
| deepak_paul |
| Posted: Mon Feb 22, 2010 6:44 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
I see a statement as below:
"If performance is an issue at all, you want a single service writing the log file, rather than trying to coordinate a consistent locking strategy among diverse logging applications." - erickson
Can you tell me whether this single file logging idea will degrade the performance at some point?
As we are doing more 500 TPS, we really need to look at the performance.. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 22, 2010 7:03 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| deepak_paul wrote: |
As we are doing more 500 TPS, we really need to look at the performance.. |
This is what Databases are designed for. They'll handle the locks.
At the rate you have quoted, I'd also question who if ever would read the log files.
That is an awful lof of output that is generated per day. You can have too much data you know.
IMHO, logging like this should really only be done when there are problems.
Update counters in a DB for every message that is process successfully.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| anilit99 |
| Posted: Tue Feb 23, 2010 8:56 am Post subject: |
|
|
Voyager
Joined: 28 May 2009
Posts: 75
Location: London, UK
|
My two cents here :
Things to remember which were already said here :
1. MQ is not a java process.
2. Log4j doesnt have locking issues (unless for that special kind of appender). So you can use it in your application. The only downside would be because it doesnt lock the file the log messages will be garbled.
We once had the exact same requirements : the throughput is extremely high, and we had to log each and every thing happening there. Some findings are listed :
1. DO NOT USE DB to log your messages. It literally kills your app.
2. Be a bit cautious if you are using a file based logging. Make sure you are not writing over the network.
3. Delegate the entire logging to some place else i.e. make it asynchronous.
Now, we have different solutions to make it asynchronous but still maintain the order of logging. As somebody suggested, write into different logs and post process them into a single file (you can write a log4j appender for that).
You can also post all the logging events to a queue which can later be processed at your wish (again here, you can maintain the order of logs).
_________________
"I almost care !" |
|
| Back to top |
|
|
| JLRowe |
| Posted: Wed Feb 24, 2010 8:46 am Post subject: |
|
|
Yatiri
Joined: 25 May 2002
Posts: 664
Location: South East London
|
log4j is designed for high performance by minimising locks and by only logging messages that you have configured it for. Writes to the logging file will be buffered.
A lot of us want more finely grained and dynamic logging than the built in facilities provide, and using a java library such as log4j or sl4j is the only way to do this. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Wed Feb 24, 2010 8:49 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
think about some critical flows needs different log files
[eg. GayathriMsgFlow.log]
think about some critical Execution Groups needs different log files
[eg. GayathriEG.log]
think about some business functionality needs different log files.
[eg:GayathriFinance.log]
something like this.
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 24, 2010 5:25 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| JLRowe wrote: |
log4j is designed for high performance by minimising locks and by only logging messages that you have configured it for. Writes to the logging file will be buffered.
A lot of us want more finely grained and dynamic logging than the built in facilities provide, and using a java library such as log4j or sl4j is the only way to do this. |
You mean to say that the standard java.logging API will not support the high throughput he is planning? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| JLRowe |
| Posted: Thu Feb 25, 2010 3:34 am Post subject: |
|
|
Yatiri
Joined: 25 May 2002
Posts: 664
Location: South East London
|
| java.util.logging is also optimized for high throughput, but people prefer the better feature set of log4j. log4j has by far the largest mind share. |
|
| Back to top |
|
|
|
|
