log4j and log file lock?

[alejandrina]

Hi all,

We have a processor farm (currently 8 machines) running the same
application with different input parameters. We use log4j to record
interesting stuff; we'd like to use a single log file in a networked
disk. We noticed that the 8 machines are trampling on each other as
they write to the log file (symptoms are nulls embedded in messages,
incomplete messages, duplicate messages....). I have assumed that this
is because access to the log file is not "locked" (using something
like Java's FileLock API).

Is there a solution to this problem? Thread syncronization is not an
option, of course, as these are different computers.

Thanks for any ideas,

Alejandrina

 

[Owen Jacobson]

Hi all,

We have a processor farm (currently 8 machines) running the same
application with different input parameters. We use log4j to record
interesting stuff; we'd like to use a single log file in a networked
disk. We noticed that the 8 machines are trampling on each other as
they write to the log file (symptoms are nulls embedded in messages,
incomplete messages, duplicate messages....). I have assumed that this
is because access to the log file is not "locked" (using something
like Java's FileLock API).

Is there a solution to this problem? Thread syncronization is not an
option, of course, as these are different computers.

Thanks for any ideas,

Alejandrina

Use the syslog appender and let a single network-available syslog
instance handle the actual log files.

-Owen

 

[Manish Pandit]

Hi all,

We have a processor farm (currently 8 machines) running the same
application with different input parameters. We use log4j to record
interesting stuff; we'd like to use a single log file in a networked
disk. We noticed that the 8 machines are trampling on each other as
they write to the log file (symptoms are nulls embedded in messages,
incomplete messages, duplicate messages....). I have assumed that this
is because access to the log file is not "locked" (using something
like Java's FileLock API).

Is there a solution to this problem? Thread syncronization is not an
option, of course, as these are different computers.

Thanks for any ideas,

Alejandrina

You can also look at the SocketAppender/SimpleSocketServer (It is in
the same package as the Syslog appender mentioned by Owen.)

http://logging.apache.org/log4j/docs/api/org/apache/log4j/net/package-summary.html

I've seen SocketAppender+SimpleSocketServer being used at several
instances which are similar to your setup. One thing to be careful
about is the message rate; which, if faster than the network
(connection to the server), will cause the client to block.

-cheers,
Manish

 

[Lew]

You can also look at the SocketAppender/SimpleSocketServer (It is in
the same package as the Syslog appender mentioned by Owen.)

http://logging.apache.org/log4j/docs/api/org/apache/log4j/net/package-summary.html

I've seen SocketAppender+SimpleSocketServer being used at several
instances which are similar to your setup. One thing to be careful
about is the message rate; which, if faster than the network
(connection to the server), will cause the client to block.

If you are at a higher log level, say, WARN, is it safe to say that
lower-level (DEBUG, INFO) calls will not risk blocking, nor otherwise have any
effect on the socket? I should think so.

Regardless of destination or concurrency issues, the trick of what to log, at
what level, is both important and fairly devoid of best practices lore.

Few sources even acknowledge the importance of logging, though nearly every
professional programmer practices it. Fewer lay out reasonable strategies, or
even discuss the importance of having a logging strategy. It's arguably more
important to have a rigorous logging discipline at all than to worry about
which one is best.

In normal production scenarios there should be relatively few log messages, so
stress on the socket should be low. Canonically, production runs at WARN or
ERROR, and there should be nearly none of those. Even INFO messages should be
sparse - verbosity is for DEBUG. If you've cranked up your log level to
DEBUG, it's because you've got bigger concerns than a little blocking.

 

[alejandrina]

You can also look at the SocketAppender/SimpleSocketServer (It is in
the same package as the Syslog appender mentioned by Owen.)

http://logging.apache.org/log4j/docs/api/org/apache/log4j/net/package...

I've seen SocketAppender+SimpleSocketServer being used at several
instances which are similar to your setup. One thing to be careful
about is the message rate; which, if faster than the network
(connection to the server), will cause the client to block.

-cheers,
Manish

Yes, I think the SocketAppender is the way to go! Thanks for the tip.
And I agree 100% with Lew that the messages should be sparse. If you
have to debug, then turn off the socket and go back to the console or
the file.

 

[alejandrina]

You can also look at the SocketAppender/SimpleSocketServer (It is in
the same package as the Syslog appender mentioned by Owen.)

http://logging.apache.org/log4j/docs/api/org/apache/log4j/net/package...

I've seen SocketAppender+SimpleSocketServer being used at several
instances which are similar to your setup. One thing to be careful
about is the message rate; which, if faster than the network
(connection to the server), will cause the client to block.

-cheers,
Manish

Thanks Manish, I was able to set up the socket logging very easily and
works as advertised.

Alejandrina