Redirecting stderr and stdout to syslog

[Lincoln Yeoh]

Hi,

I've just started to learn python (I've been using perl for some years).

How do I redirect ALL stderr stuff to syslog, even stderr from
external programs that don't explicitly change their own stderr?

Say I have a program called foo:

#!/usr/bin/python
import syslog
import os, sys
class logstderr:
def write(self, data):
syslog.syslog('STDERR: %s' % data)
syslog.openlog('test[%u]' % os.getpid() )
sys.stderr=logstderr()
cmd='ls -al asdfsdf'
os.system(cmd)
bar('foo')

And bar is a nonexistent function.

If I run test I get:
../test
ls: cannot access asdfsdf: No such file or directory

And in /var/log/messages I get:
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: Traceback (most recent
call last):
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: File "./foo", line
11, in <module>
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR:
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: bar('foo')
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: NameError
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: :
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: name 'bar' is not defined
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR:


What I want is the "ls: cannot access asdfsdf: No such file or
directory" message to go to syslog instead:
e.g.
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: ls: cannot access
asdfsdf: No such file or directory
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: Traceback (most recent
call last):
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: File "./foo", line
11, in <module>
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR:
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: bar('foo')
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: NameError
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: :
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR: name 'bar' is not defined
Aug 5 21:08:35 linux-9k3z test[2186]: STDERR:

Explanation:

I do not normally redirect STDERR and STDOUT to /dev/null for daemons I write.

Since in _theory_ nothing should be "leaking" out, if stuff does leak
out in practice, something is not quite right.

So I want all such "leaks" be redirected to syslog (or my logging
routines), so that I can see the bugs/warnings - whether in my
program or other programs/modules I call/use.

Sorry if this has been dealt with before - I haven't found the
solution in my searches though.

I do NOT want to resort to this:

#!/bin/sh
/bin/foo 2>&1 | logger -t "test: STDERR/STDOUT"



Thanks,

Link.

 

[Matthew Woodcraft]

How do I redirect ALL stderr stuff to syslog, even stderr from
external programs that don't explicitly change their own stderr?

Sending messages to syslog involves more than writing to a file
descriptor, so there's no way to make this happen without having some
process read the external programs' output and send it to syslog (which
is basically the 'logger' method that you said you didn't like).

Since in _theory_ nothing should be "leaking" out, if stuff does leak
out in practice, something is not quite right.

So I want all such "leaks" be redirected to syslog (or my logging
routines), so that I can see the bugs/warnings - whether in my
program or other programs/modules I call/use.

One reasonable way to do this is to have a separate 'breakage log' file
for this kind of message, and point standard error there. This will
catch output from external programs, and any output to standard error
which libraries in your main program (or the Python interpreter) decide
to produce.

You'd do that with something like this (untested):

STDERR = 2
se = os.open("breakage.log", os.O_WRONLY|os.O_APPEND)
os.dup2(se, STDERR)
os.close(se)

You can also use this breakage log for errors from your own program, if
there are any cases where you think things might be too messed up for
you to want to rely on your normal logging routines.

Then since the breakage log should normally be empty, you can write a
cronjob or something to keep an eye on it and notify you if anything
appears there.

-M-

 

[Lincoln Yeoh]

Sending messages to syslog involves more than writing to a file
descriptor, so there's no way to make this happen without having some
process read the external programs' output and send it to syslog (which
is basically the 'logger' method that you said you didn't like).

I actually don't mind resorting to logger, I just don't want to wrap
the python program in a shell script.

Anyway the following appears to work, not sure if there are any
gotchas (I'm new to this).

(a,b)=popen2.popen2('/bin/logger -t "test[%u]: STDERR"' % os.getpid())
os.dup2(b.fileno(),sys.stderr.fileno())
(a,b1)=popen2.popen2('/bin/logger -t "test[%u]: STDOUT"' % os.getpid())
os.dup2(b1.fileno(),sys.stdout.fileno())

It seems subprocess is to be preferred over popen2. subprocess is
more verbose for this.

With perl I normally do (close your eyes if you are allergic to perl ):

open(STDERR,"|/bin/logger -t \"${PROGNAME}[$$]: STDERR\"") or die
"Error: Unable to redirect STDERR to logger!";
open(STDOUT,"|/bin/logger -t \"${PROGNAME}[$$]: STDOUT\"") or die
"Error: Unable to redirect STDOUT to logger!";

You'd do that with something like this (untested):

STDERR = 2
se = os.open("breakage.log", os.O_WRONLY|os.O_APPEND)
os.dup2(se, STDERR)
os.close(se)

You can also use this breakage log for errors from your own program, if
there are any cases where you think things might be too messed up for
you to want to rely on your normal logging routines.

I've seen that but rejected it because it will not provide
timestamps, and the logs end up in different places - some to syslog
and some to this file.

Having the logs go to the same place also gives a hint on the order
of events - it's not 100% in order of course, but usually good
enough, and much better than not even knowing what hour each line occurred.

Regards,
Link.