Running a Python script from crontab

[Astley Le Jasper]

I need help ... I've been looking at this every evening for over a
week now. I'd like to see my kids again!

I have script that runs fine in the terminal but when I try to run it
in a crontab for either myself or root, it bails out.

The trouble is that obviously I get no console when using crontab so
can't see any traceback. I use logging which shows 'info' messages as
the script is working, but no error messages. I've peppered it with
debug messages to try to track it down to a line, but it stops it the
middle of appending data to a list. I'd expect it to bail out when
calling a module or reading/writing to a different file.

Is there any way of getting more info from the app, like popping up a
console while its running?

my crontab is:

30 15 * * * cd /home/myusername/src && python myscript.py

ALJ

 

[James Mills]

Put your main function in a big
try, except. Catch any and all
errors and log them. Example:

def main():
try:
do_something()
except Exception, error:
log("ERROR: %s" % error)
log(format_exc())

Hope this helps.

cheers
James

 

[Philip Semanchuk]

I need help ... I've been looking at this every evening for over a
week now. I'd like to see my kids again!

I have script that runs fine in the terminal but when I try to run it
in a crontab for either myself or root, it bails out.

The trouble is that obviously I get no console when using crontab so
can't see any traceback. I use logging which shows 'info' messages as
the script is working, but no error messages. I've peppered it with
debug messages to try to track it down to a line, but it stops it the
middle of appending data to a list. I'd expect it to bail out when
calling a module or reading/writing to a different file.

Is there any way of getting more info from the app, like popping up a
console while its running?

my crontab is:

30 15 * * * cd /home/myusername/src && python myscript.py

Here's what my crontab looks like on FreeBSD. (Your syntax may
differ.) Note how I'm redirecting stdout and stderr to a log file --
that helps debugging. Also, I'm setting an explicit shell and path
just to make sure that e.g. Python is findable.


----------------------------
SHELL=/usr/local/bin/bash
PATH=/bin:/usr/bin:/usr/local/bin

# min hh day mon dow
*/2 * * * * python /usr/local/foo/myscript.py >> /
var/log/me/myscript.txt 2>&1

 

[Astley Le Jasper]

James ... thanks for the suggestion. I have done this and the error
logging usually catches all my errors and logs them. I wondered if
logging itself was failing!

Philip ... thanks also. I did wonder about making the everything
explicit. I've seen that mentioned elsewhere. Writing out the stdout &
stderr to another file is a great idea. I'll have ago. But I think our
dinner guests are starting to make comments so I'd better go!

 

[burb]

use UNIX "mail" command: crontab will send letters to you and you can
look at traceback there.

 

[gregory.j.baker]

Try using the following at the begining of your Python program:

import sys

sys.stdout = open("out.txt","w")
sys.stderr = open("err.txt","w")


Then whatever would normally go to stdout or stderr goes to text files
instead. You will see everything that happened.

 

[David]

I create a file runmyscript.sh and put it in /usr/bin

#!/bin/bash
cd /home/myusername.src
python /path/to/myscript

then chmod a+x /usr/bin/runmyscript.sh

test it
../runmyscript

add it to the crontab
30 15 * * * /usr/bin/runmyscript.sh

 

[Jon Redgrave]

....

Try using the "screen" utility - change the line in your crontab:
cd /home/myusername/src && python myscript.py
to
cd /home/myusername/src && screen -dmS mypthon python -i myscript.py

then once cron has started your program attach to the console using
(using python -i leaves you at the python prompt if the script just
terminates)

I email myself and drop into the pdb debugger on an exception, then
use screen to debug the system

 

[Astley Le Jasper]

Ok ... this is odd.

I tried gregory's suggestion of redirecting the stdout & stderr to a
text file. This worked. I could see all the logging information.
However, there was no error to see this time ... the application
worked completely without any problems.

I also then tried Jon's suggestion of attaching to the console ...
this also allowed me to see all the logging info as it went through.
Again, there were no problems as the application worked without any
problems.

Which suggests that it's the logging module or something with the
output streams.

Here is my logging setup in the primary module. It directs info to the
main console and also to a logging file.....
#Set up logging

log = logging.getLogger('scrape_manager')

log.setLevel(logging.INFO)



my_filename = ss.compact_date() + '_' + time.strftime("%H%M")

ensure_dir('logs/')
#Checks to see if the dir is there and creates it if not
log_file = 'logs/systemlog_' + my_filename + ".log"


console_h = logging.StreamHandler()

file_h = logging.FileHandler(log_file,'w')


log.addHandler(console_h)

log.addHandler(file_h)


console_fmt = logging.Formatter('%(levelname)s (%(threadName)-10s) %
(module)s %(message)s',)

file_fmt = logging.Formatter('%(asctime)s\t%(levelname)s\t(%
(threadName)-10s)\t%(module)s\t%(message)s',)


console_h.setFormatter(console_fmt)

file_h.setFormatter(file_fmt)

I also call this from an auxiliary module that is use my the primary
module:

log = logging.getLogger('scrape_manager')

If I change the name of the logger in the auxilary module then I loose
all the loggiing information but the application works.

What am I doing wrong with the logging setup? I can't see how I would
change it.

ALJ

 

[Lawrence D'Oliveiro]

In message

The trouble is that obviously I get no console when using crontab so
can't see any traceback.

Cron normally sends you mail if a cron task generated any output (this
should include error messages).

 

[Astley Le Jasper]

I've included a switch to include or exclude the logging to console.
When logging only to file, the script runs fine.

Of course, I still don't understand why dual logging, and specifically
to the console, causes a problem and if anyone has any comments about
the dual output logging code above then I'd still be happy to hear
about it.

ALJ

 

[Philip Semanchuk]

I've included a switch to include or exclude the logging to console.
When logging only to file, the script runs fine.

Of course, I still don't understand why dual logging, and specifically
to the console, causes a problem and if anyone has any comments about
the dual output logging code above then I'd still be happy to hear
about it.

Trying to write non-ASCII characters perchance?

 

[Astley Le Jasper]

Trying to write non-ASCII characters perchance?

Errmmm ... that's kind of spoookey. I using UTF-8 encoding as I have a
lot of European language characters. But would that cause a problem
when running from crontab but not in the terminal?

Go on then ... spill the beans.

 

[Philip Semanchuk]

Errmmm ... that's kind of spoookey. I using UTF-8 encoding as I have a
lot of European language characters. But would that cause a problem
when running from crontab but not in the terminal?

Go on then ... spill the beans.

Oh, I don't know exactly. It's just what I thought of when you said
that the problem occurs when logging to the console but not to files.
I don't have a deep Unix background so I can't give you the details on
what "the console" means to a cron job.

In my experience, the environment in which a cron job runs is
different from the environment in which some command line scripts run
(remember my earlier suggestion about needing to explicitly set the
PATH?) and so if the console for a cron job differed from the console
that a Python program sees when run in a terminal, that would not
surprise me.

Hope it's a useful suggestion
Philip

 

[Astley Le Jasper]

Oh, I don't know exactly. It's just what I thought of when you said  
that the problem occurs when logging to the console but not to files.  
I don't have a deep Unix background so I can't give you the details on  
what "the console" means to a cron job.

In my experience, the environment in which a cron job runs is  
different from the environment in which some command line scripts run  
(remember my earlier suggestion about needing to explicitly set the  
PATH?) and so if the console for a cron job differed from the console  
that a Python program sees when run in a terminal, that would not  
surprise me.

Hope it's a useful suggestion
Philip

I tried using the path variables but it didn't help. There again, I
didn't really understand what was being passed in the path variables
and, rather than just copying and pasting, I wanted to look up further
info to get to grips with it. I've got something that works, so now
the pressure is off I can spend some time looking it up.

[sigh ... my experiences with linux haven't been that good so far. It
is, if nothing else, interesting]

Thanks

ALJ

 

[Philip Semanchuk]

Oh, I don't know exactly. It's just what I thought of when you said
that the problem occurs when logging to the console but not to files.
I don't have a deep Unix background so I can't give you the details
on
what "the console" means to a cron job.

In my experience, the environment in which a cron job runs is
different from the environment in which some command line scripts run
(remember my earlier suggestion about needing to explicitly set the
PATH?) and so if the console for a cron job differed from the console
that a Python program sees when run in a terminal, that would not
surprise me.

Hope it's a useful suggestion
Philip

I tried using the path variables but it didn't help. There again, I
didn't really understand what was being passed in the path variables
and, rather than just copying and pasting, I wanted to look up further
info to get to grips with it. I've got something that works, so now
the pressure is off I can spend some time looking it up.

[sigh ... my experiences with linux haven't been that good so far. It
is, if nothing else, interesting]

Sorry, I was misleading. I don't think the PATH has anything to do
with why your script can't log to the console when run as a cron job.
I was just pointing out that my Python scripts have had environment-
related problems when run as cron jobs, so it would not surprise me if
that's your problem too. The PATH was just an example of an
environment element that wasn't what I expected.

Just a thought -- if my weak hunch that non-ASCII output is your
problem, you might be able to test this easily (depending on how you
use the logger) by forcing all output messages to "foo". If your cron
job doesn't crash then, you know that the *content* of the output is
probably the culprit which would indeed point to an encoding problem.

HTH
Philip

 

[Lawrence D'Oliveiro]

In my experience, the environment in which a cron job runs is
different from the environment in which some command line scripts run...

Which is true, but again, cron should report the environment in the mail
message. For example, here are some headers from a recent run of the
maildir backup task I have scheduled twice a day:

Subject: Cron <ldo@theon> $HOME/bin/backupdir $HOME/.maildir
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/ldo>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=ldo>
X-Cron-Env: <USER=ldo>

 

[Astley Le Jasper]

Which is true, but again, cron should report the environment in the mail
message. For example, here are some headers from a recent run of the
maildir backup task I have scheduled twice a day:

    Subject: Cron <ldo@theon> $HOME/bin/backupdir $HOME/.maildir
    X-Cron-Env: <SHELL=/bin/sh>
    X-Cron-Env: <HOME=/home/ldo>
    X-Cron-Env: <PATH=/usr/bin:/bin>
    X-Cron-Env: <LOGNAME=ldo>
    X-Cron-Env: <USER=ldo>

Where do you get the emails from?

 

[Lawrence D'Oliveiro]

In message

Where do you get the emails from?

From my local mailbox on the machine.

 

[Philip Semanchuk]

Where do you get the emails from?

In my experience, this depends on the machine config. The machine
*should* be set up to email the user when a cron job fails. You'll log
in via terminal and get the message "You have new mail" which means
something went wrong with your cron job. As Lawrence said, run mail in
the terminal window and you'll have a message from cron. You might not
be getting these mails for some reason.