Python, subprocess, dump, gzip and Cron

Discussion in 'Python' started by Aidan, Jun 10, 2008.

  1. Aidan

    Aidan Guest

    Hi,

    I'm having a bit of trouble with a python script I wrote, though I'm not
    sure if it's related directly to python, or one of the other software
    packages...

    The situation is that I'm trying to create a system backup script that
    creates an image of the system, filters the output though gzip, and then
    uploads the data (via ftp) to a remote site.

    The problem is that when I run the script from the command line, it
    works as I expect it, but when it is run by cron I only get a 20 byte
    file where the compressed image should be... does anyone have any idea
    as to why this might be happening? Code follows

    <code>

    #!/usr/bin/python

    from subprocess import PIPE, Popen
    from ftplib import FTP

    host = 'box'

    filename = '%s.img.gz' % host
    ftp_host = '192.168.1.250'
    ftpuser, ftppass = 'admin', 'admin'
    dest_dir = '/share/%s' % host

    dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)
    gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=PIPE)

    ftp = FTP(ftp_host)
    ftp.login(ftpuser,ftppass)
    ftp.cwd(dest_dir)
    ftp.storbinary('STOR %s' % filename,gzip.stdout)
    ftp.quit()

    print "Image '%s' created" % filename

    </code>

    I appreciate all feedback. Thanks in advance.
     
    Aidan, Jun 10, 2008
    #1
    1. Advertising

  2. Aidan

    TT Guest

    On Jun 10, 2:37 pm, Aidan <> wrote:
    > Hi,
    >
    > I'm having a bit of trouble with a python script I wrote, though I'm not
    > sure if it's related directly to python, or one of the other software
    > packages...
    >
    > The situation is that I'm trying to create a system backup script that
    > creates an image of the system, filters the output though gzip, and then
    > uploads the data (via ftp) to a remote site.
    >
    > The problem is that when I run the script from the command line, it
    > works as I expect it, but when it is run by cron I only get a 20 byte
    > file where the compressed image should be...  does anyone have any idea
    > as to why this might be happening?  Code follows
    >
    > <code>
    >
    > #!/usr/bin/python
    >
    > from subprocess import PIPE, Popen
    > from ftplib import FTP
    >
    > host = 'box'
    >
    > filename = '%s.img.gz' % host
    > ftp_host = '192.168.1.250'
    > ftpuser, ftppass = 'admin', 'admin'
    > dest_dir = '/share/%s' % host
    >
    > dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)
    > gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=PIPE)
    >
    > ftp = FTP(ftp_host)
    > ftp.login(ftpuser,ftppass)
    > ftp.cwd(dest_dir)
    > ftp.storbinary('STOR %s' % filename,gzip.stdout)
    > ftp.quit()
    >
    > print "Image '%s' created" % filename
    >
    > </code>
    >
    > I appreciate all feedback.  Thanks in advance.


    it's possible that the cron doesn't have the environment variables you
    have, especially $PATH. So the script failed to find the command it
    need to create the image.
     
    TT, Jun 10, 2008
    #2
    1. Advertising

  3. Aidan

    Aidan Guest

    TT wrote:
    > On Jun 10, 2:37 pm, Aidan <> wrote:
    >> Hi,
    >>
    >> I'm having a bit of trouble with a python script I wrote, though I'm not
    >> sure if it's related directly to python, or one of the other software
    >> packages...
    >>
    >> The situation is that I'm trying to create a system backup script that
    >> creates an image of the system, filters the output though gzip, and then
    >> uploads the data (via ftp) to a remote site.
    >>
    >> The problem is that when I run the script from the command line, it
    >> works as I expect it, but when it is run by cron I only get a 20 byte
    >> file where the compressed image should be... does anyone have any idea
    >> as to why this might be happening? Code follows
    >>
    >> <code>
    >>
    >> #!/usr/bin/python
    >>
    >> from subprocess import PIPE, Popen
    >> from ftplib import FTP
    >>
    >> host = 'box'
    >>
    >> filename = '%s.img.gz' % host
    >> ftp_host = '192.168.1.250'
    >> ftpuser, ftppass = 'admin', 'admin'
    >> dest_dir = '/share/%s' % host
    >>
    >> dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)
    >> gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=PIPE)
    >>
    >> ftp = FTP(ftp_host)
    >> ftp.login(ftpuser,ftppass)
    >> ftp.cwd(dest_dir)
    >> ftp.storbinary('STOR %s' % filename,gzip.stdout)
    >> ftp.quit()
    >>
    >> print "Image '%s' created" % filename
    >>
    >> </code>
    >>
    >> I appreciate all feedback. Thanks in advance.

    >
    > it's possible that the cron doesn't have the environment variables you
    > have, especially $PATH. So the script failed to find the command it
    > need to create the image.


    *fore head slap*

    Of course... adding the full path to both those utilities on the Popen
    lines seems to have fixed it.

    Thank you very much for your assistance.
     
    Aidan, Jun 10, 2008
    #3
  4. Aidan <> at Dienstag 10 Juni 2008 07:21:

    > TT wrote:
    >> On Jun 10, 2:37 pm, Aidan <> wrote:
    >>> Hi,
    >>>
    >>> I'm having a bit of trouble with a python script I wrote, though I'm not
    >>> sure if it's related directly to python, or one of the other software
    >>> packages...
    >>>
    >>> The situation is that I'm trying to create a system backup script that
    >>> creates an image of the system, filters the output though gzip, and then
    >>> uploads the data (via ftp) to a remote site.
    >>>
    >>> The problem is that when I run the script from the command line, it
    >>> works as I expect it, but when it is run by cron I only get a 20 byte
    >>> file where the compressed image should be... does anyone have any idea
    >>> as to why this might be happening? Code follows
    >>>
    >>> <code>
    >>>
    >>> #!/usr/bin/python
    >>>
    >>> from subprocess import PIPE, Popen
    >>> from ftplib import FTP
    >>>
    >>> host = 'box'
    >>>
    >>> filename = '%s.img.gz' % host
    >>> ftp_host = '192.168.1.250'
    >>> ftpuser, ftppass = 'admin', 'admin'
    >>> dest_dir = '/share/%s' % host
    >>>
    >>> dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)

    You should avoid the use of ``shell=True`` here and use a argument list
    instead:

    dump = Popen(['dump', '0uaf', '-', '/'], stdout=PIPE)

    This results in an exception thrown if the executable doesn't exist. This
    exception can be caught and handle for instance with the logging module.

    >>> gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=PIPE)


    Same here, but why don't you use the gzip functionality from the standard
    library?

    --
    Freedom is always the freedom of dissenters.
    (Rosa Luxemburg)
     
    Sebastian \lunar\ Wiesner, Jun 10, 2008
    #4
  5. Aidan

    Aidan Guest

    Sebastian "lunar" Wiesner wrote:
    > Aidan <> at Dienstag 10 Juni 2008 07:21:
    >
    >> TT wrote:
    >>> On Jun 10, 2:37 pm, Aidan <> wrote:
    >>>> Hi,
    >>>>
    >>>> I'm having a bit of trouble with a python script I wrote, though I'm not
    >>>> sure if it's related directly to python, or one of the other software
    >>>> packages...
    >>>>
    >>>> The situation is that I'm trying to create a system backup script that
    >>>> creates an image of the system, filters the output though gzip, and then
    >>>> uploads the data (via ftp) to a remote site.
    >>>>
    >>>> The problem is that when I run the script from the command line, it
    >>>> works as I expect it, but when it is run by cron I only get a 20 byte
    >>>> file where the compressed image should be... does anyone have any idea
    >>>> as to why this might be happening? Code follows
    >>>>
    >>>> <code>
    >>>>
    >>>> #!/usr/bin/python
    >>>>
    >>>> from subprocess import PIPE, Popen
    >>>> from ftplib import FTP
    >>>>
    >>>> host = 'box'
    >>>>
    >>>> filename = '%s.img.gz' % host
    >>>> ftp_host = '192.168.1.250'
    >>>> ftpuser, ftppass = 'admin', 'admin'
    >>>> dest_dir = '/share/%s' % host
    >>>>
    >>>> dump = Popen('dump 0uaf - /',shell=True,stdout=PIPE)

    > You should avoid the use of ``shell=True`` here and use a argument list
    > instead:
    >
    > dump = Popen(['dump', '0uaf', '-', '/'], stdout=PIPE)
    >
    > This results in an exception thrown if the executable doesn't exist. This
    > exception can be caught and handle for instance with the logging module.
    >


    thanks. That exception certainly would have helped me...

    >>>> gzip = Popen('gzip',shell=True,stdin=dump.stdout,stdout=PIPE)

    >
    > Same here, but why don't you use the gzip functionality from the standard
    > library?


    is there a way I can create a gzip file-like object which can read the
    output from the dump subprocess, and has a read method which outputs the
    compressed data, which will not write to disk first? With the above
    code python doesn't have to write the system image data to disk at all,
    which helps when there is not enough disk space to hold an intermediate
    image (at least, that is my understanding of it...).

    I had a look at the gzip module, but eventually just fell back to using
    the stdin and stdout of a gzip subprocess. I'd be interested to know
    how it could be done using the python standard lib though.
     
    Aidan, Jun 10, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. halfdog
    Replies:
    12
    Views:
    12,462
  2. Fazer

    Python and cron jobs.

    Fazer, Aug 4, 2003, in forum: Python
    Replies:
    8
    Views:
    851
    Christos TZOTZIOY Georgiou
    Aug 7, 2003
  3. Fredrik Lundh

    Re: Wits end with Python and cron

    Fredrik Lundh, Sep 28, 2005, in forum: Python
    Replies:
    0
    Views:
    496
    Fredrik Lundh
    Sep 28, 2005
  4. Iwan Vosloo

    Issue combining gzip and subprocess

    Iwan Vosloo, Jul 21, 2009, in forum: Python
    Replies:
    2
    Views:
    302
    Piet van Oostrum
    Jul 22, 2009
  5. hiral
    Replies:
    2
    Views:
    605
    Jean-Michel Pichavant
    May 5, 2010
Loading...

Share This Page