Re: Python as a scripting language. Alternative to bash script?

Discussion in 'Python' started by Michael Torrie, Jun 28, 2010.

  1. On 06/28/2010 05:48 AM, Dave Pawson wrote:
    > Main queries are: Ease of calling out to bash to use something like
    > imageMagick or Java? Ease of grabbing return parameters? E.g. convert
    > can return both height and width of an image. Can this be returned to
    > the Python program? Can Python access the exit status of a program?


    Sure. I've created a module called runcmd that does 90% of what I want
    (easy access to stdout, stderr, error code). I've attached it to this
    e-mail. Feel free to use it; this post puts my code into the public domain.

    > I'd prefer the advantages of using Python, just wondering if I got so
    > far with the port then found it wouldn't do something?


    Python really isn't a shell scripting language. So there are things
    that Bash does much better, such as spawning processes and piping them
    together. I've tried over the years to create a pythonic library that
    would let me do that, but haven't found a good syntax that I like.

    It turns out, though, that much of what I use piping for in Bash is to
    run external processes to do things that I could use python modules for.
    For example, I typically pipe stuff to cut a lot to get certain fields.
    For example:

    ps ax | cut -c1-5

    In python I could simply take the output of "ps ax" and use python's
    own, superior, cutting routines (using my module):

    (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
    for x in stdout.split('\n'):
    print x.strip().split()[0]

    Sure it's a couple more lines in this case, but in other cases, python's
    abilities make it simpler than bash. A great document on how you can
    exploit python's abilities (particularly generators) to replace bash
    pipelines is here: http://www.dabeaz.com/generators/
     
    Michael Torrie, Jun 28, 2010
    #1
    1. Advertising

  2. Michael Torrie

    sturlamolden Guest

    On 28 Jun, 19:39, Michael Torrie <> wrote:

    > In python I could simply take the output of "ps ax" and use python's
    > own, superior, cutting routines (using my module):
    >
    > (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
    > for x in stdout.split('\n'):
    >     print x.strip().split()[0]


    Or you just pass the stdout of one command as stdin to another. That
    is equivalent of piping with bash.
     
    sturlamolden, Jul 6, 2010
    #2
    1. Advertising

  3. On 07/06/2010 04:12 AM, sturlamolden wrote:
    > On 28 Jun, 19:39, Michael Torrie <> wrote:
    >
    >> In python I could simply take the output of "ps ax" and use python's
    >> own, superior, cutting routines (using my module):
    >>
    >> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
    >> for x in stdout.split('\n'):
    >> print x.strip().split()[0]

    >
    > Or you just pass the stdout of one command as stdin to another. That
    > is equivalent of piping with bash.


    Consider this contrived example:

    tail -f /var/log/messages | grep openvpn

    While it's possible to set up pipes and spawn programs in parallel to
    operate on the pipes, in practice it's simpler to tell subprocess.Popen
    to use a shell and then just rely on Bash's very nice syntax for setting
    up the pipeline. Then just read the final output in python. If you set
    the stdout descriptor to non-blocking, you could read output as it came.
     
    Michael Torrie, Jul 6, 2010
    #3
  4. On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
    > On 07/06/2010 04:12 AM, sturlamolden wrote:
    >> On 28 Jun, 19:39, Michael Torrie <> wrote:
    >>
    >>> In python I could simply take the output of "ps ax" and use python's
    >>> own, superior, cutting routines (using my module):
    >>>
    >>> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
    >>> for x in stdout.split('\n'):
    >>>     print x.strip().split()[0]

    >>
    >> Or you just pass the stdout of one command as stdin to another. That
    >> is equivalent of piping with bash.

    >
    > Consider this contrived example:
    >
    > tail -f /var/log/messages | grep openvpn
    >
    > While it's possible to set up pipes and spawn programs in parallel to
    > operate on the pipes, in practice it's simpler to tell subprocess.Popen
    > to use a shell and then just rely on Bash's very nice syntax for setting
    > up the pipeline.  Then just read the final output in python.  If you set
    > the stdout descriptor to non-blocking, you could read output as it came.
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >


    Is this a discussion about the pipes module in the std library?
     
    member thudfoo, Jul 6, 2010
    #4
  5. Michael Torrie

    Chris Rebert Guest

    On Tue, Jul 6, 2010 at 1:35 PM, member thudfoo <> wrote:
    > On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
    >> On 07/06/2010 04:12 AM, sturlamolden wrote:
    >>> On 28 Jun, 19:39, Michael Torrie <> wrote:
    >>>> In python I could simply take the output of "ps ax" and use python's
    >>>> own, superior, cutting routines (using my module):
    >>>>
    >>>> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
    >>>> for x in stdout.split('\n'):
    >>>>     print x.strip().split()[0]
    >>>
    >>> Or you just pass the stdout of one command as stdin to another. That
    >>> is equivalent of piping with bash.

    >>
    >> Consider this contrived example:
    >>
    >> tail -f /var/log/messages | grep openvpn
    >>
    >> While it's possible to set up pipes and spawn programs in parallel to
    >> operate on the pipes, in practice it's simpler to tell subprocess.Popen
    >> to use a shell and then just rely on Bash's very nice syntax for setting
    >> up the pipeline.  Then just read the final output in python.  If you set
    >> the stdout descriptor to non-blocking, you could read output as it came.

    >
    > Is this a discussion about the pipes module in the std library?


    No, though that module is not irrelevant to Mr. Torrie's argument.

    Cheers,
    Chris
    --
    http://blog.rebertia.com
     
    Chris Rebert, Jul 7, 2010
    #5
  6. Michael Torrie

    Chris Rebert Guest

    On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
    > On 07/06/2010 04:12 AM, sturlamolden wrote:
    >> On 28 Jun, 19:39, Michael Torrie <> wrote:
    >>> In python I could simply take the output of "ps ax" and use python's
    >>> own, superior, cutting routines (using my module):
    >>>
    >>> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
    >>> for x in stdout.split('\n'):
    >>>     print x.strip().split()[0]

    >>
    >> Or you just pass the stdout of one command as stdin to another. That
    >> is equivalent of piping with bash.

    >
    > Consider this contrived example:
    >
    > tail -f /var/log/messages | grep openvpn
    >
    > While it's possible to set up pipes and spawn programs in parallel to
    > operate on the pipes, in practice it's simpler to tell subprocess.Popen
    > to use a shell and then just rely on Bash's very nice syntax for setting
    > up the pipeline.


    Until there's a Python variable involved that is, unless you want to
    overlook all the edge cases or do the escaping all by yourself (and
    then pray you did it right).

    Cheers,
    Chris
    --
    http://blog.rebertia.com
     
    Chris Rebert, Jul 7, 2010
    #6
  7. On 07/06/2010 09:34 PM, Chris Rebert wrote:
    > On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
    >> While it's possible to set up pipes and spawn programs in parallel to
    >> operate on the pipes, in practice it's simpler to tell subprocess.Popen
    >> to use a shell and then just rely on Bash's very nice syntax for setting
    >> up the pipeline.

    >
    > Until there's a Python variable involved that is, unless you want to
    > overlook all the edge cases or do the escaping all by yourself (and
    > then pray you did it right).


    Very good point. This is a problem that the pipes module suffers from
    as well.

    Although we learned in the other thread on escaping SQL statements that
    escaping is faster, easier and just as safe as other parameterization
    mechanisms. Uh huh.

    Back on target, a library similar to pipes that was safe (pipes is not)
    and had a pythonic syntax would be cool. pipes module works alright,
    syntax wise, but it's not a clean syntax.
     
    Michael Torrie, Jul 7, 2010
    #7
  8. Michael Torrie

    Chris Rebert Guest

    On Wed, Jul 7, 2010 at 8:31 AM, Michael Torrie <> wrote:
    > On 07/06/2010 09:34 PM, Chris Rebert wrote:
    >> On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
    >>> While it's possible to set up pipes and spawn programs in parallel to
    >>> operate on the pipes, in practice it's simpler to tell subprocess.Popen
    >>> to use a shell and then just rely on Bash's very nice syntax for setting
    >>> up the pipeline.

    >>
    >> Until there's a Python variable involved that is, unless you want to
    >> overlook all the edge cases or do the escaping all by yourself (and
    >> then pray you did it right).

    >
    > Very good point.  This is a problem that the pipes module suffers from
    > as well.
    >
    > Although we learned in the other thread on escaping SQL statements that
    > escaping is faster, easier and just as safe as other parameterization
    > mechanisms.  Uh huh.
    >
    > Back on target, a library similar to pipes that was safe (pipes is not)
    > and had a pythonic syntax would be cool.  pipes module works alright,
    > syntax wise, but it's not a clean syntax.


    Actually, your original post inspired me to take a crack at writing
    something like that yesterday:
    http://rebertia.com/code/subproc_pipelines.py

    Thoughts anyone? (Written on a whim, so no tests or docs at present.)

    Cheers,
    Chris
    --
    http://blog.rebertia.com
     
    Chris Rebert, Jul 7, 2010
    #8
  9. In message <>, Michael
    Torrie wrote:

    > While it's possible to set up pipes and spawn programs in parallel to
    > operate on the pipes, in practice it's simpler to tell subprocess.Popen
    > to use a shell and then just rely on Bash's very nice syntax for setting
    > up the pipeline.


    Just be careful about properly escaping any special characters in the file
    names. See my other thread about escaping data, where some respondents have
    expressed a strong allergy to any form of embedding one language inside
    another.
     
    Lawrence D'Oliveiro, Jul 24, 2010
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. William Park

    (patch for Bash) GTK+2 + Bash

    William Park, Jul 11, 2005, in forum: XML
    Replies:
    3
    Views:
    2,302
    William Park
    Jul 14, 2005
  2. Ron Stephens
    Replies:
    23
    Views:
    2,861
    Ron Stephens
    Apr 12, 2004
  3. DaveInSidney
    Replies:
    0
    Views:
    424
    DaveInSidney
    May 9, 2005
  4. Benjamin Kaplan
    Replies:
    13
    Views:
    1,289
    Stefan Behnel
    Jul 7, 2010
  5. Jorgen Grahn
    Replies:
    0
    Views:
    382
    Jorgen Grahn
    Jun 29, 2010
Loading...

Share This Page