Piping stdout to Python callable

Discussion in 'Python' started by Edward Diener, Aug 17, 2004.

  1. From within a function in my own module I need to take the output from a
    Python module "A", which is sending data to stdout and which can not be
    changed and which I need to invoke as a top-level module, and pipe it into
    another function in my own module so that I can read it from stdin. Is there
    an easy way to do this ? The only way I can presently think to do this is
    through "system python A.py | python MyOwnModule.py", which seems a bit
    laborious having to invoke python.exe itself twice. Any other solution would
    be most welcome.
    Edward Diener, Aug 17, 2004
    #1
    1. Advertising

  2. Op 2004-08-17, Edward Diener schreef <>:
    > From within a function in my own module I need to take the output from a
    > Python module "A", which is sending data to stdout and which can not be
    > changed and which I need to invoke as a top-level module, and pipe it into
    > another function in my own module so that I can read it from stdin. Is there
    > an easy way to do this ? The only way I can presently think to do this is
    > through "system python A.py | python MyOwnModule.py", which seems a bit
    > laborious having to invoke python.exe itself twice. Any other solution would
    > be most welcome.


    What do you mean when you say you need to invoke it as a top-level
    module? Do you mean you can't import it at all or that importing it will
    startup the process of generating output immediatly?

    What bothers you with twice invoking the interpreter? In these days
    a program that is invoked multiples times will generally be only
    loaded once in memory.

    Are threads an acceptable alternative? Does your MyOwnModule.py needs
    to write to stdout?

    --
    Antoon Pardon
    Antoon Pardon, Aug 17, 2004
    #2
    1. Advertising

  3. Antoon Pardon wrote:
    > Op 2004-08-17, Edward Diener schreef <>:
    >> From within a function in my own module I need to take the output
    >> from a Python module "A", which is sending data to stdout and which
    >> can not be changed and which I need to invoke as a top-level module,
    >> and pipe it into another function in my own module so that I can
    >> read it from stdin. Is there an easy way to do this ? The only way I
    >> can presently think to do this is through "system python A.py |
    >> python MyOwnModule.py", which seems a bit laborious having to invoke
    >> python.exe itself twice. Any other solution would be most welcome.

    >
    > What do you mean when you say you need to invoke it as a top-level
    > module? Do you mean you can't import it at all or that importing it
    > will startup the process of generating output immediatly?


    I mean that it has a "if __name__ == '__main__' line and I need to trigger
    it by calling 'python A.py'.

    >
    > What bothers you with twice invoking the interpreter?


    Nothing practically. Just seems inelegant.

    > In these days
    > a program that is invoked multiples times will generally be only
    > loaded once in memory.


    Shared libraries may be loaded once in memory but python.exe itself gets
    reloaded each time.

    >
    > Are threads an acceptable alternative? Does your MyOwnModule.py needs
    > to write to stdout?


    Threads are acceptable. MyOwnModule.py can do anything, depending on
    parameters, but the idea is that a certain parameter tells it to read from
    stdin on the other end of the pipe.

    I have implemented this via "os.system("python A.py | python MyOwnModule.py
    parameters") and it works fine. I thought there might be a better, more
    elegant way but since the above works without any problem I will stick to it
    unless you can suggest anything better. BTW "A.py" can be any number of
    types of modules which must be invoked by python.exe and writes it's results
    to stdout. I am telling you this lest you suggest that I somehow import and
    parse A.py in order to call directly into it rather than having python.exe
    invoke it. While that might be possible, it would be a real programming
    PITA.
    Edward Diener, Aug 18, 2004
    #3
  4. Edward Diener

    Rich Krauter Guest

    Edward Diener wrote:
    > Antoon Pardon wrote:
    >
    >>Op 2004-08-17, Edward Diener schreef <>:
    >>
    >>>From within a function in my own module I need to take the output
    >>>from a Python module "A", which is sending data to stdout and which
    >>>can not be changed and which I need to invoke as a top-level module,
    >>>and pipe it into another function in my own module so that I can
    >>>read it from stdin. Is there an easy way to do this ? The only way I
    >>>can presently think to do this is through "system python A.py |
    >>>python MyOwnModule.py", which seems a bit laborious having to invoke
    >>>python.exe itself twice. Any other solution would be most welcome.

    >>
    >>What do you mean when you say you need to invoke it as a top-level
    >>module? Do you mean you can't import it at all or that importing it
    >>will startup the process of generating output immediatly?

    >
    >
    > I mean that it has a "if __name__ == '__main__' line and I need to trigger
    > it by calling 'python A.py'.
    >
    >
    >>What bothers you with twice invoking the interpreter?

    >
    >
    > Nothing practically. Just seems inelegant.
    >
    >
    >>In these days
    >>a program that is invoked multiples times will generally be only
    >>loaded once in memory.

    >
    >
    > Shared libraries may be loaded once in memory but python.exe itself gets
    > reloaded each time.
    >
    >
    >>Are threads an acceptable alternative? Does your MyOwnModule.py needs
    >>to write to stdout?

    >
    >
    > Threads are acceptable. MyOwnModule.py can do anything, depending on
    > parameters, but the idea is that a certain parameter tells it to read from
    > stdin on the other end of the pipe.
    >
    > I have implemented this via "os.system("python A.py | python MyOwnModule.py
    > parameters") and it works fine. I thought there might be a better, more
    > elegant way but since the above works without any problem I will stick to it
    > unless you can suggest anything better. BTW "A.py" can be any number of
    > types of modules which must be invoked by python.exe and writes it's results
    > to stdout. I am telling you this lest you suggest that I somehow import and
    > parse A.py in order to call directly into it rather than having python.exe
    > invoke it. While that might be possible, it would be a real programming
    > PITA.
    >
    >


    You could do something like this:

    $ cat A.py
    def test(arg):
    for i in range(1,5):
    print arg*i

    $ cat B.py
    import A
    import sys
    import StringIO

    sys.stdout = StringIO.StringIO()
    A.test(5)
    fileobj,sys.stdout = sys.stdout,sys.__stdout__

    fileobj.seek(0)
    for line in fileobj:
    print line.strip()


    Worth the effort to change your code? Probably not. Kinda neat, though.

    Rich
    Rich Krauter, Aug 18, 2004
    #4
  5. Edward Diener

    Rich Krauter Guest

    Rich Krauter wrote:

    > [silly post snipped]


    Oh, duh. You you said you need to invoke the code in the
    if __name__ == '__main__': part of the code. Sorry, I saw your post
    yesterday and didn't register that then. Obviously my suggestion doesn't
    help you.

    Well, unless A.py looks something like

    def main():
    ...
    ...

    if __name__ == '__main__:
    main()


    Rich
    Rich Krauter, Aug 18, 2004
    #5
  6. Op 2004-08-18, Edward Diener schreef <>:
    > Antoon Pardon wrote:
    >> Op 2004-08-17, Edward Diener schreef <>:
    >>> From within a function in my own module I need to take the output
    >>> from a Python module "A", which is sending data to stdout and which
    >>> can not be changed and which I need to invoke as a top-level module,
    >>> and pipe it into another function in my own module so that I can
    >>> read it from stdin. Is there an easy way to do this ? The only way I
    >>> can presently think to do this is through "system python A.py |
    >>> python MyOwnModule.py", which seems a bit laborious having to invoke
    >>> python.exe itself twice. Any other solution would be most welcome.

    >>
    >> What do you mean when you say you need to invoke it as a top-level
    >> module? Do you mean you can't import it at all or that importing it
    >> will startup the process of generating output immediatly?

    >
    > I mean that it has a "if __name__ == '__main__' line and I need to trigger
    > it by calling 'python A.py'.


    >>
    >> What bothers you with twice invoking the interpreter?

    >
    > Nothing practically. Just seems inelegant.
    >
    >> In these days
    >> a program that is invoked multiples times will generally be only
    >> loaded once in memory.

    >
    > Shared libraries may be loaded once in memory but python.exe itself gets
    > reloaded each time.


    That depends on the O.S. It is possible the O.S. notices that an
    invoked program is already loaded en uses the same code-segment
    in memory for following invocations.

    >> Are threads an acceptable alternative? Does your MyOwnModule.py needs
    >> to write to stdout?

    >
    > Threads are acceptable. MyOwnModule.py can do anything, depending on
    > parameters, but the idea is that a certain parameter tells it to read from
    > stdin on the other end of the pipe.


    Then I fear there is no other solution. As far as I understand, you
    can't redirect standard output on a per thread basis, (at least not
    in python.) That means that if you redirect stdout of the A.py module
    to go into a pipe, the stdout of MyOwnModule.py will go into the
    same pipe. I don't think you want that.

    --
    Antoon Pardon
    Antoon Pardon, Aug 18, 2004
    #6
  7. Antoon Pardon wrote:
    > Op 2004-08-18, Edward Diener schreef <>:
    >> Antoon Pardon wrote:
    >>> Op 2004-08-17, Edward Diener schreef <>:
    >>>> From within a function in my own module I need to take the output
    >>>> from a Python module "A", which is sending data to stdout and which
    >>>> can not be changed and which I need to invoke as a top-level
    >>>> module, and pipe it into another function in my own module so that
    >>>> I can read it from stdin. Is there an easy way to do this ? The
    >>>> only way I can presently think to do this is through "system
    >>>> python A.py | python MyOwnModule.py", which seems a bit laborious
    >>>> having to invoke python.exe itself twice. Any other solution would
    >>>> be most welcome.
    >>>
    >>> What do you mean when you say you need to invoke it as a top-level
    >>> module? Do you mean you can't import it at all or that importing it
    >>> will startup the process of generating output immediatly?

    >>
    >> I mean that it has a "if __name__ == '__main__' line and I need to
    >> trigger it by calling 'python A.py'.

    >
    >>>
    >>> What bothers you with twice invoking the interpreter?

    >>
    >> Nothing practically. Just seems inelegant.
    >>
    >>> In these days
    >>> a program that is invoked multiples times will generally be only
    >>> loaded once in memory.

    >>
    >> Shared libraries may be loaded once in memory but python.exe itself
    >> gets reloaded each time.

    >
    > That depends on the O.S. It is possible the O.S. notices that an
    > invoked program is already loaded en uses the same code-segment
    > in memory for following invocations.
    >
    >>> Are threads an acceptable alternative? Does your MyOwnModule.py
    >>> needs to write to stdout?

    >>
    >> Threads are acceptable. MyOwnModule.py can do anything, depending on
    >> parameters, but the idea is that a certain parameter tells it to
    >> read from stdin on the other end of the pipe.

    >
    > Then I fear there is no other solution. As far as I understand, you
    > can't redirect standard output on a per thread basis, (at least not
    > in python.) That means that if you redirect stdout of the A.py module
    > to go into a pipe, the stdout of MyOwnModule.py will go into the
    > same pipe. I don't think you want that.


    Just wanted to write that someone else pointed out to me os.popen("python
    A.py") and that did work well as an alternative to using os.system(etc.) .
    Nonetheless thanks for your help. I mentioned this so that you would know
    also know about it yourself if you hadn't encountered it.
    Edward Diener, Aug 19, 2004
    #7
  8. Op 2004-08-19, Edward Diener schreef <>:
    > Antoon Pardon wrote:
    >>
    >> Then I fear there is no other solution. As far as I understand, you
    >> can't redirect standard output on a per thread basis, (at least not
    >> in python.) That means that if you redirect stdout of the A.py module
    >> to go into a pipe, the stdout of MyOwnModule.py will go into the
    >> same pipe. I don't think you want that.

    >
    > Just wanted to write that someone else pointed out to me os.popen("python
    > A.py") and that did work well as an alternative to using os.system(etc.) .
    > Nonetheless thanks for your help. I mentioned this so that you would know
    > also know about it yourself if you hadn't encountered it.


    I know about popen, but that didn't seem to resolve your concern.
    Working with popen will result in launching a second interpreter.
    Since one of your mayor concerns about your own solution was
    having two interpreters running, I didn't thought popen was a
    viable solution.

    --
    Antoon Pardon
    Antoon Pardon, Aug 19, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Christian Long
    Replies:
    3
    Views:
    10,199
    Duncan Booth
    Jun 26, 2003
  2. James McGill
    Replies:
    1
    Views:
    296
    Fredrik Lundh
    Sep 8, 2008
  3. exiquio
    Replies:
    2
    Views:
    515
    exiquio
    Oct 7, 2008
  4. Adam Funk
    Replies:
    4
    Views:
    348
    Adam Funk
    Jan 6, 2012
  5. Ulrich Eckhardt
    Replies:
    6
    Views:
    90
    Peter Otten
    Jul 12, 2013
Loading...

Share This Page