Executing python script stored as a string

Discussion in 'Python' started by Ecir Hana, Sep 1, 2009.

  1. Ecir Hana

    Ecir Hana Guest

    Hello,

    please, how to execute a python script stored as a string? But let me
    impose several limitations, so simple "exec" wont work:

    - if I understood it correctly defining a function in the string and
    exec-ing it created the function in current scope. This is something I
    really don't want

    - simple exec also blocks the rest of the program

    - I also would like the string to be able to use and return some parts
    of the caller

    So to give an example what I try to achieve:

    result = []
    def up(s):
    result.append(s.upper())

    code = '''
    up("abc")
    print 'hello'
    i = i + 3
    def x(s):
    up(s)
    x('def')
    print i
    '''

    somehow_execute(code)

    Couple of points:

    - the script in string should behave just like any other ordinary
    python script executed in separate process, except it should also know
    about a function caller "up". Nothing else. (I read that something
    similar is possible while embedding python into your C project - that
    you could invoke the VM and provide some default "imports")

    - if the other script runs in separate process how should it call the
    remote function? And how to pass its arguments? I really hope I don't
    have to serialize every communication, maybe I should use threading
    instead of process? All I want is that running it wont block the
    caller and that it cannot modify callers code/variables/scope (apart
    from calling the predefined callers' functions). Or maybe even better,
    let it block the caller but provide a way to stop its execution?

    - how to know that the script finished? I was thinking about atexit()
    - could it work here?

    Think of it as a text editor with a special ability to execute its
    content, while providing access of some of its functionality to the
    script.

    The reason I *think* I cannot just simple import the "editor" module
    into the script is that the"editor" is GUI application and script
    should have access to just this instance of editor.

    Anyway, I hope I was not too confusing. Thanks for any help!
     
    Ecir Hana, Sep 1, 2009
    #1
    1. Advertising

  2. On Mon, 31 Aug 2009 16:29:45 -0700, Ecir Hana wrote:

    > Hello,
    >
    > please, how to execute a python script stored as a string? But let me
    > impose several limitations, so simple "exec" wont work:
    >
    > - if I understood it correctly defining a function in the string and
    > exec-ing it created the function in current scope. This is something I
    > really don't want



    You can pass in a global and local namespaces to exec as arguments:

    >>> x = 4
    >>> ns = {'x': 4}
    >>> exec "x += 1" in ns
    >>> x

    4
    >>> ns['x']

    5


    See the docs for details.


    > - simple exec also blocks the rest of the program



    Run it in a thread.


    > - I also would like the string to be able to use and return some parts
    > of the caller


    You can copy the parts of the current scope into the namespace you pass
    to exec, then later copy the revised values out again.

    But are you sure you really want to take this approach? exec is up to ten
    times slower than just executing the code directly. And if the string is
    coming from an untrusted source, it is a *huge* security risk.



    > Couple of points:
    >
    > - the script in string should behave just like any other ordinary python
    > script executed in separate process, except it should also know about a
    > function caller "up". Nothing else. (I read that something similar is
    > possible while embedding python into your C project - that you could
    > invoke the VM and provide some default "imports")


    If you want it to execute in a separate *process*, that's a whole
    different question. If you do that, you get separation of code for free,
    as well as separate namespaces. My approach would be to have a special
    module "common" which subprocesses can import, to get access to the
    shared functions. You will probably need to create some sort of message
    passing infrastructure to get results out of the subprocess into the
    parent process.


    > - if the other script runs in separate process how should it call the
    > remote function? And how to pass its arguments? I really hope I don't
    > have to serialize every communication, maybe I should use threading
    > instead of process?


    If you want separate processes, they're *separate*. Threads are not.


    > All I want is that running it wont block the caller
    > and that it cannot modify callers code/variables/scope (apart from
    > calling the predefined callers' functions). Or maybe even better, let it
    > block the caller but provide a way to stop its execution?


    As far as I know, you can't kill threads, you can only ask them to kill
    themselves.


    > - how to know that the script finished? I was thinking about atexit() -
    > could it work here?


    I doubt it. You would need to poll each thread to see if it has completed.



    > Think of it as a text editor with a special ability to execute its
    > content, while providing access of some of its functionality to the
    > script.


    Something like this?

    In the text editor, you have contents:

    text goes here
    and more text
    # Python script starts here
    x = 'a'
    up(x)
    print "foo"
    # Python script stops here
    more text again


    and the user selects lines 4 and 5 and chooses the command "Execute". The
    script executes, and its output (foo) is appended to the end of the file:

    text goes here
    and more text
    # Python script starts here
    x = 'a'
    up(x)
    print "foo"
    # Python script stops here
    more text again
    foo

    Is this what you mean?

    If so, I think you are making this much too complicated for such a simple
    use-case. Just publish an API which the script can use, and have the main
    text editor application specify a "script" namespace containing only that
    API. That could be a module:

    >>> import math # pretend this is your API shared module
    >>> exec "myvalue = 42" in math.__dict__
    >>> math.myvalue

    42



    Then execute the text using exec, but don't bother about putting it into
    a thread or subprocess. That just makes it harder to implement, and you
    have to worry about concurrency issues.



    --
    Steven
     
    Steven D'Aprano, Sep 1, 2009
    #2
    1. Advertising

  3. Ecir Hana

    Ecir Hana Guest

    On Sep 1, 5:31 am, Steven D'Aprano
    <> wrote:
    >
    > You can pass in a global and local namespaces to exec as arguments:
    >
    > >>> x = 4
    > >>> ns = {'x': 4}
    > >>> exec "x += 1" in ns
    > >>> x

    > 4
    > >>> ns['x']

    >
    > 5
    >
    > See the docs for details.


    Thanks! This is very useful!

    > You can copy the parts of the current scope into the namespace you pass
    > to exec, then later copy the revised values out again.
    >
    > But are you sure you really want to take this approach? exec is up to ten
    > times slower than just executing the code directly. And if the string is
    > coming from an untrusted source, it is a *huge* security risk.


    I don't know if I should use exec. I don't really mind that it's slow
    (btw., why is it so?). But I don't quite understand why is it security
    risk. How is it different to run:
    exec 'format(your_hdd)'
    than:
    /bin/python format.py
    ?

    > As far as I know, you can't kill threads, you can only ask them to kill
    > themselves.


    Also, I'm not sure if I follow. What does this mean? If a thread runs:

    while True:
    pass

    it is not possible to kill it from another thread? (Bacause it doesn't
    check whether some other thread asks to stop it..?)

    > Something like this?


    Well, something more like:

    data = [1, 2, 3]
    map(lambda x: x * 2, data)
    display_data_in_editor_viewport(data) #this renders into part of main
    editor window (may take some time)

    > If so, I think you are making this much too complicated for such a simple
    > use-case. Just publish an API which the script can use, and have the main
    > text editor application specify a "script" namespace containing only that
    > API. That could be a module:
    >
    > >>> import math  # pretend this is your API shared module
    > >>> exec "myvalue = 42" in math.__dict__
    > >>> math.myvalue

    >
    > 42
    >
    > Then execute the text using exec, but don't bother about putting it into
    > a thread or subprocess. That just makes it harder to implement, and you
    > have to worry about concurrency issues.


    Ok, I could try exec, thanks for the explanation. But what about those
    security concerns you mentioned above?

    Thanks a lot, very informative!
     
    Ecir Hana, Sep 1, 2009
    #3
  4. Ecir Hana

    Ecir Hana Guest

    On Sep 1, 5:31 am, Steven D'Aprano
    <> wrote:

    > But are you sure you really want to take this approach? exec is up to ten
    > times slower than just executing the code directly.


    Oh, you mean because of parsing and compiling? But otherwise it's as
    fast as regular python? That's perfectly ok.

    Or maybe I misunderstood you - what do you mean by "executing the code
    directly"?
     
    Ecir Hana, Sep 1, 2009
    #4
  5. On Tue, 01 Sep 2009 01:34:33 -0700, Ecir Hana wrote:

    >> You can copy the parts of the current scope into the namespace you pass
    >> to exec, then later copy the revised values out again.
    >>
    >> But are you sure you really want to take this approach? exec is up to
    >> ten times slower than just executing the code directly. And if the
    >> string is coming from an untrusted source, it is a *huge* security
    >> risk.

    >
    > I don't know if I should use exec. I don't really mind that it's slow
    > (btw., why is it so?).


    Because it has to parse and compile the string into a code object before
    it can run it.


    > But I don't quite understand why is it security
    > risk. How is it different to run:
    > exec 'format(your_hdd)'
    > than:
    > /bin/python format.py
    > ?


    It's not different. But read what I said -- "if the string is coming from
    an UNTRUSTED source" -- presumably you trust yourself. If you run 'exec
    "format(your_hdd)"' it is because *you* want to format your hard disk.

    Now imagine you have a web-app which gets a string from the user and
    calls exec on it. Then you might have this:

    exec "search('%d')" % user_input

    and the user, who is halfway across the world, enters the following
    search string:

    places to eat'); import os; os.system('#rm -rf /

    Your web app will go right ahead and erase itself. That's why you need to
    keep untrusted strings away from exec, execfile, and eval.



    >> As far as I know, you can't kill threads, you can only ask them to kill
    >> themselves.

    >
    > Also, I'm not sure if I follow. What does this mean? If a thread runs:
    >
    > while True:
    > pass
    >
    > it is not possible to kill it from another thread? (Bacause it doesn't
    > check whether some other thread asks to stop it..?)


    No, I believe that the only way to halt that is to halt the entire
    process.

    Possibly there is a way to have a thread halt itself after a certain
    amount of time? I'm not an expert on threads, I've hardly ever used them.



    --
    Steven
     
    Steven D'Aprano, Sep 1, 2009
    #5
  6. Ecir Hana

    Ecir Hana Guest

    On Sep 1, 11:32 am, Steven D'Aprano
    <> wrote:
    > > But I don't quite understand why is it security
    > > risk. How is it different to run:
    > > exec 'format(your_hdd)'
    > > than:
    > > /bin/python format.py
    > > ?

    >
    > It's not different. But read what I said -- "if the string is coming from
    > an UNTRUSTED source" -- presumably you trust yourself. If you run 'exec
    > "format(your_hdd)"' it is because *you* want to format your hard disk.
    >
    > Now imagine you have a web-app which gets a string from the user and
    > calls exec on it. Then you might have this:
    >
    > exec "search('%d')" % user_input
    >
    > and the user, who is halfway across the world, enters the following
    > search string:
    >
    > places to eat'); import os; os.system('#rm -rf /
    >
    > Your web app will go right ahead and erase itself. That's why you need to
    > keep untrusted strings away from exec, execfile, and eval.


    Ah, I see! Ok.

    > No, I believe that the only way to halt that is to halt the entire
    > process.
    >
    > Possibly there is a way to have a thread halt itself after a certain
    > amount of time? I'm not an expert on threads, I've hardly ever used them.


    Thank you once again!
     
    Ecir Hana, Sep 1, 2009
    #6
  7. On Tuesday 01 September 2009 11:32:29 Steven D'Aprano wrote:

    > Possibly there is a way to have a thread halt itself after a certain
    > amount of time? I'm not an expert on threads, I've hardly ever used them.


    Not automagically, as far as I can see.
    You are on your own if you want to somehow kill a thread.

    What I do is to wrap what I want to do in a while clause:

    while runbool:
    do_what_you_want()

    where runbool is a global that I set to false elsewhere when I want to stop.

    There is of course nothing to stop you writing a thread with something like:

    import time
    start_time = time.time()
    while time.time() - start_time < some_parameter:
    do_what_you_want()

    Which will have the effect of running for a while and then stop.
    I cannot see much use use for that as it is, as it will be bloody-minded about
    the time, and might chop off before the whole job is done, but one could make
    it more intelligent, such that it keeps track of idle time, and aborts after
    say a second (or a minute) of not doing anything useful.

    Unlike you, I am a thread and queue fanatic - use them all the time.

    - Hendrik
     
    Hendrik van Rooyen, Sep 1, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. sarmin kho
    Replies:
    1
    Views:
    389
    Cameron Laird
    Jun 4, 2004
  2. Replies:
    3
    Views:
    1,110
    James Stroud
    Dec 13, 2005
  3. Etienne Van tonder
    Replies:
    1
    Views:
    140
    Tomaso Tosolini
    Dec 28, 2007
  4. ashish
    Replies:
    4
    Views:
    309
    ashish
    Sep 20, 2012
  5. ashish
    Replies:
    5
    Views:
    549
    Piet van Oostrum
    Sep 20, 2012
Loading...

Share This Page