namespace/dictionary quandry

[Jack Carter]

I have been delegated to produce a tool that has
python embedded in it. The desire is to have a
command line interface that inherits all the python
scripting functionality so people can use the tool
either with a flat commandline entry like you would
a debugger or with a script using scoping, loops and
conditional.

I am going from knowing nothing about python to
almost nothing so the learning curve is rather
steep at this point.

I have a dummy program that inherits code.InteractiveConsole
functionality. I can grab each input line and scan it for
commands that match my tool's syntax ("attach") and if it
does, I change the line to reflect a python legal function
call associated with that commond (DoAttach) and from there
do what I want with the line. The actual input lines are
executed with a self.push() call.

The problem I have is in timely symbol resolution. I really
want to run eval() on each of the arguments at the time
the DoAttach() function is executed, but don't know how
and where from does one get the dictionary with the proper
name space. If I run eval at the time of initial reading
of the input it works for flat first level input, but not
for nested scoping:

cli.py
bosco=5
if 1:
attach bosco
bosco=7
attach bosco

attach bosco

This will result in:
... attach bosco
... bosco=7
... attach bosco
...
DoAttach: ['5']
DoAttach: ['5'] <--- WRONG (at least, not what I want) DoAttach: ['7']

How does one export the correct dictionary?

Attached is the toy program that creates this output.

Keep in mind that I am probably aproaching this all
wrong. Any relevant suggestions would be most appreciated,

Jack

*****************
File cli.py:
*****************
#!/usr/bin/env python

import myparse

cli = myparse.CLI(globals())
cli.interact()


*****************
File myparse.py:
*****************
import code
import re
import string
import sys

def DoAttach(args):


print "DoAttach:", args
pass


class CLI(code.InteractiveConsole):
"""Simple test of a Python interpreter augmented with custom
commands."""

commands = { \
"attach" : "DoAttach"
}

def __init__(self, locals = None):

# Call super-class initializer
code.InteractiveConsole.__init__(self, locals, "<console>")

# Compile regular expression for finding commmands
self.regexp = re.compile('[a-z]*')


def interact(self):

my_dictionary = self.locals

# Set the primary and secondary prompts
sys.ps1 = ">>> "
sys.ps2 = "... "

# Input Loop
is_more = 0
bosco = 0
while 1:
try :
# Display the appropriate prompt
if not sys.stdin.isatty():
prompt = ""
elif is_more:
prompt = sys.ps2
else:
prompt = sys.ps1

# Read the next line of input
#self.write("interact 1\n")
line = self.raw_input(prompt)

# TODO: add logging of input line here...

# Process complete lines
if 1:
line = self.process(line)

# Push incomplete lines onto input stack
if line or is_more:
is_more = self.push(line)

# Handle CTRL-C
except KeyboardInterrupt:
self.write("\nKeyboardInterrupt\n")
is_more = 0
self.resetbuffer()

# Handle CTRL-D
except EOFError:
self.write("\n")
is_more = 0
self.resetbuffer()
raise SystemExit


def process(parent, line):

# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)
white_spaces = len_1-len_2
if white_spaces is not 0:
front_padding = line[0:white_spaces]

match = parent.regexp.match(temp_line)
if match is not None:

# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command
dictionary
function = parent.commands.get(cmd_string)

if function is not None:

# Split argument string into individual arguments
args = string.split(arg_string)

# Recursively evaluate all arguments
i = 0
while i < len(args):
try:

# Grab value and position of this argument
unevaluated, pos, i = args, i, i + 1

# Have Python attempt evaluation of the argument

evaluated = eval(unevaluated, parent.locals)

# Did the evaluation produce a "new" result?
if str(evaluated) != str(unevaluated):

# Place the evaluation in the argument list
args = args[os] + \
string.split(str(evaluated)) + \
args[pos + 1:]

# Go back to the beginning of the argument
list
i = 0

except (SyntaxError, NameError):
pass



# Convert to Python function-call syntax for this
command
line = "myparse." + function + "(" + str(args) + ")"
if white_spaces is not 0:
line = front_padding + line

# Return the line to be processed by Python
return line

 

[Peter Otten]

This will result in:
... attach bosco
... bosco=7
... attach bosco
...
DoAttach: ['5']
DoAttach: ['5'] <--- WRONG (at least, not what I want)DoAttach: ['7']

The suite

attach bosco
bosco = 7
attach bosco

is only executed after the final empty line. Therefore any access to the
right dictionary in the process() method will give you the current value
bosco==5 which is not what you want.
You are in effect translating the suite into (simplified)

DoAttach(5)
bosco = 7
DoAttach(5) # the previous line has not yet been executed

I think the easiest way to get the desired effect ("late binding") is like
so:

def process(self, line):
temp_line = line.lstrip()
front_padding = line[:len(line)-len(temp_line)]

match = self.regexp.match(temp_line)
if match:
line = front_padding + "myparse.DoAttach(bosco)"

return line

I. e. change the generated line to contain variable names instead of values
and leave the resolution to python. (You hint you don't wont that either.
Why?)

By the way, I wasn't able to run your code - the indentation is seriously
messed up. I recommend you ensure a 4-space indentation in all code you
currently have before you move on. That will spare you a lot of trouble
later on.

Another minor issue: 'value is 0' may or may not work depending on the
python implementation ('value is 1000' won't work even in current CPython).
With 'value == 0' you are on the safe side.

Peter

 

[Jack Carter]

Peter,

I guess I just don't understand.

Basically this is to go into a tool that accepts
commandline arguments, one at a time, but will also
allow scripting for testing purposes. Thus the desire
to leverage off python for the commandline interface,
but not for the whole program.

The input from the user will be either from the console or
read from a command file. A simple example of what could
be entered is:

bosco=5
if 1:
print bosco
attach bosco 9
bosco=7
print bosco
attach bosco 9

The result I would expect would be for my DoAttach()
routine to receive the python evaluated value of each
of the arguments leaving the ones that it doesn't understand
alone. Whether this happens automagically or by hand I don't
care as long as I get what right value.

In the above trying to follow your advice, remembering that
at this stage of the game I am probably missing the point, this
is the result I expect:

johmar % demo.py.... print bosco
.... attach bosco 9
.... bosco=7
.... print bosco
.... attach bosco 9
....
5
DoAttach: [5, 9]
7
DoAttach: [7, 9]

This is what I get:

johmar % demo.py.... print bosco
.... attach bosco 9
.... bosco=7
.... print bosco
.... attach bosco 9
....
5
DoAttach: ['bosco', '9']
7
DoAttach: ['bosco', '9']

Now I realize that this is probably due to the fact that
I have the lines:

if white_spaces:
line = front_padding + "myparse." + function + "(" +
str(args) + ")"
else :
line = "myparse." + function + "(" + str(args) + ")"

which put make the arguments strings, but that is because I don't
know how to appropriately pack the "line" for later parsing. Maybe
that is the crux of my problem. Remember, there will be many commands
for my tool and the arguments will be variable length and this code
will not know what variables the gentle use would use.

Here is the simplified code with hopefully the tabs expanded
base on your earlier input. Hopefully you'll see the obvious
error of my way and point it out.

Thanks ever so much,

Jack

**********************************
demo.py
**********************************
#!/usr/bin/env python

import myparse

cli = myparse.CLI(globals())
cli.interact()

**********************************
myparse.py
**********************************
import code
import re
import string
import sys

################################################################################
#
# DoAttach
#
# Dummy function that I will eventually use to do
# real stuff.
#
################################################################################
def DoAttach(args):

print "DoAttach:", args
pass


class CLI(code.InteractiveConsole):
"""Simple test of a Python interpreter augmented with custom commands."""

commands = { \
"attach" : "DoAttach"
}

def __init__(self, locals = None):

# Call super-class initializer
code.InteractiveConsole.__init__(self, locals, "<console>")

# Compile regular expression for finding commmands
self.regexp = re.compile('[a-z]*')


##################################################################
#
# interact
#
# This will read and process input lines from within
# my main application as though on a python commandline.
#
##################################################################
def interact(self):

# Set the primary and secondary prompts
sys.ps1 = ">>> "
sys.ps2 = "... "

# Input Loop
is_more = 0
bosco = 0
while 1:
try :
# Display the appropriate prompt
if not sys.stdin.isatty():
prompt = ""
elif is_more:
prompt = sys.ps2
else:
prompt = sys.ps1

# Read the next line of input
#self.write("interact 1\n")
line = self.raw_input(prompt)

# TODO: add logging of input line here...

# Process complete lines
if 1:
line = self.process(line)

# Push incomplete lines onto input stack
if line or is_more:
is_more = self.push(line)

# Handle CTRL-C
except KeyboardInterrupt:
self.write("\nKeyboardInterrupt\n")
is_more = 0
self.resetbuffer()

# Handle CTRL-D
except EOFError:
self.write("\n")
is_more = 0
self.resetbuffer()
raise SystemExit

##################################################################
#
# process
#
# This will determine if the input command is either
# from my application's command language or a python
# construct.
#
##################################################################
def process(parent, line):

# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)

white_spaces = len_1-len_2
if white_spaces:
front_padding = line[0:white_spaces]

match = parent.regexp.match(temp_line)
if match is not None:

#parent.write("process 1\n")
# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command dictionary
function = parent.commands.get(cmd_string)

if function is not None:

# Split argument string into individual arguments
args = string.split(arg_string)

# Convert to Python function-call syntax for this command
if white_spaces:
line = front_padding + "myparse." + function + "(" +
str(args) + ")"
else :
line = "myparse." + function + "(" + str(args) + ")"

# Return the line to be processed by Python
return line

 

[Peter Otten]

Basically this is to go into a tool that accepts
commandline arguments, one at a time, but will also
allow scripting for testing purposes. Thus the desire
to leverage off python for the commandline interface,
but not for the whole program.

The input from the user will be either from the console or
read from a command file. A simple example of what could
be entered is:

bosco=5
if 1:
print bosco
attach bosco 9
bosco=7
print bosco
attach bosco 9

The result I would expect would be for my DoAttach()
routine to receive the python evaluated value of each
of the arguments leaving the ones that it doesn't understand
alone. Whether this happens automagically or by hand I don't
care as long as I get what right value.

In the above trying to follow your advice, remembering that
at this stage of the game I am probably missing the point, this
is the result I expect:

johmar % demo.py... print bosco
... attach bosco 9
... bosco=7
... print bosco
... attach bosco 9
...
5
DoAttach: [5, 9]
7
DoAttach: [7, 9]

This is what I get:

johmar % demo.py... print bosco
... attach bosco 9
... bosco=7
... print bosco
... attach bosco 9
...
5
DoAttach: ['bosco', '9']
7
DoAttach: ['bosco', '9']

Now I realize that this is probably due to the fact that
I have the lines:

if white_spaces:
line = front_padding + "myparse." + function + "(" +
str(args) + ")"
else :
line = "myparse." + function + "(" + str(args) + ")"

which put make the arguments strings, but that is because I don't
know how to appropriately pack the "line" for later parsing. Maybe
that is the crux of my problem. Remember, there will be many commands
for my tool and the arguments will be variable length and this code
will not know what variables the gentle use would use.

Here is the simplified code with hopefully the tabs expanded
base on your earlier input. Hopefully you'll see the obvious
error of my way and point it out.

Thanks ever so much,

Jack

**********************************
demo.py
**********************************
#!/usr/bin/env python

import myparse

cli = myparse.CLI(globals())
cli.interact()

**********************************
myparse.py
**********************************
import code
import re
import string
import sys

################################################################################
#
# DoAttach
#
# Dummy function that I will eventually use to do
# real stuff.
#
################################################################################
def DoAttach(args):

print "DoAttach:", args
pass


class CLI(code.InteractiveConsole):
"""Simple test of a Python interpreter augmented with custom
commands."""

commands = { \
"attach" : "DoAttach"
}

def __init__(self, locals = None):

# Call super-class initializer
code.InteractiveConsole.__init__(self, locals, "<console>")

# Compile regular expression for finding commmands
self.regexp = re.compile('[a-z]*')


##################################################################
#
# interact
#
# This will read and process input lines from within
# my main application as though on a python commandline.
#
##################################################################
def interact(self):

# Set the primary and secondary prompts
sys.ps1 = ">>> "
sys.ps2 = "... "

# Input Loop
is_more = 0
bosco = 0
while 1:
try :
# Display the appropriate prompt
if not sys.stdin.isatty():
prompt = ""
elif is_more:
prompt = sys.ps2
else:
prompt = sys.ps1

# Read the next line of input
#self.write("interact 1\n")
line = self.raw_input(prompt)

# TODO: add logging of input line here...

# Process complete lines
if 1:
line = self.process(line)

# Push incomplete lines onto input stack
if line or is_more:
is_more = self.push(line)

# Handle CTRL-C
except KeyboardInterrupt:
self.write("\nKeyboardInterrupt\n")
is_more = 0
self.resetbuffer()

# Handle CTRL-D
except EOFError:
self.write("\n")
is_more = 0
self.resetbuffer()
raise SystemExit

##################################################################
#
# process
#
# This will determine if the input command is either
# from my application's command language or a python
# construct.
#
##################################################################
def process(parent, line):

# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)

white_spaces = len_1-len_2
if white_spaces:
front_padding = line[0:white_spaces]

match = parent.regexp.match(temp_line)
if match is not None:

#parent.write("process 1\n")
# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command dictionary
function = parent.commands.get(cmd_string)

if function is not None:

# Split argument string into individual arguments
args = string.split(arg_string)

# Convert to Python function-call syntax for this command
if white_spaces:
line = front_padding + "myparse." + function + "(" +
str(args) + ")"
else :
line = "myparse." + function + "(" + str(args) + ")"

# let's add some feedback
print "fed to the snake:", line

# Return the line to be processed by Python
return line

Now

fed to the snake: myparse.DoAttach(['bosco', '2'])
DoAttach: ['bosco', '2']
You build a function call

myparse.DoAttach(['bosco', '2'])

But what you need would rather be

myparse.DoAttach([bosco, 2])

To achieve that you have to somehow extract (Python-compatible) expressions
for the arguments given in the line

attach bosco 2

which can be arbitrarily complex depending on how you defined your custom
language. Assumming that you use the simplest possible spec, a
space-separated list of already Python-compatible expressions that gives
you

def parseArgs(args):
return args.split()

def makePythonCall(func, args):
return "%s([%s])" % (func, ", ".join(args))

and the process() method will become:

def process(parent, line):

# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)

white_spaces = len_1-len_2
if white_spaces:
front_padding = line[0:white_spaces]

match = parent.regexp.match(temp_line)
if match is not None:

#parent.write("process 1\n")
# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command dictionary
function = parent.commands.get(cmd_string)

if function is not None:

args = parseArgs(arg_string)
line = makePythonCall("myparse." + function, args)
if white_spaces:
line = front_padding + line
print "fed to the snake:", line

# Return the line to be processed by Python
return line


Testing it:
fed to the snake: myparse.DoAttach([bosco, 2])
DoAttach: [1, 2]fed to the snake: if 1:
.... attach bosco
fed to the snake: myparse.DoAttach([bosco])
.... bosco = 3
fed to the snake: bosco = 3
.... attach bosco
fed to the snake: myparse.DoAttach([bosco])
....
fed to the snake:
DoAttach: [1]
DoAttach: [3]

Works here, but is not very robust:
fed to the snake: myparse.DoAttach([=, 99])
File "<console>", line 1
myparse.DoAttach([=, 99])
^
SyntaxError: invalid syntaxfed to the snake: myparse.DoAttach([(bosco)])
DoAttach: [3]fed to the snake: myparse.DoAttach([bosco, "so, what"])
DoAttach: [3, 'so, what'] # note the comma

Peter

 

[Jack Carter]

Peter,

Actually although your solution is a good one, it won't
really help me because of the nature of our command language.

We have commands in the form of:

result = <command_name> <filename> -e <experiment_name>

I currently scan ahead for <command_name> and could do so for
the rest to prevent them from being evaluated by python by
packing them with quotes and then unpacking them later. The
problem lies in the case where the name being used in the
commandline may or may not be a formal name such as a filename
or may be a variable that will evaluate into a filename.

This will be used in a list of filenames or a list of experiment
names iterated through a for loop.

for name in (['file1','file2','file3']):

.... expClose name
....
expClose: ['file1']
expClose: ['file2']
expClose: ['file3']

But when the user tries to use a formal name that is
not through a python variable he/she hits an undefine
name error:
Traceback (most recent call last):
File "<console>", line 1, in ?
NameError: name 'file4' is not defined

If I prescan the <filename> and pack it with quotes I
lose the python evaluation of the name.

The same problem will occur with all my other option arguments
for which there are many.

The solution it would seem would be to do the evaluation
later within the called function. That way I could assume
that all failed eval()'ed names are literals meant for my
command and not a python variable/name.

If this makes sense, the problem I need to solve is how to
deliver the correct namespace dictionary to the called function
so I can invoke eval with it.

Does this make sense?

Regards,

Jack

 

[Alex Martelli]

The solution it would seem would be to do the evaluation
later within the called function. That way I could assume
that all failed eval()'ed names are literals meant for my
command and not a python variable/name.

If this makes sense, the problem I need to solve is how to
deliver the correct namespace dictionary to the called function
so I can invoke eval with it.

Does this make sense?

Not very, but then I didn't follow the previous LONG posts on this
thread, so I'll just answer this specific question and hope it helps.
I'll assume the known variable-names are in some dictionary (such as a
locals() or globals() or vars(something)):

class WeirdNamespace:
def __init__(self, d): self.d = d
def __getitem__(self, n): return self.d.get(n,repr(n))

voila: if n is a key in dict d, this returns the corresponding value,
otherwise it returns n suitably quoted. Just pass to eval a
WeirdNamespace(d) rather than the bare d.

It appears to me your user interface is courting trouble: if I mispell
'variablename' as 'varaiblename' I end up creating a file I didn't mean
to rather than getting a clean error about unknown variable names. But
I'll assume you know your users better than I do and that they _do_
really desire with all their hearts this unholy confusion between
variables and constants...


Alex

 

[Jack Carter]

Subject: Re: namespace/dictionary quandry


Not very, but then I didn't follow the previous LONG posts on this

Alex,

The length was really taken up with a testcase to make
the problem less hand wavey.

thread, so I'll just answer this specific question and hope it helps.
I'll assume the known variable-names are in some dictionary (such as a
locals() or globals() or vars(something)):

class WeirdNamespace:
def __init__(self, d): self.d = d
def __getitem__(self, n): return self.d.get(n,repr(n))

So, how and or where does this fit in with my example?
Does both the call to the function where I want to do
the eval() and self.push(line) command have to be in the
same namespace and or file for this to work?

voila: if n is a key in dict d, this returns the corresponding value,
otherwise it returns n suitably quoted. Just pass to eval a
WeirdNamespace(d) rather than the bare d.

It appears to me your user interface is courting trouble: if I mispell
'variablename' as 'varaiblename' I end up creating a file I didn't mean
to rather than getting a clean error about unknown variable names. But
I'll assume you know your users better than I do and that they _do_
really desire with all their hearts this unholy confusion between
variables and constants...

This is driven by customers that who want both a type-in command
language and a variant using all the power of python for test
scripting purposes. If you misspell something and there is the
possibility of it screwing something up that we can't tell either
in the parser or downstream in the backend of the tool, well that's
life in the big city.

Thanks,

Jack

 

[Peter Otten]

So, how and or where does this fit in with my example?
Does both the call to the function where I want to do
the eval() and self.push(line) command have to be in the
same namespace and or file for this to work?

I use a slight variation of Alex' suggestion:

class Locals(dict):
def __getitem__(self, name):
try:
return dict.__getitem__(self, name)
except KeyError:
return name

class CLI(code.InteractiveConsole):
"""Simple test of a Python interpreter augmented with custom
commands."""

commands = { \
"attach" : "DoAttach"
}

def __init__(self, locals = None):

# Call super-class initializer
code.InteractiveConsole.__init__(self, locals, "<console>")
self.locals = Locals(self.locals)

# Compile regular expression for finding commmands
self.regexp = re.compile('[a-z]*')

[The rest of the code is the same as in my previous post]

Now try it:
fed to the snake: for name in "abc":
.... attach name
fed to the snake: myparse.DoAttach([name])
.... attach noname
fed to the snake: myparse.DoAttach([noname])
....
fed to the snake:
DoAttach: ['a']
DoAttach: ['noname']
DoAttach: ['b']
DoAttach: ['noname']
DoAttach: ['c']
DoAttach: ['noname']

Unfortunately this will only work with Python 2.4.

Here's a solution that might work for 2.3:

[Does not require the above modifications]

def process(self, line):
indent = line.lstrip()
# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)

white_spaces = len_1-len_2
if white_spaces:
front_padding = line[0:white_spaces]

match = self.regexp.match(temp_line)
if match is not None:

#self.write("process 1\n")
# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command dictionary
function = self.commands.get(cmd_string)

if function is not None:

args = parseArgs(arg_string)
for arg in args:
if arg not in self.locals:
self.locals[arg] = arg
line = makePythonCall("myparse." + function, args)
if white_spaces:
line = front_padding + line
print "fed to the snake:", line

# Return the line to be processed by Python
return line

I just ensure that all arguments not already in self.locals are added with
their name as their value. (I also renamed 'parent' to 'self' - I could not
stand it any longer

Note that I don't particularly like both hacks and would rather use plain
old python functions with standard python syntax instead of your custom
language.


Peter

 

[Alex Martelli]

...
The length was really taken up with a testcase to make
the problem less hand wavey.

If you can't make a shorter example, I'm not gonna make time to study
that one, sorry.

So, how and or where does this fit in with my example?
Does both the call to the function where I want to do
the eval() and self.push(line) command have to be in the
same namespace and or file for this to work?

All it does is implement what you asked: make a mapping that assumes all
unknown variable names are to be taken as literals, starting from a
mapping of all 'known variable names' to their values.

This is driven by customers that who want both a type-in command
language and a variant using all the power of python for test
scripting purposes. If you misspell something and there is the
possibility of it screwing something up that we can't tell either
in the parser or downstream in the backend of the tool, well that's
life in the big city.

I suspect the first time your users want to use a file named 'for'
they'll scream bloody murder, but hopefully that won't happen before
they've paid you enough for you to retire in comfort to a nice
Carribbean island without extradition treaties, so, cheer up.


Alex

 

[Jack Carter]

Subject: Re: namespace/dictionary quandry

I just ensure that all arguments not already in self.locals are added with
their name as their value. (I also renamed 'parent' to 'self' - I could not
stand it any longer

Novice error on my part. Thanks for the correction.

Note that I don't particularly like both hacks and would rather use plain
old python functions with standard python syntax instead of your custom
language.

It makes the programmers job easier, but not the customer
who may just want a non-gui debugger commandline session tool.
That customer doesn't want to know anything about python or
function calls even though underneath the covers that's what
they are getting. At the same time, the testers want the full
scripting power of python.

To make life even more fun, there is a C++ front end and C++
back end that deal with many processes over many nodes. Some
of the commands are syncronous and return a value that feed
into the python name space and others are asyncronous that
may implode somewhere in the future and need to bring the
whole mess to some sane state to handle the problem

I guess it is my own private hell to work on.

Your change seems to work great!

Thanks,

Jack