Optparse object containing generators: only seem to work if givenparticular names?

[John O'Hagan]

Here's a strange one for you:

I have a generator function which produces lists of numbers and takes options
which influence the output. The generator contains a loop, and to enable the
options to have a different value on each iteration, the options may
themselves be instances of the same generator, in the form of attributes of
an optparse object. In the body of the loop, the next() method is called on
each generator to get the new value each time.

This enables me to either:

- pass an option either a single value as and have it yielded repeatedly by a
dummy generator,

- or, by using the (arbitrarily chosen) word "CTRL:" after the option,
followed by another set of options - in quotes - suitable for the main
generator, to pass a changing value.

The problem: whether it works or not depends on what name is given to the
destination in parser.values()! Strange but true.

I haven't yet been able to figure out if there's any pattern to this, but
names that work include: "add", "A", "rand", and "updown". Some that don't
are "descend", "random", and "Z".

When it doesn't work, the error is:

TypeError: unsupported operand type(s) for +: 'int' and 'generator'

So apparently when given one of the "wrong" names, calling options.
[wrongname].next() yields another generator object, whereas with the "right"
names, it yields the expected value.

Below is a working "sandbox" version of the relevant part of the actual
program, but with a hard-coded value for the number list and only a single
option, --add, which either adds the number given next on the command line to
each element of the list, or, if followed by CTRL: '' " (that's a pair of
empty quotes) will sequentially do the same for each member of the list. (If
the option is repeated inside the quotes, the altered list will be used to
apply the addition, and so on.)

If you don't believe me (and I'm finding this hard to believe myself), try
some different names in the two inline-commented places (of course, both have
to be the same). Some will work, others won't.

I'm baffled, and hope someone here may be interested in figuring out why this
is happening.

Thanks,

John O'Hagan

Here's the code:

import sys
from optparse import OptionParser

def my_callback(option, opt_str, value, parser):

rargs = parser.rargs

if rargs[0] == "CTRL:" :
setattr(parser.values, option.dest+"_ctrl", rargs[1].split())
else:
value = abs(int(rargs[0]))
setattr(parser.values, option.dest, value)

def parse(option_list):

parser = OptionParser()

parser.add_option("--add", action="callback", callback=my_callback,
dest="add") ## The troublesome name!

(options, args) = parser.parse_args(option_list)
return options


def numerical_ctrl(generator):

for number_list in generator:
for number in number_list:
yield number

def dummy_ctrl(value):
while 1:
yield value

#This next bit should ideally be incorporated into the callback function above
#(later!)
#It converts the attributes of the "options" object to generators.

def iterizer( options ):

for opt in vars( options ) :

if "_ctrl" in opt:
ctrl_opts_list = getattr( options, opt )
setattr(options, opt, dummy_ctrl( getattr( options, opt ) ) )
opt = opt.replace( "_ctrl", "" )
ctrl_opts = parse( ctrl_opts_list )
ctrl_opts = iterizer( ctrl_opts )
generator = phrase_engine( ctrl_opts )
generator = numerical_ctrl( generator )
else:
generator = dummy_ctrl( getattr( options, opt ) )

setattr (options, opt , generator)

return options

#The number list generator:

def phrase_engine(options):

counter = 0
while counter < 10:

sequence = [1, 2, 3]

add = options.add.next() ##Here is the troublesome name again.

if add :
sequence = [ i + add for i in sequence ]

yield sequence

counter += 1

options = parse( sys.argv[ 1: ] )
options = iterizer(options)

for i in phrase_engine(options):
print i

 

[Peter Otten]

Here's a strange one for you:

I have a generator function which produces lists of numbers and takes
options which influence the output. The generator contains a loop, and to
enable the options to have a different value on each iteration, the
options may themselves be instances of the same generator, in the form of
attributes of an optparse object. In the body of the loop, the next()
method is called on each generator to get the new value each time.

This enables me to either:

- pass an option either a single value as and have it yielded repeatedly
by a dummy generator,

- or, by using the (arbitrarily chosen) word "CTRL:" after the option,
followed by another set of options - in quotes - suitable for the main
generator, to pass a changing value.

The problem: whether it works or not depends on what name is given to the
destination in parser.values()! Strange but true.

I haven't yet been able to figure out if there's any pattern to this, but
names that work include: "add", "A", "rand", and "updown". Some that don't
are "descend", "random", and "Z".

When it doesn't work, the error is:

TypeError: unsupported operand type(s) for +: 'int' and 'generator'

So apparently when given one of the "wrong" names, calling options.
[wrongname].next() yields another generator object, whereas with the
["right"
names, it yields the expected value.

Below is a working "sandbox" version of the relevant part of the actual
program, but with a hard-coded value for the number list and only a single
option, --add, which either adds the number given next on the command line
to each element of the list, or, if followed by CTRL: '' " (that's a pair
of empty quotes) will sequentially do the same for each member of the
list. (If the option is repeated inside the quotes, the altered list will
be used to apply the addition, and so on.)

If you don't believe me (and I'm finding this hard to believe myself), try
some different names in the two inline-commented places (of course, both
have to be the same). Some will work, others won't.

I'm baffled, and hope someone here may be interested in figuring out why
this is happening.

Thanks,

John O'Hagan

Here's the code:

import sys
from optparse import OptionParser

def my_callback(option, opt_str, value, parser):

rargs = parser.rargs

if rargs[0] == "CTRL:" :
setattr(parser.values, option.dest+"_ctrl", rargs[1].split())
else:
value = abs(int(rargs[0]))
setattr(parser.values, option.dest, value)

def parse(option_list):

parser = OptionParser()

parser.add_option("--add", action="callback", callback=my_callback,
dest="add") ## The troublesome name!

(options, args) = parser.parse_args(option_list)
return options


def numerical_ctrl(generator):

for number_list in generator:
for number in number_list:
yield number

def dummy_ctrl(value):
while 1:
yield value

#This next bit should ideally be incorporated into the callback function
#above (later!)
#It converts the attributes of the "options" object to generators.

def iterizer( options ):

for opt in vars( options ) :

if "_ctrl" in opt:
ctrl_opts_list = getattr( options, opt )
setattr(options, opt, dummy_ctrl( getattr( options, opt ) ) )
opt = opt.replace( "_ctrl", "" )
ctrl_opts = parse( ctrl_opts_list )
ctrl_opts = iterizer( ctrl_opts )
generator = phrase_engine( ctrl_opts )
generator = numerical_ctrl( generator )
else:
generator = dummy_ctrl( getattr( options, opt ) )

setattr (options, opt , generator)

return options

#The number list generator:

def phrase_engine(options):

counter = 0
while counter < 10:

sequence = [1, 2, 3]

add = options.add.next() ##Here is the troublesome name again.

if add :
sequence = [ i + add for i in sequence ]

yield sequence

counter += 1

options = parse( sys.argv[ 1: ] )
options = iterizer(options)

for i in phrase_engine(options):
print i

Just a quick note: Your iterizer() implementation seems to depend on xxx
occuring before xxx_ctrl. That's only the case for some values:
{'random_ctrl': 1, 'random': 2}

While you can enforce that with sorted(vars(options)) I suggest that you try
to simplify your code a bit -- you know, debugging being twice as hard as
coding...

Peter

 

[John O'Hagan]

Here's a strange one for you:

I have a generator function which produces lists of numbers and takes
options which influence the output. The generator contains a loop, and to
enable the options to have a different value on each iteration, the
options may themselves be instances of the same generator, in the form of
attributes of an optparse object. In the body of the loop, the next()
method is called on each generator to get the new value each time. [..]

The problem: whether it works or not depends on what name is given to the
destination in parser.values()! Strange but true. [...]

Here's the code:

import sys
from optparse import OptionParser

def my_callback(option, opt_str, value, parser):

rargs = parser.rargs

if rargs[0] == "CTRL:" :
setattr(parser.values, option.dest+"_ctrl", rargs[1].split())
else:
value = abs(int(rargs[0]))
setattr(parser.values, option.dest, value)

def parse(option_list):

parser = OptionParser()

parser.add_option("--add", action="callback", callback=my_callback,
dest="add") ## The troublesome name!

(options, args) = parser.parse_args(option_list)
return options


def numerical_ctrl(generator):

for number_list in generator:
for number in number_list:
yield number

def dummy_ctrl(value):
while 1:
yield value

#This next bit should ideally be incorporated into the callback function
#above (later!)
#It converts the attributes of the "options" object to generators.

def iterizer( options ):

for opt in vars( options ) :

if "_ctrl" in opt:
ctrl_opts_list = getattr( options, opt )
setattr(options, opt, dummy_ctrl( getattr( options, opt ) ) )
opt = opt.replace( "_ctrl", "" )
ctrl_opts = parse( ctrl_opts_list )
ctrl_opts = iterizer( ctrl_opts )
generator = phrase_engine( ctrl_opts )
generator = numerical_ctrl( generator )
else:
generator = dummy_ctrl( getattr( options, opt ) )

setattr (options, opt , generator)

return options

#The number list generator:

def phrase_engine(options):

counter = 0
while counter < 10:

sequence = [1, 2, 3]

add = options.add.next() ##Here is the troublesome name again.

if add :
sequence = [ i + add for i in sequence ]

yield sequence

counter += 1

options = parse( sys.argv[ 1: ] )
options = iterizer(options)

for i in phrase_engine(options):
print i

Just a quick note: Your iterizer() implementation seems to depend on xxx

occuring before xxx_ctrl.

[...]

Thanks for your eagle eye, Peter, that's exactly what it was; I hadn't
realized the original xxx option was still in the game, a fact that was
masked by that ugly "ctrl_" kludge.

While you can enforce that with sorted(vars(options)) I suggest that you
try to simplify your code a bit -- you know, debugging being twice as hard
as coding...

[...]
That's excellent advice, my callback function now simply stores the string
after "CTRL:" as a word list (without creating a new destination), and
iterizer() just checks if each option's value is a list:

def iterizer(options):

for opt in vars(options) :
value = getattr(options, opt)
if type(value) is list:
ctrl_opts = parse(value)
ctrl_opts = iterizer(ctrl_opts)
generator = phrase_engine(ctrl_opts)
generator = numerical_ctrl(generator)
else:
generator = dummy_ctrl(getattr(options, opt))
setattr (options, opt , generator)

return options

I have a feeling type-checking is a little bit naughty as well, but it's a
foolproof way to tell which kind of option has been applied, and it works!

Thanks again,

John