Optparse object containing generators: only seem to work if givenparticular names?

Discussion in 'Python' started by John O'Hagan, Oct 31, 2008.

  1. John O'Hagan

    John O'Hagan Guest

    Here's a strange one for you:

    I have a generator function which produces lists of numbers and takes options
    which influence the output. The generator contains a loop, and to enable the
    options to have a different value on each iteration, the options may
    themselves be instances of the same generator, in the form of attributes of
    an optparse object. In the body of the loop, the next() method is called on
    each generator to get the new value each time.

    This enables me to either:

    - pass an option either a single value as and have it yielded repeatedly by a
    dummy generator,

    - or, by using the (arbitrarily chosen) word "CTRL:" after the option,
    followed by another set of options - in quotes - suitable for the main
    generator, to pass a changing value.

    The problem: whether it works or not depends on what name is given to the
    destination in parser.values()! Strange but true.

    I haven't yet been able to figure out if there's any pattern to this, but
    names that work include: "add", "A", "rand", and "updown". Some that don't
    are "descend", "random", and "Z".

    When it doesn't work, the error is:

    TypeError: unsupported operand type(s) for +: 'int' and 'generator'

    So apparently when given one of the "wrong" names, calling options.
    [wrongname].next() yields another generator object, whereas with the "right"
    names, it yields the expected value.

    Below is a working "sandbox" version of the relevant part of the actual
    program, but with a hard-coded value for the number list and only a single
    option, --add, which either adds the number given next on the command line to
    each element of the list, or, if followed by CTRL: '' " (that's a pair of
    empty quotes) will sequentially do the same for each member of the list. (If
    the option is repeated inside the quotes, the altered list will be used to
    apply the addition, and so on.)

    If you don't believe me (and I'm finding this hard to believe myself), try
    some different names in the two inline-commented places (of course, both have
    to be the same). Some will work, others won't.

    I'm baffled, and hope someone here may be interested in figuring out why this
    is happening.

    Thanks,

    John O'Hagan

    Here's the code:

    import sys
    from optparse import OptionParser

    def my_callback(option, opt_str, value, parser):

    rargs = parser.rargs

    if rargs[0] == "CTRL:" :
    setattr(parser.values, option.dest+"_ctrl", rargs[1].split())
    else:
    value = abs(int(rargs[0]))
    setattr(parser.values, option.dest, value)

    def parse(option_list):

    parser = OptionParser()

    parser.add_option("--add", action="callback", callback=my_callback,
    dest="add") ## The troublesome name!

    (options, args) = parser.parse_args(option_list)
    return options


    def numerical_ctrl(generator):

    for number_list in generator:
    for number in number_list:
    yield number

    def dummy_ctrl(value):
    while 1:
    yield value

    #This next bit should ideally be incorporated into the callback function above
    #(later!)
    #It converts the attributes of the "options" object to generators.

    def iterizer( options ):

    for opt in vars( options ) :

    if "_ctrl" in opt:
    ctrl_opts_list = getattr( options, opt )
    setattr(options, opt, dummy_ctrl( getattr( options, opt ) ) )
    opt = opt.replace( "_ctrl", "" )
    ctrl_opts = parse( ctrl_opts_list )
    ctrl_opts = iterizer( ctrl_opts )
    generator = phrase_engine( ctrl_opts )
    generator = numerical_ctrl( generator )
    else:
    generator = dummy_ctrl( getattr( options, opt ) )

    setattr (options, opt , generator)

    return options

    #The number list generator:

    def phrase_engine(options):

    counter = 0
    while counter < 10:

    sequence = [1, 2, 3]

    add = options.add.next() ##Here is the troublesome name again.

    if add :
    sequence = [ i + add for i in sequence ]

    yield sequence

    counter += 1

    options = parse( sys.argv[ 1: ] )
    options = iterizer(options)

    for i in phrase_engine(options):
    print i
    John O'Hagan, Oct 31, 2008
    #1
    1. Advertising

  2. John O'Hagan

    Peter Otten Guest

    Re: Optparse object containing generators: only seem to work if given particular names?

    John O'Hagan wrote:

    > Here's a strange one for you:
    >
    > I have a generator function which produces lists of numbers and takes
    > options which influence the output. The generator contains a loop, and to
    > enable the options to have a different value on each iteration, the
    > options may themselves be instances of the same generator, in the form of
    > attributes of an optparse object. In the body of the loop, the next()
    > method is called on each generator to get the new value each time.
    >
    > This enables me to either:
    >
    > - pass an option either a single value as and have it yielded repeatedly
    > by a dummy generator,
    >
    > - or, by using the (arbitrarily chosen) word "CTRL:" after the option,
    > followed by another set of options - in quotes - suitable for the main
    > generator, to pass a changing value.
    >
    > The problem: whether it works or not depends on what name is given to the
    > destination in parser.values()! Strange but true.
    >
    > I haven't yet been able to figure out if there's any pattern to this, but
    > names that work include: "add", "A", "rand", and "updown". Some that don't
    > are "descend", "random", and "Z".
    >
    > When it doesn't work, the error is:
    >
    > TypeError: unsupported operand type(s) for +: 'int' and 'generator'
    >
    > So apparently when given one of the "wrong" names, calling options.
    > [wrongname].next() yields another generator object, whereas with the
    > ["right"
    > names, it yields the expected value.
    >
    > Below is a working "sandbox" version of the relevant part of the actual
    > program, but with a hard-coded value for the number list and only a single
    > option, --add, which either adds the number given next on the command line
    > to each element of the list, or, if followed by CTRL: '' " (that's a pair
    > of empty quotes) will sequentially do the same for each member of the
    > list. (If the option is repeated inside the quotes, the altered list will
    > be used to apply the addition, and so on.)
    >
    > If you don't believe me (and I'm finding this hard to believe myself), try
    > some different names in the two inline-commented places (of course, both
    > have to be the same). Some will work, others won't.
    >
    > I'm baffled, and hope someone here may be interested in figuring out why
    > this is happening.
    >
    > Thanks,
    >
    > John O'Hagan
    >
    > Here's the code:
    >
    > import sys
    > from optparse import OptionParser
    >
    > def my_callback(option, opt_str, value, parser):
    >
    > rargs = parser.rargs
    >
    > if rargs[0] == "CTRL:" :
    > setattr(parser.values, option.dest+"_ctrl", rargs[1].split())
    > else:
    > value = abs(int(rargs[0]))
    > setattr(parser.values, option.dest, value)
    >
    > def parse(option_list):
    >
    > parser = OptionParser()
    >
    > parser.add_option("--add", action="callback", callback=my_callback,
    > dest="add") ## The troublesome name!
    >
    > (options, args) = parser.parse_args(option_list)
    > return options
    >
    >
    > def numerical_ctrl(generator):
    >
    > for number_list in generator:
    > for number in number_list:
    > yield number
    >
    > def dummy_ctrl(value):
    > while 1:
    > yield value
    >
    > #This next bit should ideally be incorporated into the callback function
    > #above (later!)
    > #It converts the attributes of the "options" object to generators.
    >
    > def iterizer( options ):
    >
    > for opt in vars( options ) :
    >
    > if "_ctrl" in opt:
    > ctrl_opts_list = getattr( options, opt )
    > setattr(options, opt, dummy_ctrl( getattr( options, opt ) ) )
    > opt = opt.replace( "_ctrl", "" )
    > ctrl_opts = parse( ctrl_opts_list )
    > ctrl_opts = iterizer( ctrl_opts )
    > generator = phrase_engine( ctrl_opts )
    > generator = numerical_ctrl( generator )
    > else:
    > generator = dummy_ctrl( getattr( options, opt ) )
    >
    > setattr (options, opt , generator)
    >
    > return options
    >
    > #The number list generator:
    >
    > def phrase_engine(options):
    >
    > counter = 0
    > while counter < 10:
    >
    > sequence = [1, 2, 3]
    >
    > add = options.add.next() ##Here is the troublesome name again.
    >
    > if add :
    > sequence = [ i + add for i in sequence ]
    >
    > yield sequence
    >
    > counter += 1
    >
    > options = parse( sys.argv[ 1: ] )
    > options = iterizer(options)
    >
    > for i in phrase_engine(options):
    > print i


    Just a quick note: Your iterizer() implementation seems to depend on xxx
    occuring before xxx_ctrl. That's only the case for some values:

    >>> dict(add_ctrl=1, add=2)

    {'add': 2, 'add_ctrl': 1}
    >>> dict(random_ctrl=1, random=2)

    {'random_ctrl': 1, 'random': 2}

    While you can enforce that with sorted(vars(options)) I suggest that you try
    to simplify your code a bit -- you know, debugging being twice as hard as
    coding...

    Peter
    Peter Otten, Oct 31, 2008
    #2
    1. Advertising

  3. John O'Hagan

    John O'Hagan Guest

    On Fri, 31 Oct 2008, Peter Otten wrote:
    > John O'Hagan wrote:
    > > Here's a strange one for you:
    > >
    > > I have a generator function which produces lists of numbers and takes
    > > options which influence the output. The generator contains a loop, and to
    > > enable the options to have a different value on each iteration, the
    > > options may themselves be instances of the same generator, in the form of
    > > attributes of an optparse object. In the body of the loop, the next()
    > > method is called on each generator to get the new value each time.

    [..]
    > >
    > > The problem: whether it works or not depends on what name is given to the
    > > destination in parser.values()! Strange but true.

    [...]
    > >
    > > Here's the code:
    > >
    > > import sys
    > > from optparse import OptionParser
    > >
    > > def my_callback(option, opt_str, value, parser):
    > >
    > > rargs = parser.rargs
    > >
    > > if rargs[0] == "CTRL:" :
    > > setattr(parser.values, option.dest+"_ctrl", rargs[1].split())
    > > else:
    > > value = abs(int(rargs[0]))
    > > setattr(parser.values, option.dest, value)
    > >
    > > def parse(option_list):
    > >
    > > parser = OptionParser()
    > >
    > > parser.add_option("--add", action="callback", callback=my_callback,
    > > dest="add") ## The troublesome name!
    > >
    > > (options, args) = parser.parse_args(option_list)
    > > return options
    > >
    > >
    > > def numerical_ctrl(generator):
    > >
    > > for number_list in generator:
    > > for number in number_list:
    > > yield number
    > >
    > > def dummy_ctrl(value):
    > > while 1:
    > > yield value
    > >
    > > #This next bit should ideally be incorporated into the callback function
    > > #above (later!)
    > > #It converts the attributes of the "options" object to generators.
    > >
    > > def iterizer( options ):
    > >
    > > for opt in vars( options ) :
    > >
    > > if "_ctrl" in opt:
    > > ctrl_opts_list = getattr( options, opt )
    > > setattr(options, opt, dummy_ctrl( getattr( options, opt ) ) )
    > > opt = opt.replace( "_ctrl", "" )
    > > ctrl_opts = parse( ctrl_opts_list )
    > > ctrl_opts = iterizer( ctrl_opts )
    > > generator = phrase_engine( ctrl_opts )
    > > generator = numerical_ctrl( generator )
    > > else:
    > > generator = dummy_ctrl( getattr( options, opt ) )
    > >
    > > setattr (options, opt , generator)
    > >
    > > return options
    > >
    > > #The number list generator:
    > >
    > > def phrase_engine(options):
    > >
    > > counter = 0
    > > while counter < 10:
    > >
    > > sequence = [1, 2, 3]
    > >
    > > add = options.add.next() ##Here is the troublesome name again.
    > >
    > > if add :
    > > sequence = [ i + add for i in sequence ]
    > >
    > > yield sequence
    > >
    > > counter += 1
    > >
    > > options = parse( sys.argv[ 1: ] )
    > > options = iterizer(options)
    > >
    > > for i in phrase_engine(options):
    > > print i

    >
    > Just a quick note: Your iterizer() implementation seems to depend on xxx
    >
    > occuring before xxx_ctrl.


    [...]

    Thanks for your eagle eye, Peter, that's exactly what it was; I hadn't
    realized the original xxx option was still in the game, a fact that was
    masked by that ugly "ctrl_" kludge.

    >
    > While you can enforce that with sorted(vars(options)) I suggest that you
    > try to simplify your code a bit -- you know, debugging being twice as hard
    > as coding...
    >

    [...]
    That's excellent advice, my callback function now simply stores the string
    after "CTRL:" as a word list (without creating a new destination), and
    iterizer() just checks if each option's value is a list:

    def iterizer(options):

    for opt in vars(options) :
    value = getattr(options, opt)
    if type(value) is list:
    ctrl_opts = parse(value)
    ctrl_opts = iterizer(ctrl_opts)
    generator = phrase_engine(ctrl_opts)
    generator = numerical_ctrl(generator)
    else:
    generator = dummy_ctrl(getattr(options, opt))
    setattr (options, opt , generator)

    return options

    I have a feeling type-checking is a little bit naughty as well, but it's a
    foolproof way to tell which kind of option has been applied, and it works!

    Thanks again,

    John
    John O'Hagan, Nov 1, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chris Pudney
    Replies:
    1
    Views:
    814
    Chris Pudney
    Dec 5, 2003
  2. wanwan
    Replies:
    3
    Views:
    417
    Alex Martelli
    Oct 14, 2005
  3. Replies:
    5
    Views:
    349
    CBFalconer
    Jun 28, 2007
  4. Paul Childs
    Replies:
    1
    Views:
    314
    Matimus
    Jun 3, 2008
  5. defn noob

    Generators can only yield ints?

    defn noob, Aug 22, 2008, in forum: Python
    Replies:
    4
    Views:
    257
Loading...

Share This Page