Bounds checking

Discussion in 'Python' started by Martin De Kauwe, Mar 18, 2011.

  1. Hi,

    if one has a set of values which should never step outside certain
    bounds (for example if the values were negative then they wouldn't be
    physically meaningful) is there a nice way to bounds check? I
    potentially have 10 or so values I would like to check at the end of
    each iteration. However as the loop is over many years I figured I
    probably want to be as optimal as possible with my check. Any
    thoughts?

    e.g. this is my solution

    # module contain data
    # e.g. print state.something might produce 4.0
    import state as state

    def main():
    for i in xrange(num_days):
    # do stuff

    # bounds check at end of iteration
    bounds_check(state)


    def bounds_check(state):
    """ check state values are > 0 """
    for attr in dir(state):
    if not attr.startswith('__') and getattr(state, attr) < 0.0:
    print "Error state values < 0: %s" % (attr)
    sys.exit()

    if __name__ == "__main__":
    sys.exit(main())

    thanks

    Martin
    Martin De Kauwe, Mar 18, 2011
    #1
    1. Advertising

  2. Martin De Kauwe wrote:
    > Hi,
    >
    > if one has a set of values which should never step outside certain
    > bounds (for example if the values were negative then they wouldn't be
    > physically meaningful) is there a nice way to bounds check? I
    > potentially have 10 or so values I would like to check at the end of
    > each iteration. However as the loop is over many years I figured I
    > probably want to be as optimal as possible with my check. Any
    > thoughts?
    >
    > e.g. this is my solution
    >
    > # module contain data
    > # e.g. print state.something might produce 4.0
    > import state as state
    >
    > def main():
    > for i in xrange(num_days):
    > # do stuff
    >
    > # bounds check at end of iteration
    > bounds_check(state)
    >
    >
    > def bounds_check(state):
    > """ check state values are > 0 """
    > for attr in dir(state):
    > if not attr.startswith('__') and getattr(state, attr) < 0.0:
    > print "Error state values < 0: %s" % (attr)
    > sys.exit()
    >
    > if __name__ == "__main__":
    > sys.exit(main())
    >
    > thanks
    >
    > Martin
    >

    Don't check for bounds, fix any bug in the code that would set your
    values out of bounds and use asserts while debugging.

    Otherwise if you really need dynamic checks, it will cost you cpu, for
    sure. Howeverver you could for instance override the __setatttr__ of
    state object, and call the attribute's associated function.

    class State(object):
    funcTable = {
    'foo': lambda x: x >= 0.0
    }

    def __init__(self):
    self.foo = 0

    def __setattr__(self, attribute, value):
    if not self.funcTable.get(attribute, lambda x: True)(value):
    sys.exit('error out of bound')
    return object.__setattr(self, attribute, value)


    Untested, however it's just an idea. I'm not even sure that would be
    less cpu consuming :D
    That way only attributes in functable execute a (cpu consuming ?) test
    function, all other attributes will execute 'lambda x: True'.

    The check occurs everytime you set an attribute however.

    JM
    Jean-Michel Pichavant, Mar 18, 2011
    #2
    1. Advertising

  3. Martin De Kauwe

    Mel Guest

    Jean-Michel Pichavant wrote:
    > Martin De Kauwe wrote:


    > Don't check for bounds, fix any bug in the code that would set your
    > values out of bounds and use asserts while debugging.

    [ ... ]
    > def __setattr__(self, attribute, value):
    > if not self.funcTable.get(attribute, lambda x: True)(value):
    > sys.exit('error out of bound')
    > return object.__setattr(self, attribute, value)


    Offhand, my only quibble is that sys.exit is not helpful for debugging.
    Much better to raise an error:

    if not self.funcTable.get(attribute, lambda x: True)(value):
    raise ValueError ('error out of bound')

    or define a subclass of ValueError just for this purpose. On error, the
    program will stop just as dead, but you'll get a trace.

    Mel.
    Mel, Mar 18, 2011
    #3

  4. > Don't check for bounds, fix any bug in the code that would set your
    > values out of bounds and use asserts while debugging.
    >


    whilst that is a nice idea in practice this just is not a practical
    solution.


    > Otherwise if you really need dynamic checks, it will cost you cpu, for
    > sure.


    Yes I agree and I hadn't decided whether to add it or not as there
    aren't any current issues. However I can see that the check would
    overall be safer. I was just wondering if there was some super smartie
    pants solution :p


    Howeverver you could for instance override the __setatttr__ of
    > state object, and call the attribute's associated function.
    >
    > class State(object):
    >     funcTable = {
    >        'foo': lambda x: x >= 0.0
    >     }
    >
    >     def __init__(self):
    >        self.foo = 0
    >
    >     def __setattr__(self, attribute, value):
    >        if not self.funcTable.get(attribute, lambda x: True)(value):
    >            sys.exit('error out of bound')
    >        return object.__setattr(self, attribute, value)
    >
    > Untested, however it's just an idea. I'm not even sure that would be
    > less cpu consuming :D


    thanks I will look at what you suggested.
    Martin De Kauwe, Mar 18, 2011
    #4

  5. > Offhand, my only quibble is that sys.exit is not helpful for debugging.  
    > Much better to raise an error:
    >
    >         if not self.funcTable.get(attribute, lambda x: True)(value):
    >             raise ValueError ('error out of bound')
    >
    > or define a subclass of ValueError just for this purpose.  On error, the
    > program will stop just as dead, but you'll get a trace.
    >
    >         Mel.


    I think generally I prefer my code to die and as long as I know where
    (from the statement) the error occurred I know generally what point I
    have to debug up until. Can you explain how your solution would be
    easier with a trace as I don't tend to use the raise/assert
    functionality so I am interested.

    thanks
    Martin De Kauwe, Mar 18, 2011
    #5
  6. Martin De Kauwe

    Terry Reedy Guest

    On 3/18/2011 10:24 AM, Martin De Kauwe wrote:

    > def bounds_check(state):
    > """ check state values are> 0 """
    > for attr in dir(state):
    > if not attr.startswith('__') and getattr(state, attr)< 0.0:
    > print "Error state values< 0: %s" % (attr)


    dir() has to do a bit a computation. I would be tempted to give 'state'
    a set of attributes to check. Call it 'nonnegatives'.
    for attr in nonnegatives:
    if ...

    This allows for attributes not subject to that check.

    --
    Terry Jan Reedy
    Terry Reedy, Mar 19, 2011
    #6
  7. On Fri, 18 Mar 2011 07:24:33 -0700, Martin De Kauwe wrote:

    > Hi,
    >
    > if one has a set of values which should never step outside certain
    > bounds (for example if the values were negative then they wouldn't be
    > physically meaningful) is there a nice way to bounds check? I
    > potentially have 10 or so values I would like to check at the end of
    > each iteration.


    assert all(x >= 0 for x in (a, b, c, d, e, f, g, h, i, j))


    > However as the loop is over many years I figured I
    > probably want to be as optimal as possible with my check. Any thoughts?
    >
    > e.g. this is my solution
    >
    > # module contain data
    > # e.g. print state.something might produce 4.0 import state as state
    >
    > def main():
    > for i in xrange(num_days):
    > # do stuff
    >
    > # bounds check at end of iteration
    > bounds_check(state)


    Why don't you do the range check *before* storing it in state? That way
    you can identify the calculation that was wrong, instead of merely
    noticing that at some point some unknown calculation went wrong.


    > def bounds_check(state):
    > """ check state values are > 0 """
    > for attr in dir(state):
    > if not attr.startswith('__') and getattr(state, attr) < 0.0:


    You're looking at every single attribute, including those of super
    classes, when you only want to check "10 or so" attributes. That's
    probably not wise. At the very least, dir() will be a fairly expensive
    call.

    If you insist on checking state *after* the value is stored, instead of
    preventing it from being stored in the first place, it is better to make
    the state object responsible for doing it's own bounds checking. That way
    only the state object needs to be updated when you change those ten
    attributes, instead of some arbitrary number of places scattered all
    throughout your code.


    > print "Error state values < 0: %s" % (attr)
    > sys.exit()


    Python already has a mechanism for printing an error message and exiting:
    raise.

    if condition:
    raise ValueError("attribute %s is negative" % name)

    This will still halt the application, but it does so without stomping all
    over normal conventions for command line applications (error messages
    should go to stderr, not stdout; the return result should be non-zero) as
    well as normal conventions for Python code (the caller should be able to
    easily catch the exception -- catching sys.exit can be done, but it is a
    pretty unusual thing to do).


    --
    Steven
    Steven D'Aprano, Mar 19, 2011
    #7
  8. On Fri, 18 Mar 2011 15:35:40 -0700, Martin De Kauwe wrote:

    >> Don't check for bounds, fix any bug in the code that would set your
    >> values out of bounds and use asserts while debugging.
    >>
    >>

    > whilst that is a nice idea in practice this just is not a practical
    > solution.


    Sorry, are you trying to say that it is not practical to write correct
    code that isn't buggy? Well, you're honest, at least, still I can't help
    but feel that you're admitting defeat before even starting.


    >> Otherwise if you really need dynamic checks, it will cost you cpu, for
    >> sure.

    >
    > Yes I agree and I hadn't decided whether to add it or not as there
    > aren't any current issues. However I can see that the check would
    > overall be safer. I was just wondering if there was some super smartie
    > pants solution :p


    Make each of the attributes a computed attribute with a setter that
    raises an exception if you try to store a negative value. That way each
    attribute is responsible for ensuring that itself is never negative.

    Here's a toy example:


    >>> class Demo(object):

    .... def __init__(self, x):
    .... self.x = x
    .... def _getx(self):
    .... return self._x
    .... def _setx(self, value):
    .... if value < 0:
    .... raise ValueError('attempt to set x to negative value')
    .... self._x = value
    .... x = property(_getx, _setx)
    ....
    >>> d = Demo(-42)

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 3, in __init__
    File "<stdin>", line 8, in _setx
    ValueError: attempt to set x to negative value
    >>> d = Demo(42)
    >>> d.x

    42
    >>> d.x = -23

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 8, in _setx
    ValueError: attempt to set x to negative value
    >>> d.x = 23
    >>> d.x

    23


    --
    Steven
    Steven D'Aprano, Mar 19, 2011
    #8

  9. > dir() has to do a bit a computation. I would be tempted to give 'state'
    > a set of attributes to check. Call it 'nonnegatives'.
    >     for attr in nonnegatives:
    >        if ...
    >
    > This allows for attributes not subject to that check.
    >
    > --
    > Terry Jan Reedy


    Agreed. I was trying to just write a dummy example quickly to go with
    my question and that was just the way that came to mind so that I
    could loop over them to test. It wasn't a great example sorry! In
    reality I would just have like you said a subset a list I guess of the
    ones I would check.
    Martin De Kauwe, Mar 19, 2011
    #9

  10. > assert all(x >= 0 for x in (a, b, c, d, e, f, g, h, i, j))


    yep neat!



    > Why don't you do the range check *before* storing it in state? That way
    > you can identify the calculation that was wrong, instead of merely
    > noticing that at some point some unknown calculation went wrong.


    I guess no reason really. I suppose in my mind I was thinking it was
    an unlikely safeguard but I liked the idea of adding so would just do
    it at the end of a time step. In reality I think there is practically
    no difference and this way it is done once, in a single location vs.
    potential 10 separate checks? I don't see the advantage?

    > You're looking at every single attribute, including those of super
    > classes, when you only want to check "10 or so" attributes. That's
    > probably not wise. At the very least, dir() will be a fairly expensive
    > call.


    yes your right, sorry I was just trying to post a quick example to go
    with my question. I would use a list of the attributes to check.

    >
    > If you insist on checking state *after* the value is stored, instead of
    > preventing it from being stored in the first place, it is better to make
    > the state object responsible for doing it's own bounds checking. That way
    > only the state object needs to be updated when you change those ten
    > attributes, instead of some arbitrary number of places scattered all
    > throughout your code.


    yes I think this is what I was getting at, I will look at your
    suggestion thanks.
    Martin De Kauwe, Mar 19, 2011
    #10

  11. > Sorry, are you trying to say that it is not practical to write correct
    > code that isn't buggy? Well, you're honest, at least, still I can't help
    > but feel that you're admitting defeat before even starting.


    No. What I am saying is the code is written has been well tested and
    *appears* to be working well. However the code is complicated and
    there is potential for bugs. I think I am just been practical here,
    evidently I can't think of everything, but there are some clear and
    obvious errors that would be worth checking for. I can only explain
    this in the terms of the code (sorry)...but for example the model
    estimates plant photosynthesis and then allocates the carbon. So one
    clear example is that the model cuts back carbon production if there
    is water stress for the plant. This involves "removing" carbon from
    the state. Clearly if you ended up in a situation where there is
    negative carbon in a leaf, i.e. the leaf doesn't exist well this is
    not physically possible and would be a code issue. Whilst this is
    unlikely I think it would be nice to have a catch for it. Another
    example would be the amount of soil water available to the plant,
    again there can be zero but not negative soil water. It wouldn't be
    meaningful. I hope that makes sense?

    thanks
    Martin De Kauwe, Mar 19, 2011
    #11
  12. On Sat, 19 Mar 2011 01:38:10 -0700, Martin De Kauwe wrote:

    >> Why don't you do the range check *before* storing it in state? That way
    >> you can identify the calculation that was wrong, instead of merely
    >> noticing that at some point some unknown calculation went wrong.

    >
    > I guess no reason really. I suppose in my mind I was thinking it was an
    > unlikely safeguard but I liked the idea of adding so would just do it at
    > the end of a time step. In reality I think there is practically no
    > difference and this way it is done once, in a single location vs.
    > potential 10 separate checks? I don't see the advantage?


    You should always aim to fail as close as possible to the source of the
    error as is practical. That decreases the amount of debugging required
    when something fails: instead of searching your entire program, you only
    have to search a very small amount of code.



    --
    Steven
    Steven D'Aprano, Mar 19, 2011
    #12
  13. On Mar 19, 8:40 pm, Steven D'Aprano <steve
    > wrote:
    > On Sat, 19 Mar 2011 01:38:10 -0700, Martin De Kauwe wrote:
    > >> Why don't you do the range check *before* storing it in state? That way
    > >> you can identify the calculation that was wrong, instead of merely
    > >> noticing that at some point some unknown calculation went wrong.

    >
    > > I guess no reason really. I suppose in my mind I was thinking it was an
    > > unlikely safeguard but I liked the idea of adding so would just do it at
    > > the end of a time step. In reality I think there is practically no
    > > difference and this way it is done once, in a single location vs.
    > > potential 10 separate checks? I don't see the advantage?

    >
    > You should always aim to fail as close as possible to the source of the
    > error as is practical. That decreases the amount of debugging required
    > when something fails: instead of searching your entire program, you only
    > have to search a very small amount of code.
    >
    > --
    > Steven


    OK I take your point and can see the superior logic! I shall amend
    what I was planning
    Martin De Kauwe, Mar 20, 2011
    #13
  14. Martin De Kauwe wrote:
    >> Sorry, are you trying to say that it is not practical to write correct
    >> code that isn't buggy? Well, you're honest, at least, still I can't help
    >> but feel that you're admitting defeat before even starting.
    >>

    >
    > No. What I am saying is the code is written has been well tested and
    > *appears* to be working well. However the code is complicated and
    > there is potential for bugs. I think I am just been practical here,
    > evidently I can't think of everything, but there are some clear and
    > obvious errors that would be worth checking for. I can only explain
    > this in the terms of the code (sorry)...but for example the model
    > estimates plant photosynthesis and then allocates the carbon. So one
    > clear example is that the model cuts back carbon production if there
    > is water stress for the plant. This involves "removing" carbon from
    > the state. Clearly if you ended up in a situation where there is
    > negative carbon in a leaf, i.e. the leaf doesn't exist well this is
    > not physically possible and would be a code issue. Whilst this is
    > unlikely I think it would be nice to have a catch for it. Another
    > example would be the amount of soil water available to the plant,
    > again there can be zero but not negative soil water. It wouldn't be
    > meaningful. I hope that makes sense?
    >
    > thanks
    >

    Not that much. You'll spot bugs where negative numbers will be set to
    some attribute but what if 42 is put instead of 43 in one of your
    attribute ? Same consequence, it will mess up with your model but none
    of your check will spot that error.

    Try to identify those complicated functions you mentioned, and write
    unitary tests for them. Use a set of input parameters with the expected
    function return value. That way you'll be able to spot errors, whether
    the attribute is negative or not.

    JM


    PS : writing unitary tests takes time, a lot of time.
    Jean-Michel Pichavant, Mar 21, 2011
    #14
  15. On Mar 21, 9:43 pm, Jean-Michel Pichavant <>
    wrote:
    > Martin De Kauwe wrote:
    > >> Sorry, are you trying to say that it is not practical to write correct
    > >> code that isn't buggy? Well, you're honest, at least, still I can't help
    > >> but feel that you're admitting defeat before even starting.

    >
    > > No. What I am saying is the code is written has been well tested and
    > > *appears* to be working well. However the code is complicated and
    > > there is potential for bugs. I think I am just been practical here,
    > > evidently I can't think of everything, but there are some clear and
    > > obvious errors that would be worth checking for. I can only explain
    > > this in the terms of the code (sorry)...but for example the model
    > > estimates plant photosynthesis and then allocates the carbon. So one
    > > clear example is that the model cuts back carbon production if there
    > > is water stress for the plant. This involves "removing" carbon from
    > > the state. Clearly if you ended up in a situation where there is
    > > negative carbon in a leaf, i.e. the leaf doesn't exist well this is
    > > not physically possible and would be a code issue. Whilst this is
    > > unlikely I think it would be nice to have a catch for it. Another
    > > example would be the amount of soil water available to the plant,
    > > again there can be zero but not negative soil water. It wouldn't be
    > > meaningful. I hope that makes sense?

    >
    > > thanks

    >
    > Not that much. You'll spot bugs where negative numbers will be set to
    > some attribute but what if 42 is put instead of 43 in one of your
    > attribute ? Same consequence, it will mess up with your model but none
    > of your check will spot that error.
    >
    > Try to identify those complicated functions you mentioned, and write
    > unitary tests for them. Use a set of input parameters with the expected
    > function return value. That way you'll be able to spot errors, whether
    > the attribute is negative or not.
    >
    > JM
    >
    > PS : writing unitary tests takes time, a lot of time.


    Hi,

    Yes you make a good point. I think what I was trying to get across is
    there are certain scenarios which can't physically happen and they are
    straight forward to check for. I just thought it might be nice to add
    a check for these. However you make a good case and I definitely take
    your point. I guess I should look into unitary tests then (any
    suggestions?)

    ta.
    Martin De Kauwe, Mar 21, 2011
    #15
  16. Martin De Kauwe wrote:
    > On Mar 21, 9:43 pm, Jean-Michel Pichavant <>
    > wrote:
    >
    >> Martin De Kauwe wrote:
    >>
    >>>> Sorry, are you trying to say that it is not practical to write correct
    >>>> code that isn't buggy? Well, you're honest, at least, still I can't help
    >>>> but feel that you're admitting defeat before even starting.
    >>>>
    >>> No. What I am saying is the code is written has been well tested and
    >>> *appears* to be working well. However the code is complicated and
    >>> there is potential for bugs. I think I am just been practical here,
    >>> evidently I can't think of everything, but there are some clear and
    >>> obvious errors that would be worth checking for. I can only explain
    >>> this in the terms of the code (sorry)...but for example the model
    >>> estimates plant photosynthesis and then allocates the carbon. So one
    >>> clear example is that the model cuts back carbon production if there
    >>> is water stress for the plant. This involves "removing" carbon from
    >>> the state. Clearly if you ended up in a situation where there is
    >>> negative carbon in a leaf, i.e. the leaf doesn't exist well this is
    >>> not physically possible and would be a code issue. Whilst this is
    >>> unlikely I think it would be nice to have a catch for it. Another
    >>> example would be the amount of soil water available to the plant,
    >>> again there can be zero but not negative soil water. It wouldn't be
    >>> meaningful. I hope that makes sense?
    >>>
    >>> thanks
    >>>

    >> Not that much. You'll spot bugs where negative numbers will be set to
    >> some attribute but what if 42 is put instead of 43 in one of your
    >> attribute ? Same consequence, it will mess up with your model but none
    >> of your check will spot that error.
    >>
    >> Try to identify those complicated functions you mentioned, and write
    >> unitary tests for them. Use a set of input parameters with the expected
    >> function return value. That way you'll be able to spot errors, whether
    >> the attribute is negative or not.
    >>
    >> JM
    >>
    >> PS : writing unitary tests takes time, a lot of time.
    >>

    >
    > Hi,
    >
    > Yes you make a good point. I think what I was trying to get across is
    > there are certain scenarios which can't physically happen and they are
    > straight forward to check for. I just thought it might be nice to add
    > a check for these. However you make a good case and I definitely take
    > your point. I guess I should look into unitary tests then (any
    > suggestions?)
    >
    > ta.
    >


    if your python version < 2.7:
    http://pypi.python.org/pypi/unittest2

    http://docs.python.org/library/unittest.html

    This is the standard way of writing unitary tests in python (AKA pyunit)

    JM
    Jean-Michel Pichavant, Mar 21, 2011
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Casey Hawthorne
    Replies:
    21
    Views:
    860
    Roedy Green
    Jun 5, 2004
  2. Chris

    Array bounds checking

    Chris, Jul 5, 2005, in forum: Java
    Replies:
    5
    Views:
    772
  3. Casey Hawthorne
    Replies:
    16
    Views:
    1,195
  4. Jim Cook

    Macro argument bounds checking?

    Jim Cook, Sep 17, 2003, in forum: C Programming
    Replies:
    7
    Views:
    811
    Dave Thompson
    Sep 22, 2003
  5. Julian Zhang

    Bounds Checking?

    Julian Zhang, Jan 12, 2004, in forum: C Programming
    Replies:
    3
    Views:
    588
    E. Robert Tisdale
    Jan 20, 2004
Loading...

Share This Page