range() is not the best way to check range?

S

Steve Holden

Grant said:
Interesting point. Does the phrase "for pete's sake as god
intended" equate pete with god?

It's possible that pete is not god and yet god's intentions are
in pete's best interest, so something could be "for pete's
sake" and "as god intended" without pete being god.

That said, "for pete's sake" is probably a just an cleaned up
version of "for god's sake", so I probably did call pete god.
No, actually you called god pete ;-)

regards
Steve
 
S

Simon Forman

Nick said:
Sets are pretty fast too, and have the advantage of flexibility in
that you can put any numbers in you like

I know this is self-evident to most of the people reading this, but I
thought it worth pointing out that this is a great way to test
membership in range(lo, hi, step) without doing "the necessary
algebra".

i.e. n in set(xrange(0, 10000, 23)) ...


Peace,
~Simon
 
D

Diez B. Roggisch

Simon said:
I know this is self-evident to most of the people reading this, but I
thought it worth pointing out that this is a great way to test
membership in range(lo, hi, step) without doing "the necessary
algebra".

i.e. n in set(xrange(0, 10000, 23)) ...

No, its not. It works, but it works by no means faster than

n in range(0, 10000, 23)

Both need O(n), which is a bit slow compared to

(((n - 15) % 23) == 0 and n >= 15 and n < 10000)

that will run in O(1)

Diez
 
S

Simon Forman

Diez said:
No, its not. It works, but it works by no means faster than

n in range(0, 10000, 23)

Both need O(n), which is a bit slow compared to

(((n - 15) % 23) == 0 and n >= 15 and n < 10000)

that will run in O(1)

Diez

You're right, of course. I should have said that if you're testing
such a membership, say, 30000 times AND you (like me) are slightly
rusty on the algebra and prefer to let the computer do it, then you
could use something like:

test_set = set(xrange(0, 10000, 23))

if n in test_set:
...

;-)
~Simon
 
P

Paul Boddie

Grant said:
It's unclear what you're referring to as "the range".

The notion of something describing a range of values which can be
expanded to a list or, of relevance here, whose boundaries can be
tested efficiently.
Perhaps you're thinking of a slice? Somethign like

if (0:10000).contains(x):

Did you mean...?

(0:10000) # SyntaxError
slice(0, 10000).contains(x) # AttributeError
3 in slice(0, 10000) # TypeError

Something like this might suffice if slice could have a __contains__
method or if people thought of slices as natural things to test
against. Perhaps we could ask the original questioner why they chose to
use range in such a way - it might indicate a background in languages
which encourage the construction of ranges and their use in comparisons
- although being told to RTFM may have scared them off.

Paul
 
K

K.S.Sreeram

Simon said:
I know this is self-evident to most of the people reading this, but I
thought it worth pointing out that this is a great way to test
membership in range(lo, hi, step) without doing "the necessary
algebra".

i.e. n in set(xrange(0, 10000, 23)) ...

This is very very misleading... here are some timings :

python -mtimeit "n=5000" "n in set(xrange(0,10000))"
1000 loops, best of 3: 1.32 msec per loop

python -mtimeit "n=5000" "n in xrange(0,10000)"
1000 loops, best of 3: 455 usec per loop

python -mtimeit "n=5000" "0 <= n < 10000"
1000000 loops, best of 3: 0.217 usec per loop

sets are fast only if you create them *once* and use them again and
again. even in that case, the sets use up O(n) memory.

with comparison operators, you don't need extra memory *and* there is no
pre-computation required.

[sreeram;]


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEvRuPrgn0plK5qqURAiuQAJ4g1p7DW5LfNj+mlrXHwwXa6FAwRACdFpwu
KVnwhHLGp8sfn5J2qaW1eGA=
=+8z4
-----END PGP SIGNATURE-----
 
G

Grant Edwards

No, actually you called god pete ;-)

Well that's certainly wrong, because we all know god's name is
Howard.

Our father who art in heaven,
Howard be thy name,
....
 
G

Grant Edwards

The notion of something describing a range of values which can be
expanded to a list or, of relevance here, whose boundaries can be
tested efficiently.

I didn't mean to imply that would actually work, but I thought
maybe that's what you were proposing.
Did you mean...?

(0:10000) # SyntaxError
slice(0, 10000).contains(x) # AttributeError
3 in slice(0, 10000) # TypeError

Something like this might suffice if slice could have a __contains__
method or if people thought of slices as natural things to test
against.

A slice seems to me to be the obvious way to represent a finite
length algebraic sequence of integers. However, obvioiusness
is in the eye of the beholder since as you point out below to
the OP, a range() was the obvious way to it.
 
B

bearophileHUGS

Dan Bishop:
xrange already has __contains__. The problem is, it's implemented by a
highly-inefficient sequential search. Why not modify it to merely
check the bounds and (value - start) % step == 0?

I think this is a nice idea.

Bye,
bearophile
 
S

Simon Forman

K.S.Sreeram said:
This is very very misleading... here are some timings :

Yes it is. I'm sorry about that.
python -mtimeit "n=5000" "n in set(xrange(0,10000))"
1000 loops, best of 3: 1.32 msec per loop

python -mtimeit "n=5000" "n in xrange(0,10000)"
1000 loops, best of 3: 455 usec per loop

python -mtimeit "n=5000" "0 <= n < 10000"
1000000 loops, best of 3: 0.217 usec per loop

sets are fast only if you create them *once* and use them again and
again. even in that case, the sets use up O(n) memory.

That's what I meant. But I didn't state it clearly.

One of the things I like most about python is that it allows you to
specify the problem that you want to solve without a great deal of
difficulty as to *how* to specify it. To me, and perhaps others, "T =
set(xrange(0, 10000, 23))" and "n in T" are somewhat easier to read
and write than "not n % 23 and 0 <= n < 10000", YMMV.

In the given case a set of ~(10000 / 23) ints would not usually be too
burdensome on ram, and the code runs close to the same speed as
compared to the direct calculation:

from timeit import Timer

times = 100000
Max = 10000
n = 5000
T = set(xrange(0, Max, 23))

s1 = 'n in T'
s2 = 'not n %% 23 and 0 <= n < %s' % Max

setup = 'from __main__ import n, T'


S1 = Timer(s1, setup).repeat(number=times)
S2 = Timer(s2, setup).repeat(number=times)


print "%.3f usec/pass" % (1000000 * min(S1) / times)
print "%.3f usec/pass" % (1000000 * min(S2) / times)

On my machine this printed:
0.476 usec/pass
0.552 usec/pass

with comparison operators, you don't need extra memory *and* there is no
pre-computation required.

When I set Max = 100000000 in the above test code there was serious
disk thrashing... ;-)
[sreeram;]


FWIW, in production code I would certainly use the comparison
operators. A kilobyte saved is a kilobyte earned.

Peace,
~Simon
 
S

Summercoolness

it seems that range() can be really slow:

if i in range (0, 10000):


My original use was like this:

if i in range (iStart, iEnd):
listData.append(a)

in which iStart is 1000 and iEnd is 1008

so in that case, the program ran fine...
but later on, i wanted to include all data, so I relaxed the range by
setting iStart to 0 and iEnd to 9999 and later on i found that the
program was slow due to this.

So looks like the usage of

if sDay in ("Tue", "Wed", "Thu"):

is more like good use of "in a list" but in range(0,10000) will be a
big search in a list.
 
T

tac-tics

Simon said:
To me, and perhaps others, "T =
set(xrange(0, 10000, 23))" and "n in T" are somewhat easier to read
and write than "not n % 23 and 0 <= n < 10000", YMMV.

Eh? How is the first easier to read than the second?? You have a nested
function call in the first!

Regardless, testing if a member is part of a ranged set is always going
to be slower. It's the nature of what you're doing. Building a set and
then searching it takes much longer than a single modulus and
subtraction (which is all an integer comparison is).
 
J

John Machin

xrange already has __contains__.

As pointed out previously, xrange is a function and one would not expect
it to have a __contains__ method.

The objects returned by xrange do not (according to my reading of the
2.4.3 version of Objects/rangeobject.c) have a __contains__ method.

I find it difficult to believe that an inefficient __contains__ has been
implemented since.

Perhaps you are unaware that the mere fact that an object supports the
"in" operation does not mean that this support is provided by a
__contains__ method. The following section of the manual may help:

"""
The membership test operators (in and not in) are normally implemented
as an iteration through a sequence. However, container objects can
supply the following special method with a more efficient
implementation, which also does not require the object be a sequence.

__contains__( self, item)
Called to implement membership test operators. Should return true
if item is in self, false otherwise. For mapping objects, this should
consider the keys of the mapping rather than the values or the key-item
pairs.
"""
 
P

Paul Boddie

John said:
As pointed out previously, xrange is a function and one would not expect
it to have a __contains__ method.

Well, you pointed out that range is a function, but xrange seems to be
a type...
['__class__', '__delattr__', '__doc__', '__getattribute__',
'__getitem__', '__hash__', '__init__', '__iter__', '__len__',
'__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__',
'__setattr__', '__str__']

No __contains__ method, though, at least in 2.4.1.
The objects returned by xrange do not (according to my reading of the
2.4.3 version of Objects/rangeobject.c) have a __contains__ method.

As confirmed by the above evidence.
I find it difficult to believe that an inefficient __contains__ has been
implemented since.

So do I. As you go on to say, the usual sequence traversal mechanisms
are probably used to support the "in" operator. Whether it's a pressing
matter to add support for a more efficient mechanism depends on how
often people want to use ranges in the way described. Perhaps I'll
write a patch - who knows? ;-)

Paul
 
S

Simon Forman

tac-tics said:
Eh? How is the first easier to read than the second?? You have a nested
function call in the first!

I find the first form more immediately comprehensible than the latter.
I know what xrange() does, and I know what set() does, and "nested
function calls" give me no trouble, whereas the latter form with a
modulus, negation, and comparisons would take me a bit longer both to
compose and/or understand.

If this is not the case for you then by all means please disregard my
posting. YMMV.
Regardless, testing if a member is part of a ranged set is always going
to be slower.

Yes. Roughly 0.0000001 seconds slower on my five year old computer.
I'm not worried.
It's the nature of what you're doing. Building a set and
then searching it takes much longer than a single modulus and
subtraction (which is all an integer comparison is).

Building the set, yes, but searching the set is very close to the same
speed, even for rather large sets. If you were performing the search
30000 times (like in the OP) it would only take about three thousandths
of a second longer, and that's on my old slow computer.

If I were doing this a thousand times more often, or on a range of a
million or more, or in production code, or with ranges that changed
often, then I would certainly take the time to write out the latter
form.


Peace,
~Simon
 
D

Dan Bishop

Paul said:
John said:
As pointed out previously, xrange is a function and one would not expect
it to have a __contains__ method.

Well, you pointed out that range is a function, but xrange seems to be
a type...
['__class__', '__delattr__', '__doc__', '__getattribute__',
'__getitem__', '__hash__', '__init__', '__iter__', '__len__',
'__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__',
'__setattr__', '__str__']

No __contains__ method, though, at least in 2.4.1.
The objects returned by xrange do not (according to my reading of the
2.4.3 version of Objects/rangeobject.c) have a __contains__ method.

As confirmed by the above evidence.
I find it difficult to believe that an inefficient __contains__ has been
implemented since.

So do I. As you go on to say, the usual sequence traversal mechanisms
are probably used to support the "in" operator. Whether it's a pressing
matter to add support for a more efficient mechanism depends on how
often people want to use ranges in the way described. Perhaps I'll
write a patch - who knows? ;-)

My mistake. I should have looked at dir(xrange) before posting.

But the point remains that xrange's "implicit __contains__" runs in
linear time when a constant-time algorithm exists.
 
A

Alex Martelli

Paul Boddie said:
Well, range is a function in the current implementation, although its
usage is similar to that one would get if it were a class, particularly
a subclass of list or one providing a list-style interface. With such a
class, you could provide a __contains__ method which could answer the
question of what the range contains based on the semantics guaranteed
by a range (in contrast to a normal list).

You'd also have to override just about every mutating method to switch
back to a "normal" __contains__ (or change self's type on the fly) -- a
pretty heavy price to pay.

I have often noticed that subclassing list, dict and maybe set has this
kind of issue: the need to track every possible change to the object.

Maybe a good mechanism to have for the purpose would be to add to
mutable types a "hook" method, say __mutator__, which gets called either
right before or right after any mutating method (there are different
tradeoffs for before-calls and after-calls), presumably passing along
the *a and **k for generality (although it might be faster for the base
case to avoid that); the base types would have a no-op implementation,
but subtypes could easily override just the hook to facilitate their
task of maintaining extra state (could be as little as a per-instance
flag recording whether the object is guaranteed to be still "pristine").
At C level, that might be an extra slot tp_mutator, left NULL in base
types to indicate "no mutator-hook method implemented here".

Like any other addition of, or change to, functionality, this would of
course be a proposal for 2.6, since 2.5 is feature-frozen now.


Alex
 
P

Paul Boddie

Alex said:
You'd also have to override just about every mutating method to switch
back to a "normal" __contains__ (or change self's type on the fly) -- a
pretty heavy price to pay.

A subclass of list is probably a bad idea in hindsight, due to various
probable requirements of it actually needing to be a list with all its
contents, whereas we wanted to avoid having anything like a list around
until the contents of this "lazy list" were required by the program. If
we really wanted to subclass something, we could consider subclassing
the slice class/type, but that isn't subclassable in today's Python for
some reason, and it doesn't really provide anything substantial,
anyway. However, Python being the language it is, an appropriately
behaving class is quite easily written from scratch.

Paul
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top