Question about idioms for clearing a list

T

Tim Hochberg

I am confused. Could you explain this ? I was under the impression said
above(mapping don't support slicing), until after I read the language
reference. I don't think it is slicing as in the list slicing sense but
it does use the term "extend slicing".

http://www.python.org/doc/2.4.2/ref/slicings.html

"The semantics for an extended slicing are as follows. The primary must
evaluate to a mapping object, and it is indexed with a key that is
constructed from the slice list, as follows. If the slice list contains
at least one comma, the key is a tuple containing the conversion of the
slice items; otherwise, the conversion of the lone slice item is the
key. The conversion of a slice item that is an expression is that
expression. The conversion of an ellipsis slice item is the built-in
Ellipsis object. The conversion of a proper slice is a slice object
(see section 3.2) whose start, stop and step attributes are the values
of the expressions given as lower bound, upper bound and stride,
respectively, substituting None for missing expressions."


This is in place to support multidimensional arrays, such as in numpy.
If, for instance, you have a 9x9 array A, then A[3:6,3:6] will extract a
3x3 block from the center of it. A[3:6,3:6] is equivalent to
A[(slice(3,6,None), slice(3,6,None))] and the resulting tuple gets
passed through the mapping interface, but it is not a mapping in any
real sense.

I don't think there's anything in core Python that uses this and it's
not really relevant to this thread.

-tim
 
B

bonono

Tim said:
I am confused. Could you explain this ? I was under the impression said
above(mapping don't support slicing), until after I read the language
reference. I don't think it is slicing as in the list slicing sense but
it does use the term "extend slicing".

http://www.python.org/doc/2.4.2/ref/slicings.html

"The semantics for an extended slicing are as follows. The primary must
evaluate to a mapping object, and it is indexed with a key that is
constructed from the slice list, as follows. If the slice list contains
at least one comma, the key is a tuple containing the conversion of the
slice items; otherwise, the conversion of the lone slice item is the
key. The conversion of a slice item that is an expression is that
expression. The conversion of an ellipsis slice item is the built-in
Ellipsis object. The conversion of a proper slice is a slice object
(see section 3.2) whose start, stop and step attributes are the values
of the expressions given as lower bound, upper bound and stride,
respectively, substituting None for missing expressions."


This is in place to support multidimensional arrays, such as in numpy.
If, for instance, you have a 9x9 array A, then A[3:6,3:6] will extract a
3x3 block from the center of it. A[3:6,3:6] is equivalent to
A[(slice(3,6,None), slice(3,6,None))] and the resulting tuple gets
passed through the mapping interface, but it is not a mapping in any
real sense.

I don't think there's anything in core Python that uses this and it's
not really relevant to this thread.
Thanks. I would say that it is not relevant to the OP's question but
the thread has turned to "anyone who read the doc should know about
slicing" and that prompted me to go and read the doc(I don't have the
OP's mentioned need in my coding so far and never read about the full
pontential of slicing, just use it as the right hand side intuitively)
about slicing and found the documentation to be a bit lacking(only a
brief mentioning of slicing in the turtorial and this confusing
section).

Beside, just from reading this page, it seems that a[start:stop:stride]
is not a valid construction for sequence object 'a'.

To me, reading the doc makes me more confuse about what is slicing.
 
M

Magnus Lycka

Ed said:
I'm a fairly average programmer (better than average compared to my
immediate colleagues). I've read every tutorial I can get my hands
on, but I have no _memory_ of ever coming across the del keyword, let
alone that it is fundamental to Python, and I have no idea what
collections school is. I doubtless have read of it at some point, but
as no importance has ever been attached to it, I have probably not
remembered it.

Similarly, I remember slices simply because they are handy, not
because I have ever heard of them being fundamental before.

Ok, I can understand that, but I think that you really understand
that the strength of a programming language such as Python is that
it's like lego bricks. You have some basic pieces, and you can
combine them to into something unique that does what you want.

There are plenty of statements, operators, functions, types, modules
and other things in Python already. I can well imagine that you had
forgotten about del, and that you don't immediately think about slices
when you wonder how to empty a list. It's like when I build lego with
my son. I guess he has around 2000 pieces, and it's not always easy
to spot what you need. It was difficult enough when I was a kid. Now
there are so many different kinds of pieces, shaped to fulfil some
niche usecase.

One thing that I'm sure of is this: Making more kinds of odd-shaped
"pieces", especially prepared to solve the specific problem I'm facing
right now, won't make it easier to understand or use Python in the
long run.

I've used Python for almost 10 years now, and I still learn new
things, and I sometimes come across things that I once new but
had forgotten.

It might work for a while to add a new convenience function as soon
as someone finds that they don't immediately now how to solve a
certain problem. It's my impression that that's pretty much the idea
with PHP. It's not Python though. PHP is only successful in a fairly
narrow (if important) niche, it has failed in getting used outside
its niche, and I assume we'll see a decline in its use one the web
pretty soon, just as it happened with Perl. (Whether RoR, something
Python based or something entirely new will replace it is beyond my
radar screen though.)
 
M

Magnus Lycka

Ed said:
Is it obvious to a newbie what the difference between mappings and
"not-mappings", and is it obvious exactly what is and isn't a mapping?

Should it be necessary to "know" python before it becomes easy to use?

QOTW! (You are joking, aren't you? :)

I can undestand how people can turn a bit defensive when some
people who are more skilled in programming than in diplomacy
basically tells the confused that they are ignorant and should
just learn the language.

On the other hand, I think that Art Siegel said it very well:
"My ability to grok Python to any extent, from that starting point,
was based on an understanding that it was my responsibility to come
to Python, not it to me."
(Actually, that's a much better QOTW! It gives me flashbacks from
a book I once read, http://www.amazon.com/gp/product/0060958324 )

There are just a few statements in Python. Some of them should
really have been functions (print and exec) but anyway, they are
still few enough to learn. There are also a few ways to get inside
composite objects, x.y, x[y], x[y:z]. These things are among the
most fundamental in Python.

If luminaries like Fredrik Lundh and Raymond Hettiger tells you
that things should be done in a certain way, it's really just
silly to argue further. I've programmed Python since 1996, but if
these guys tell me I'm wrong, I won't bother to argue. If I don't
understand their position, I'll try to study the subject further,
and I might ask them to clarify, but I'll assume that they are
right. That assumption has worked so far for me.

There are warts and quirks in Python, everybody will agree to that,
but Python is *not* Perl. A basic motto has always been to avoid
synonyms, to try to provide as few ways to do one thing, rather
than to provide as many ways, as possible. This has proven very
successful in making Python easy to learn and use. The opposite
approach has made Perl into the favourite programimng language
among people who think that "if the program was difficult to write,
it should be difficult to read as well!"

When you can't transfer one approach from one type to another,
there is a reason for that. A few times it might be due to some
obscure practical aspect of the implementation, but most of the
time, it's completely intentional, and finding these aspects of
Python, learning why they are the way they are, and embracing the
language rather than trying to fight it, is actually very rewarding.

Tuples aren't just immutable lists--they have different purposes
beyond that. Ordereded and unordered collections are conceptually
different. Neither call-by-referece nor call-by-value describes
parameter passing in Python well, the clue lies in understanding
how assignments and objects work, and you need to understand the
difference between mutable and immutable objects etc. There are
a number of fundamental concepts that you need to understand to
really use Python well.

Python works very smoothly, and you can do a lot of productive
work with it even if you don't know these things, but in time
you'll trip over them, and as you do, you need to grok these
concepts to get further. Asking for some syntactic change or
some convenience method so that you can get a little further
without understanding the fundamental concept isn't the way to
get beyond these little stumbling blocks.

It's a bit as if you call the design department of your car
manufacturer and tell them how they should redesign the suspension
since it was so bumpy when you drove around with flat tires.
Saying that it's too much to ask that you keep the tires filled
with the right amount of air won't meet much sympathy.
 
S

Simon Brunning

you seem to be missing that we're talking about a programming language
here. nothing is obvious if you don't know anything about the language;
a lot of things are obvious once you've learned a little about it. a little is
all it takes. why is that so hard to understand ?

"The only 'intuitive' user interface is the nipple. After that, it's
all learned." - Bruce Ediger
 
E

Ed Singleton

Ok, I can understand that, but I think that you really understand
that the strength of a programming language such as Python is that
it's like lego bricks. You have some basic pieces, and you can
combine them to into something unique that does what you want.

There are plenty of statements, operators, functions, types, modules
and other things in Python already. I can well imagine that you had
forgotten about del, and that you don't immediately think about slices
when you wonder how to empty a list. It's like when I build lego with
my son. I guess he has around 2000 pieces, and it's not always easy
to spot what you need. It was difficult enough when I was a kid. Now
there are so many different kinds of pieces, shaped to fulfil some
niche usecase.

One thing that I'm sure of is this: Making more kinds of odd-shaped
"pieces", especially prepared to solve the specific problem I'm facing
right now, won't make it easier to understand or use Python in the
long run.

I agree utterly with this, particularly the general philosophy of
having simple bricks that work in as many different places as
possible.

The point is that having to use del to clear a list appears to the
inexperienced as being an odd shaped brick when they've already used
the .clear() brick in other places.

Having bricks that work in lots of places makes the language
'guessable'. "I've never cleared a list before, but I've cleared
dictionaries and I guess the same way would work here".

The problem is you have to be very careful when talking about this,
not to use the C-word, because that's the hobgoblin of little minds,
whereas in almost every other field it is considered an important part
of usability.
 
F

Fredrik Lundh

Ed said:
Having bricks that work in lots of places makes the language
'guessable'. "I've never cleared a list before, but I've cleared
dictionaries and I guess the same way would work here".

f = open("foo")
f.clear()

sys.stdout.clear()

os.getcwd().clear()

shelve.clear()

s = "sky"
s.clear()

</F>
 
E

Ed Singleton

QOTW! (You are joking, aren't you? :)

It was a genuine question. Judging from one of your other responses,
that you are still learning new things and remembering forgotten ones
after 10 years, I assume the answer is "no, it is not necessary".
(I'm also assuming you find Python easy to use).

The thing that first attracted me to Python was that I only had to
read a page or two of the tutorial before I could get started on doing
little things with Python and find it easy.
I can undestand how people can turn a bit defensive when some
people who are more skilled in programming than in diplomacy
basically tells the confused that they are ignorant and should
just learn the language.

On the other hand, I think that Art Siegel said it very well:
"My ability to grok Python to any extent, from that starting point,
was based on an understanding that it was my responsibility to come
to Python, not it to me."
(Actually, that's a much better QOTW! It gives me flashbacks from
a book I once read, http://www.amazon.com/gp/product/0060958324 )

There are just a few statements in Python. Some of them should
really have been functions (print and exec) but anyway, they are
still few enough to learn. There are also a few ways to get inside
composite objects, x.y, x[y], x[y:z]. These things are among the
most fundamental in Python.

If luminaries like Fredrik Lundh and Raymond Hettiger tells you
that things should be done in a certain way, it's really just
silly to argue further. I've programmed Python since 1996, but if
these guys tell me I'm wrong, I won't bother to argue. If I don't
understand their position, I'll try to study the subject further,
and I might ask them to clarify, but I'll assume that they are
right. That assumption has worked so far for me.

I didn't know they were luminaries. I apologise. I've never heard of
any of the people on the list before I came to Python and the only
people I've learned to trust are Kent Johnson and Alan Gauld. I guess
if I start to see those guys say things that turn out to be right I
might start assuming they are right as well.

Maybe there should be a list of luminaries on the website so we know
who not to argue with?
There are warts and quirks in Python, everybody will agree to that,
but Python is *not* Perl. A basic motto has always been to avoid
synonyms, to try to provide as few ways to do one thing, rather
than to provide as many ways, as possible. This has proven very
successful in making Python easy to learn and use. The opposite
approach has made Perl into the favourite programimng language
among people who think that "if the program was difficult to write,
it should be difficult to read as well!"

I think people concentrate on the total number of ways to do
something, when the obviousness or consistency of one one of those
ways is more important.

For example one of the few things Microsoft does okay at, is in having
a pretty consistent interface across their applications. There's a
menu bar that (theoretically) has all the commands you can perform in
a structured format. But you can also use reasonably consistent
keyboard shortcuts to do something, or context menus for things that
you do a lot.

Having several different ways means that everyone can use the way they
prefer. Beginners tend to use the menu bar a lot as they can always
find what they want there, but can start using the keyboard shortcuts
as they start to perform the action a lot and start to become more
experienced.

I know this isn't entirely applicable but hopefully you see the analogy.
When you can't transfer one approach from one type to another,
there is a reason for that. A few times it might be due to some
obscure practical aspect of the implementation, but most of the
time, it's completely intentional, and finding these aspects of
Python, learning why they are the way they are, and embracing the
language rather than trying to fight it, is actually very rewarding.

No, it's rewarding for a certain type of person. It's not a rewarding
activity for every type of person. I find it deeply frustrating when
I have to constantly look things up to see what is the way of doing
things for this type. I'd rather spend my time writing code then
looking things up to see which way I have to do it now.
Tuples aren't just immutable lists--they have different purposes
beyond that. Ordereded and unordered collections are conceptually
different. Neither call-by-referece nor call-by-value describes
parameter passing in Python well, the clue lies in understanding
how assignments and objects work, and you need to understand the
difference between mutable and immutable objects etc. There are
a number of fundamental concepts that you need to understand to
really use Python well.

Python works very smoothly, and you can do a lot of productive
work with it even if you don't know these things, but in time
you'll trip over them, and as you do, you need to grok these
concepts to get further. Asking for some syntactic change or
some convenience method so that you can get a little further
without understanding the fundamental concept isn't the way to
get beyond these little stumbling blocks.

It's a bit as if you call the design department of your car
manufacturer and tell them how they should redesign the suspension
since it was so bumpy when you drove around with flat tires.
Saying that it's too much to ask that you keep the tires filled
with the right amount of air won't meet much sympathy.

It probably wouldn't meet with much sympathy, but that's not to say
it's not a decent point for someone to make.

Inflatable tyres are getting quite antiquated now. There's plenty of
substances that would be better to fill tyres with than air. there
are already tyres that don't burst when you push a 6-inch nail into
them because they are filled with a hard foam. They work better,
don't puncture, last longer, but hey is it too much to ask for you to
keep your tyres filled with air and replace them every time you get a
puncture and carry a fifth wheel around in case you get a puncture and
there's no way to fix the tyre?

Ed
 
S

Scott David Daniels

Magnus said:
... I sometimes come across things that I once new but had forgotten.

I'm sorry, and I mean no offense, _but_ I think _new_ there is a
lovely typo. :) I stopped, corrected it in my head, proceeded,
and then I backed up, put it back and laughed out loud.

--Scott David Daniels
(e-mail address removed)
 
M

Magnus Lycka

A slim lady in a brown overcoat appears and says with a silly
French accent: "Lizten very carefully, I vill zay ziz only onze."

BTW, I happen to reply to Ed's post now, but I'm really responding
to a fairly common attitude which I find somewhat counterproductive.
I hope people are openminded and able to see the point I'm trying
to make.

Ed said:
The point is that having to use del to clear a list appears to the
inexperienced as being an odd shaped brick when they've already used
the .clear() brick in other places.

Agreed. The smart way to go from this stage of surprise is
not to assume that Python is broken, but to try to understand
how lists are different from e.g. dicts, and why the so-much-
smarter-than-me Python designers made it like this. Hubris is
considered a virtue in the Perl community. While not mentioned
so much, I'd say that humility is considered a virtue here.

Not that Python is perfect, but when you don't get a
"sorry, this change would break existing code, won't happen
until Python 3.0"-resonse, but a "study this more"-response,
the smart thing is to open your mind and try to fully grok
this.

There is a very strong ambition among the Python developers
to avoid duplication of features. This is one of the keys to
Python's ease of use and readability. Don't bother suggesting
synonyms. While there are thousands of situations where adding
just another method would make life easier in the short run,
life would be much harder if there were thousands of extra
methods in the Python core!

It isn't always possible to use one approch to a problem in all
cases. If two approaches are used, they don't usually overlap.
The "extra" approach is only used where the normal approach
doesn't work.
Having bricks that work in lots of places makes the language
'guessable'. "I've never cleared a list before, but I've cleared
dictionaries and I guess the same way would work here".

I think we both agree that Python is very useful in this regard.
It's more consistent than other languages I've worked with, and
when things seems inconsistent, that's probably deliberate, and
the appearent lack of consistency is a hint that we need to grok
how these cases are different.

You really happen to arrive at this from the wrong direction. :)
The .clear() methods in dicts and sets are there because there
is no other convenient way to empty these containers. There is
no support in these kinds of containers to refer to the whole
contents without copying. There is no <something> which lets
you do "del aDict[<something>]" to clean a dict. You can do
"for key in aDict: del aDict[key]", but since this is a fairly
common thing to do, there is a shortcut for that called .clear().
I'm pretty sure the reason to implement it was speed rather than
convenience. As you know, "There should be one-- and preferably
only one --obvious way to do it." "Although practicality beats
purity." In other words, moving a very common loop from Python
to C was more important than a minimal interface. Don't forget
to "import this" and ponder a bit...

Statements and operators are really fundamental in Python. We
don't support n = 1.add(2), since we have the '+' operator.
You can't do x.clear() to remove arbitrary variables x from a
namespace, but del x works for every x, whether it's a dict,
a module or an int etc. Python basically just use methods when
the statements and operators aren't enough.

To really understand more about this, it might be useful to
ask why we can't use del to somehow empty dicts instead. In
the best case, that might possibly lead to the removal or at
least deprecation of the .clear() method in Python 3.0, but I
doubt that people would find some way that had the required
beauty.
The problem is you have to be very careful when talking about this,
not to use the C-word, because that's the hobgoblin of little minds,
whereas in almost every other field it is considered an important part
of usability.

That hobgoblin-phrase isn't very helpful. First of all, not all
who respond to questions at comp.lang.python know what they are
talking about. This is the internet. Take everything you read with
a grain of salt. But another thing I've learnt in my years as a
consultant, is that when experienced people directly and strongly
reject an idea, they usually do that based on intuition and
experience, and even if they are arguing poorly for their case,
it's in your best interest to try to understand why they react
like they do, or you'll fall down in that tar pit they were trying
to warn you about. It's like being in a foreign country and meeting
someone who is waving his arms and shouting incomprehensibly to
you. Just because you don't understand what he's saying, you shouldn't
assume that he's just talking jibberish and can be safely ignored.

The smart approach is neither to stubbornly repeat your already
rejected idea, nor to try to crush the arguments of those who
oppose you, but rather to openly and humbly try your best to see
their side of this. Don't just accept it. Try to figure out why
these really bright people have taken this standpoint. They've
probably heard your question 20 times before, and might be bored
to go through all the arguments again.

(Part of this is to figure out who are the guys you should really
listen to. If people are allowed to commit code to the Python code
base, that's a hint that they know what they are talking about...
http://sourceforge.net/project/memberlist.php?group_id=5470
In other words, you cn safely ignore me... ;^)

I'm sure Python can and will be improved in various ways, and that
some people who are fairly new to Python might come up with bright
ideas. I also suspect that those old farts (like me) who have used
it for ages might respond in a stubborn and reactionary way. That
doesn't change the fact that the overwhelming amount of
comp.lang.python threads about flaws in Python and suggestions for
improvements in the core would make Python worse rather than better
if they were implemented.

I know that the last time (the one time) I claimed that Python had
a really bizarre bug, it turned out that I was just being blind and
sloppy.
 
T

Terry Reedy

I am confused. Could you explain this ? I was under the impression said
above(mapping don't support slicing), until after I read the language
reference. I don't think it is slicing as in the list slicing sense but
it does use the term "extend slicing".

http://www.python.org/doc/2.4.2/ref/slicings.html

I believe that extended slicing was introduced for multidimensional slicing
of multidimensional arrays in Numerical Python. Apparently, since such
arrays are not indexed sequentially, they are considered as mappings. The
paragraph on extended slicing is much clearer if one knows how they are
used in one of the numerical Python packages.

In the mathematical sense, a sequence can be considered to be a mapping of
counts to whatever. In some languages, sequences have been implemented
literally as a mapping by 'associative arrays', the equivalent of Python's
dict. And dicts are sometimes used in Python to implement sparse arrays.

A set is not a mapping unless its members are all ordered pairs, as in a
dict.

Terry Jan Reedy
 
T

Terry Reedy

Magnus Lycka said:
Statements and operators are really fundamental in Python. We
don't support n = 1.add(2), since we have the '+' operator.

Actually, since the 2.2 union of type and new-style classes, which gave
attributes to all types ....
3.0

I only see this as useful for making bound methods:
4

which is certainly nicer than the older method of writing a 'makeoper'
function that returned a nested function with either a default param or a
closure.

Terry Jan Reedy
 
B

Bryan Olson

Magnus said:
Agreed. The smart way to go from this stage of surprise is
not to assume that Python is broken, but to try to understand
how lists are different from e.g. dicts, and why the so-much-
smarter-than-me Python designers made it like this.

That two of Python's three built-in mutable collections support
clear() is clearly a historical artifact.
Not that Python is perfect, but when you don't get a
"sorry, this change would break existing code, won't happen
until Python 3.0"-resonse, but a "study this more"-response,
the smart thing is to open your mind and try to fully grok
this.

The original question was about idioms and understanding, but
there's more to the case for list.clear. Python is "duck typed".
Consistency is the key to polymorphism: type X will work as an
actual parameter if and only if X has the required methods and
they do the expected things.

Emptying out a collection is logically the same thing whether
that collection is a list, set, dictionary, or user-defined
SortedBag. When different types can support the same operation,
they should also support the same interface. That's what
enables polymorphism.
 
M

Magnus Lycka

Bryan said:
The original question was about idioms and understanding, but
there's more to the case for list.clear. Python is "duck typed".
Consistency is the key to polymorphism: type X will work as an
actual parameter if and only if X has the required methods and
they do the expected things.

Emptying out a collection is logically the same thing whether
that collection is a list, set, dictionary, or user-defined
SortedBag. When different types can support the same operation,
they should also support the same interface. That's what
enables polymorphism.

I agree that emptying is logically the same thing for all of these
types. Beyond that, they don't seem to have a lot in common. It's
quite possible to support a duck typing approach that works for all
sorts of sequences, but it's fairly meaningless to use ducktyping
for conceptually different types such as dicts, lists and sets.

Do you really have a usecase for this? It seems to me that your
argument is pretty hollow.

For lists, which are mutable sequences, you add new data with .insert,
..append or .extend. You replace or remove existing data using indexing
l[x] or slicing l[x:y] in del or assignment statements. You can also
remove data with .pop or .remove. These overlapping methods have
specific use. l.remove(x) is short for del l[l.index(x)] (it's also
faster, and that sometimes matter) and .pop() is there to support a
stack-like behaviour.

Dicts use indexing d[x] or .update to either add new data or to replace
existing data. There is no distinction between these operations in
dicts, since dicts are semantically so different from sequences. They
have content, but no order, no specific "positions". You can delete
one item at a time with del d[x], but since slices don't make sense
for dicts, there is a d.clear() method to achieve this common task
quickly. One could imagine that it was allowed to write "del d[]" or
something like that, but then we also expect x = d[] and d[] = x to
work... We probably don't want that.

Sets are also semantically different, and thus use a different set of
operations. They share the lack of order with dicts, but they aren't
pairs, so the semantics is naturally different. They don't support
indexing at all, since they neither have order nor keys.

As far as I understand, the only operation which is currently used
by all three collections is .pop, but that takes a different number
or parameters, since these collections are conceptually different!

Then we have Queues. They have a different purpose, and again, a
different API, since they provide features such as blocking or non-
blocking reads and writes.
 
B

Bryan Olson

Magnus said:
I agree that emptying is logically the same thing for all of these
types. Beyond that, they don't seem to have a lot in common. It's
quite possible to support a duck typing approach that works for all
sorts of sequences, but it's fairly meaningless to use ducktyping
for conceptually different types such as dicts, lists and sets.

Do you really have a usecase for this? It seems to me that your
argument is pretty hollow.

Sure:

if item_triggering_end in collection:
handle_end(whatever)
collection.clear()


Or maybe moving everything from several collections into
a single union:

big_union = set()
for collection in some_iter:
big_union.update(t)
collection.clear()



[...]
As far as I understand, the only operation which is currently used
by all three collections is .pop, but that takes a different number
or parameters, since these collections are conceptually different!

The all support len, iteration, and membership test.

Many algorithms make sense for either sets or lists. Even if they
cannot work on every type of collection, that's no reason not
to help them be as general as logic allows.
 
A

Alex Martelli

Bryan Olson said:
Sure:

if item_triggering_end in collection:
handle_end(whatever)
collection.clear()

Or maybe moving everything from several collections into
a single union:

big_union = set()
for collection in some_iter:
big_union.update(t)
collection.clear()

I was thinking of something different again, from a use case I did have:

def buncher(sourceit, sentinel, container, adder, clearer):
for item in sourceit:
if item == sentinel:
yield container
clearer()
else
adder(item)
yield container

s = set()
for setbunch in buncher(src, '', s, s.add, s.clear): ...

d = dict()
for dictbunch in buncher(src, '', d, lambda x: d.setdefault(x,''),
d.clear): ...

L = list()
for listbunch in buncher(src, '', L, L.append,
lambda: L.__setslice__(0,len(L),[])): ...

the dict case is semi-goofy (since some arbitrary value must be set, one
ends up with a lambda willy nilly), but the list case is even worse,
with that horrid "lambda calling __setslice__" (eek).

BTW, we do have other mutable collections...:

import collections
q = collections.deque()
for qbunch in buncher(src, q, q.append, q.clear): ...

just as neat as a set.

So what is the rationale for having list SO much harder to use in such a
way, than either set or collections.deque? (For dict, I can see the
rationale for not having an 'addkey', even though the presence of class
method 'fromkeys' weakens that rationale... but for list, I cannot see
any reason that makes sense to me).


Alex
 
R

Raymond Hettinger

[Alex Martelli]
I was thinking of something different again, from a use case I did have:

def buncher(sourceit, sentinel, container, adder, clearer):
for item in sourceit:
if item == sentinel:
yield container
clearer()
else
adder(item)
yield container

s = set()
for setbunch in buncher(src, '', s, s.add, s.clear): ...

I'm curious, what is the purpose of emptying and clearing the same
container? ISTM that the for-loop's setbunch assignment would then be
irrelevant since id(setbunch)==id(s). IOW, the generator return
mechanism is not being used at all (as the yielded value is constant
and known in advance to be identical to s).

Just for jollies, I experimented with other ways to do the same thing:

from itertools import chain, groupby

def buncher(sourceit, sentinel, container, updater, clearer):
# Variant 1: use iter() to do sentinel detection and
# use updater() for fast, high volume updates/extensions
it = iter(src)
for item in it:
updater(chain([item], iter(it.next, sentinel)))
yield container
clearer()

s = set()
for setbunch in buncher(src, '', s, s.update, s.clear):
print setbunch, id(setbunch)

def buncher(sourceit, sentinel, container, updater, clearer):
# Variant 2: use groupby() to the bunching and
# use updater() for fast, high volume updates/extensions
for k, g in groupby(sourceit, lambda x: x != sentinel):
if k:
updater(g)
yield container
clearer()

s = set()
for setbunch in buncher(src, '', s, s.update, s.clear):
print setbunch

Of course, if you give-up the seemingly unimportant in-place update
requirement, then all three versions get simpler to implement and call:

def buncher(sourceit, sentinel, constructor):
# Variant 3: return a new collection for each bunch
for k, g in groupby(sourceit, lambda x: x != sentinel):
if k:
yield constructor(g)

for setbunch in buncher(src, '', set):
print setbunch

Voila, the API is much simpler; there's no need initially create the
destination container; and there's no need for adaptation functions
because the constructor API's are polymorphic:

constructor = list
constructor = set
constructor = dict.fromkeys



[Alex]
d = dict()
for dictbunch in buncher(src, '', d, lambda x: d.setdefault(x,''),
d.clear): ...

L = list()
for listbunch in buncher(src, '', L, L.append,
lambda: L.__setslice__(0,len(L),[])): ...

Hmm, is your original buncher a candidate for adapters? For instance,
could the buncher try to adapt any collection input to support its
required API of generic adds, clears, updates, etc.?



[Alex]
So what is the rationale for having list SO much harder to use in such a
way, than either set or collections.deque?

Sounds like a loaded question ;-)

If you're asking why list's don't have a clear() method, the answer is
that they already had two ways to do it (slice assignment and slice
deletion) and Guido must have valued API compactness over collection
polymorphism. The latter is also evidenced by set.add() vs
list.append() and by the two pop() methods having a different
signatures.

If you're asking why your specific case looked so painful, I suspect
that it only looked hard because the adaptation was force-fit into a
lambda (the del-statement or slice assignment won't work as an
expression). You would have had similar difficulties embedding
try/except logic or a print-statement. Guido, would of course
recommend using a plain def-statement:

L = list()
def L_clearer(L=L):
del L[:]
for listbunch in buncher(src, '', L, L.append, L_clearer):
print listbunch



While I question why in-place updating was needed in your example, it
did serve as a nice way to show-off various approaches to adapting
non-polymorphic API's for a generic consumer function with specific
needs.

Nice post,


Raymond
 
B

bonono

Raymond said:
[Alex]
So what is the rationale for having list SO much harder to use in such a
way, than either set or collections.deque?

Sounds like a loaded question ;-)

If you're asking why list's don't have a clear() method, the answer is
that they already had two ways to do it (slice assignment and slice
deletion) and Guido must have valued API compactness over collection
polymorphism. The latter is also evidenced by set.add() vs
list.append() and by the two pop() methods having a different
signatures.
Sounds to me that it is a preference(style, whatever), rather than some
other posts of this thread argued that "del L[:]" is better.
If you're asking why your specific case looked so painful, I suspect
that it only looked hard because the adaptation was force-fit into a
lambda (the del-statement or slice assignment won't work as an
expression). You would have had similar difficulties embedding
try/except logic or a print-statement. Guido, would of course
recommend using a plain def-statement:

L = list()
def L_clearer(L=L):
del L[:]
for listbunch in buncher(src, '', L, L.append, L_clearer):
print listbunch
Is that really "clearer" ? While it is still very localized(just read a
few lines up for the definition), buncher(src, '', L.append, L.clear)
seems to be clearer to me, especially there are two similar construct
on set/dict above, even the Alex's lambda form conveys more info, IMO.
Usng the new partial function, may be it can be written as :

buncher(src.'', L, L.append, partial(L.__delslice__, 0, sys.maxint))

#assuming list can have at at most maxint items.
 
M

Magnus Lycka

Bryan said:
Sure:

if item_triggering_end in collection:
handle_end(whatever)
collection.clear()

Or maybe moving everything from several collections into
a single union:

big_union = set()
for collection in some_iter:
big_union.update(t)
collection.clear()

I don't understand the second one. Where did 't' come from?
Anyway, tiny code snippets are hardly usecases. Are these from
real code? If they are, why aren't there support for emptying
lists? Have you patched your Python? Didn't you actually need
to support lists?

I still don't see any convincing usecase for the kind of
ducktyping you imply. There are certainly situations where
people have used lists or dicts before there were sets in
Python, and want to support both variants for a while at least,
but since their APIs are so differnt for these types, .clear()
seems like a non-issue.

If this was a problem in the real world, I bet we'd see a lot
of code with functions similar to this:

def clear(container):
try:
del container[:]
except TypeError:
container.clear()

If you *do* have this problem, this is a very simple workaround.
The all support len, iteration, and membership test.

Ok. Forgot that. id(), str() and repr() as well. Still, after almost
10 years of Python programming I can't remember that I ever ran into
a situation where I ever needed one single piece of code to empty
an arbitrary container. It's trivial to solve, so I wouldn't have
stopped to think about it for even a minute if it happened, but I
still don't think it happened.This was never a problem for me, and
I don't think I saw anyone else complain about it either, and I've
seen plenty of complaints! ;)

I can understand the argument about making it easy to remember how
to perform an action. I think the current situation is correct. To
introduce redunancy in this case (del x[:] <==> x.clear()) would not
be an improvement of Python. In the long run, such a strategy of
synonyms would make Python much more like Perl, and we don't want
that. So I can understand that the question pops up though (but
not why it gets such proportions).

I don't buy this duck-typing argument though. Considering how little
it would change in unifying these divergent APIs, it still sounds
as hollow to me.
Many algorithms make sense for either sets or lists. Even if they
cannot work on every type of collection, that's no reason not
to help them be as general as logic allows.
.... add=list.append
.... def clear(self):
.... del self[:]
....
>>> b = BryansList([1,2,3,4,5])
>>> b [1, 2, 3, 4, 5]
>>> b.add(6)
>>> b.clear()
>>> b
[]

Happy now? You can keep it, I don't need it. :)
Most of us consider minimal interfaces a virtue.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,734
Messages
2,569,441
Members
44,832
Latest member
GlennSmall

Latest Threads

Top