spliting a list by nth items

[Steven Bethard]

I feel like this has probably been answered before, but I couldn't
find something quite like it in the archives. Feel free to point me
somewhere if you know where this has already been answered.

I have a list in a particular order that I want to split into two
lists: the list of every nth item, and the list of remaining items.
It's important to maintain the original order in both lists.

So the first list is simple:

nth_items = items[::n]

The second list is what's giving me trouble. So far, the best I've
come up with is:

non_nth_items = list(items)
del non_nth_items[::n]

Is this the right way to do this?

Steve

 

[Peter Hansen]

I feel like this has probably been answered before, but I couldn't
find something quite like it in the archives. Feel free to point me
somewhere if you know where this has already been answered.

I have a list in a particular order that I want to split into two
lists: the list of every nth item, and the list of remaining items.
It's important to maintain the original order in both lists.

So the first list is simple:

nth_items = items[::n]

The second list is what's giving me trouble. So far, the best I've
come up with is:

non_nth_items = list(items)
del non_nth_items[::n]

Is this the right way to do this?

There's probably no "right" way. Your way isn't the way I
would have thought to do it, I think. But it's brilliant.
(Also simple and clear.)

-Peter

 

[Michael Hoffman]

I have a list in a particular order that I want to split into two
lists: the list of every nth item, and the list of remaining items.
It's important to maintain the original order in both lists.

So the first list is simple:

nth_items = items[::n]

The second list is what's giving me trouble.

non_nth_items = [x for x in items if x % n]

 

[Larry Bates]

I believe you meant somthing like:

non_nth_items = [items[i-1] for i in range(1,len(items)-1) if i % n]

I don't think the "if x % n" won't work on his list of items

Larry Bates

I have a list in a particular order that I want to split into two
lists: the list of every nth item, and the list of remaining items. It's
important to maintain the original order in both lists.

So the first list is simple:

nth_items = items[::n]

The second list is what's giving me trouble.

non_nth_items = [x for x in items if x % n]

 

[Michael Hoffman]

I believe you meant somthing like:

non_nth_items = [items[i-1] for i in range(1,len(items)-1) if i % n]

I don't think the "if x % n" won't work on his list of items

D'oh! I wasn't thinking--the test list of items I used was
range(something), so of course it worked.

What I meant <wink> was:

non_nth_items = [x for index, x in enumerate(items) if index % n]

Thanks for the correction.

 

[Steven Bethard]

I believe you meant somthing like:

non_nth_items = [items[i-1] for i in range(1,len(items)-1) if i % n]

Yeah, the items list is not of the form range(x) -- in fact, my current use
for this is with a list of filenames -- so Michael Hoffman's solution:

non_nth_items = [x for x in items if x % n]

wouldn't work. In the full fledged problem, I actually have a start parameter
for the slice as well as the step parameter (n), so I could probably do
something along these lines (inspired by your two suggestions):

non_nth_items = [item for i, item in enumerate(items) if (i - start) % n]

instead of my original

non_nth_items = list(items)
del non_nth_items[start::n]

The enumerate/range solution is a little more verbose, but it does only go
through the list once. Thanks for the good suggestions!

STeve

 

[Raymond Hettinger]

[Steven Bethard]

so I could probably do
something along these lines (inspired by your two suggestions):

non_nth_items = [item for i, item in enumerate(items) if (i - start) % n]

instead of my original

non_nth_items = list(items)
del non_nth_items[start::n]

The enumerate/range solution is a little more verbose, but it does only go
through the list once. Thanks for the good suggestions!

The enumerate solution has a lot going for it. It is more flexible and easily
accomodates criteria other than every nth item. More importantly, it
demonstrates an important guiding principle: multiple list deletions are a code
smell indicating that building a new list is a better approach. On this
newsgroup, I've seen people tie themselves in knots with multiple in-place
deletions during iteration when the new list approach was clearer, more robust,
and faster.

The runtime behavior of the non_nth_items[start::n] approach is implementation
dependent. One can easily imagine a O(n**2) process running behind the scenes.
CPython is a smarter than that, but the code for list_ass_subscript() in
listobject.c is not a pretty sight.


Raymond Hettinger

 

[Steven Bethard]

The enumerate solution has a lot going for it. It is more flexible and
easily accomodates criteria other than every nth item.

Yeah, I'm not so much worried about this, or I wouldn't be using
items[start::n]
to make the nth_items list in the first place. (If I was planning to allow
for other criteria, I'd probably use filter or ifilter instead of the
slicing...)

More importantly, it demonstrates an important guiding principle: multiple
list deletions are a code smell indicating that building a new list is a
better approach.

This was kinda my feeling -- hence why I posted. Part of the problem for me
was that because the two lists were so closely related semantically, I was
looking for a syntax that was also closely related. Sometimes this is a good
intuition to have, sometimes not.

The runtime behavior of the non_nth_items[start::n] approach is
implementation dependent.

Ahh, I didn't know that. Good thing to know...

Thanks,

Steve

 

[Elaine Jackson]

x = [0,1,2,3,4,5,6,7,8,9]

n = 3
y = [x for i in range(len(x)) if i%n==0]
z = [x for i in range(len(x)) if i%n<>0]
x [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
y [0, 3, 6, 9]
z

[1, 2, 4, 5, 7, 8]


| I feel like this has probably been answered before, but I couldn't
| find something quite like it in the archives. Feel free to point me
| somewhere if you know where this has already been answered.
|
| I have a list in a particular order that I want to split into two
| lists: the list of every nth item, and the list of remaining items.
| It's important to maintain the original order in both lists.
|
| So the first list is simple:
|
| nth_items = items[::n]
|
| The second list is what's giving me trouble. So far, the best I've
| come up with is:
|
| non_nth_items = list(items)
| del non_nth_items[::n]
|
| Is this the right way to do this?
|
| Steve
| --
| You can wordify anything if you just verb it.
| - Bucky Katt, Get Fuzzy

 

[Carlos Ribeiro]

I was one of the frequent posters on the many tools/ides/gui threads
that appeared here over the past weeks. I'm now partly satisfied. My
search for a good IDE that is usable in my lowend machine is still not
completed. But I found a good setup using a free editor and PythonWin
working together.

I'm using Crimson Editor, a small freeware editor. The most
interesting part is that it isn't the most featured of all editors
I've tried; it isn't as powerful as vi or emacs, but has one feature
that I wanted the most, that is a fairly simple project management
tool. It simply allows me to organize a list of files that I'm using
for a project on the left pane. I can click on a file to open it. It's
amazingly useful when you're dealing with "hot" files stored on
several different directories, and you don't have to remember where is
it located, or to manually walk the open file dialog looking for it.

I'm using PythonWin for testing, mostly because the Windows console is
so braindead, and because PythonWin help is quick well organized (it's
about the same that you find in the web, but a little bit more
convenient).

I'm also evaluating Wing IDE 2.0; I had a few issues with it that were
solved, but for some tasks I found it too heavy for my limited
resources. Anyway, I'm keeping an eye on it.

--
http://mail.python.org/mailman/listinfo/python-list





--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Jorge Godoy]

I'm using Crimson Editor, a small freeware editor. The most
interesting part is that it isn't the most featured of all editors
I've tried; it isn't as powerful as vi or emacs, but has one feature
that I wanted the most, that is a fairly simple project management
tool. It simply allows me to organize a list of files that I'm using
for a project on the left pane. I can click on a file to open it. It's
amazingly useful when you're dealing with "hot" files stored on
several different directories, and you don't have to remember where is
it located, or to manually walk the open file dialog looking for it.

I'm an Emacs user, but you might try looking at http://ecb.sf.net as a
'plugin' to (X)Emacs. It comes installed with XEmacs...

Also, I can't help with Windows. I find that tools there are very poor
when compared to what we have on unices

 

[Fernando Perez]

I'm using PythonWin for testing, mostly because the Windows console is
so braindead, and because PythonWin help is quick well organized (it's
about the same that you find in the web, but a little bit more
convenient).

<blatant plug>

You may want to try ipython (http://ipython.scipy.org). It gives you under
Windows (or linux, or OSX) colored tracebacks, tab-completion, directory
management, integrated pdb loading on uncaught exceptions, profiler support,
and quite a bit more.

I don't use windows, but under linux my development setup is Xemacs+ a terminal
with ipython running. Thanks to ctypes and UNCreadline, you get exactly the
same functionality under windows (along with your editor of choice). I
honestly find it to be a _very_ efficient setup.

</plug>

Cheers,

f

 

[Bryan]

I was one of the frequent posters on the many tools/ides/gui threads
that appeared here over the past weeks. I'm now partly satisfied. My
search for a good IDE that is usable in my lowend machine is still not
completed. But I found a good setup using a free editor and PythonWin
working together.

I'm using Crimson Editor, a small freeware editor. The most
interesting part is that it isn't the most featured of all editors
I've tried; it isn't as powerful as vi or emacs, but has one feature
that I wanted the most, that is a fairly simple project management
tool. It simply allows me to organize a list of files that I'm using
for a project on the left pane. I can click on a file to open it. It's
amazingly useful when you're dealing with "hot" files stored on
several different directories, and you don't have to remember where is
it located, or to manually walk the open file dialog looking for it.

I'm using PythonWin for testing, mostly because the Windows console is
so braindead, and because PythonWin help is quick well organized (it's
about the same that you find in the web, but a little bit more
convenient).

I'm also evaluating Wing IDE 2.0; I had a few issues with it that were
solved, but for some tasks I found it too heavy for my limited
resources. Anyway, I'm keeping an eye on it.

i do a lot of python (and jython) programming and i use crimson editor and pythonwin also. in fact, i use crimson
editor for everything. i've even associated just about every file type in windows to crimson editor


bryan

 

[Fernando Perez]

Another tool that I use that I just started using but I'm finding
indispensable is IPython. Features:

* tab completion - this is probably the hugest benefit.
* executes (some? all?) OS commands without having the exit out -
tab completion works on filesystem paths here

The @rehash magic command will load _all_ of your $PATH as aliases, by default
only the most common things are loaded. You can just type alias to see what's
active. Note that the pysh profile (ipython -p pysh) preloads your whole path,
and adds a few more modifications, so you can use ipython as a system shell,
but with python syntax (and full two-way communication of variables). Here's
an example:

fperez@maqroll[~/test]|8> $$files=ls
fperez@maqroll[~/test]|9> type(files)
<9> <type 'list'>
fperez@maqroll[~/test]|10> len files
-------------------------> len(files)
<10> 33
fperez@maqroll[~/test]|11> for f in files:
|..> if len(f) > 10:
|..> wc -l $f
|..>
4 inspectbug.py
73 ramptest.py

Granted, it's silly, but I don't know in the shell how I could easily say "run
wc -l over all files whose name is longer than 10 characters". I can _never_
remember how to do complicated logic/looping in the shell, while I can
certainly do such things trivially in python. The pysh profile puts ipython in
a mode which bridges this gap. Incidentally, pysh is actually just a special
configuration mode for ipython, consisting of about only 36 lines of code (the
rest is regular ipython).

* probably more....but I'm still digging into this

Lots more

Best,

f

 

[Dan Perl]

I'm also evaluating Wing IDE 2.0; I had a few issues with it that were
solved, but for some tasks I found it too heavy for my limited
resources. Anyway, I'm keeping an eye on it.

I didn't follow the entire IDE dispute and I apologize if my comments are
now redundant, but here are my 2 cents.

I just bought the Komodo Personal edition. I had already used the trial
download for a month and I've been quite satisfied with it. That's after
using PythonWin for a few months, but it would be hard to go back to
PythonWin now.

Before making the decision to buy Komodo, I also evaluated Wing IDE 2.0,
their Personal edition. Please indulge me, but I have to take a load off my
chest about Wing IDE.

Right off the bat, the trial period was only 10 days. I've evaluated quite
a few tools that had a free trial period and I don't think I can remember of
ever having only 10 days before. Then, the GUI is kind of awkward in a few
places (the "project tree view", or something like that, and the toolbars).
There is more. The Personal edition does not have an object browser (it's
only in their more expensive editions). With all due respect, that's a
minimal requirement in a commercial IDE these days and Komodo has one in its
Personal edition (for the same price as Wing IDE).

I left my biggest beef with Wing IDE for the end. I debugged one of my
scripts (that I knew it works) and got an unexplainable exception. It turns
out that Wing IDE falsely detects some exceptions. They know about it and
they suggest that if you KNOW that it's a false exception you should flag it
to be ignored. Yes, Wingware people, I know it's a false exception, but
what am I using an IDE with debugging for if I have to figure out false
exceptions and ignore them? And how do I know that that particular
exception will not be valid some day in one of my scripts? And please, do
not insult our intelligence with some marketing mumbo-jumbo that Wing IDE's
exception detection is so powerful that it even detects exceptions that are
not there. It's a BUG, it's not a feature, folks! And no excuse will make
up for it!

The choice between Wing IDE and Komodo was clear for me.

As a side note, among free open-source IDEs, I thought that pydev (a plugin
for eclipse) shows promise, but it's not ready yet for the big leagues.

There. Thanks for listening.

Dan

 

[Peter Hansen]

I debugged one of my
scripts (that I knew it works) and got an unexplainable exception. It turns
out that Wing IDE falsely detects some exceptions. They know about it and
they suggest that if you KNOW that it's a false exception you should flag it
to be ignored. Yes, Wingware people, I know it's a false exception, but

What exactly is a "false exception"? I have never heard of such
a thing, and can't imagine what it might be. Python raises
exceptions, they are subclasses of Exception or (old-style)
strings, they have tracebacks, they come from a variety of
sources, there's lots of things to know about exceptions but
"falseness" is not something about which I'm aware...

-Peter

 

[Carlos Ribeiro]

What exactly is a "false exception"? I have never heard of such
a thing, and can't imagine what it might be. Python raises
exceptions, they are subclasses of Exception or (old-style)
strings, they have tracebacks, they come from a variety of
sources, there's lots of things to know about exceptions but
"falseness" is not something about which I'm aware...

They're false in the sense that that they're not supposed to be raised
to the attention of the user. What happens is that Wing IDE is super
sensitive with regards to exceptions; when an exception is raised in
some internal module, Wing tries to outsmart the library and warn the
user that an exception happened. A "false exception" is one that would
happen and be silently treated by the library itself, without nobody
noticing, but that's catched by Wing IDE nonetheless, in a totally
unnecessary and obstrusive way. You can see that it's a case where the
tool tries to be too smart for its own good.

p.s. Why do they do it? For what I could see, the intention seems to
be able to catch situations where a valid exception that should be
raised to the user attention is silented by a generic error handler.
It really may be useful in some situations, but it isn't a reasonable
default in my opinion.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Dan Perl]

Yes, that's what I was talking about. Thanks, Carlos, for explaining it. I
actually should have said false detection of exceptions or wrong detection
of exceptions. I don't remember what Wingware are calling it. Actually,
the way they were qualifying it was something like "if you do not see this
exception when running the script outside the IDE, then you should probably
flag it to be ignored". And they are giving a list of builtin modules where
those exceptions are usually "falsely" detected.

I didn't try to figure out what their exception detection mechanism is and
why they have this problem. I just uninstalled the IDE. So 10 days for the
trial was more than enough after all. ;-) I think I had it installed for
about 2 days. I can tolerate a bug and I wouldn't be so riled up, but what
got me was the marketing spin and how they were just excusing the bug, like
they would never fix it.

The "false exceptions" and the lack of a class browser made me decide not to
choose Wing IDE this time. Their attitude about the "false exceptions" made
me cut the trial short and never to consider Wing IDE again.

Dan

 

[Dan Perl]

I've done a little more digging into the Wing IDE documentation and this is
one of the better explanations I've found on the "false exceptions":

"As you try out the various demos from the tree on the left of the wxPython
demo app, you will sometimes see Wing IDE report exceptions in its Error
List dialog, which will come to the front as the wxPython demo pauses at the
exception. These are not program errors or a malfunction of the IDE. As
described at the end of the Error List text, these are caused by Wing's
proactive exception detection algorithm, which cannot see into the C and C++
code that is handling these exceptions outside of the debugger.
To get past them, select "Ignore this exception location" in the Error List
window and click on Continue Execution. You will see 3-4 of these the first
time Wing IDE encounters them. After that, your ignore list is stored in the
project so you will never see them again, even in future debug sessions.
Subsequently, you will benefit from Wing's ability to stop immediately at
the point of exception, rather than after the fact. This makes understanding
the conditions leading to an error much easier and speeds up debug
turn-around."

I can see a benefit in stopping at the point of exception, but if that
causes false positives (term used by one of their support people) with the
builtin modules that are implemented in C/C++, then I'd rather just live
without this benefit. Apparently, there is a way to disable that behavior
(I found out about it only now), but Wing keeps recommending the "Ignore
this exception location" solution.

And here is the kind of comment that really bothers me. An actual statement
from their documentation ("Debugging Zope with Wing IDE"):

"Another useful exception related feature is the ability to ignore
exceptions selectively by line of code on which they occur. You can choose
this option in the Error List window that comes up when an exception occurs.
In Zope, as in other Python apps that contain some non-Python code, Wing
will incorrectly identify some exceptions as unhandled because it cannot see
whether enclosing C code is going to handle the exception. This usually
happens in 1-2 places in Zope and can be dealt with by ignoring subsequent
instances of the exception. The zope.wpr project distributed with the
Wing+Zope bundle already has these ignores in place."

No, this is NOT a "useful exception related feature", it's a workaround for
a bug. And it's 1-2 times for Zope, it was 3-4 times in the comments above,
how do they get these estimates? I got this problem (er, useful exception
related feature) with the first script I debugged.

Dan

 

[Steve Holden]

Yes, that's what I was talking about. Thanks, Carlos, for explaining it. I
actually should have said false detection of exceptions or wrong detection
of exceptions. I don't remember what Wingware are calling it. Actually,
the way they were qualifying it was something like "if you do not see this
exception when running the script outside the IDE, then you should probably
flag it to be ignored". And they are giving a list of builtin modules where
those exceptions are usually "falsely" detected.

As a recent convert to Wing I think you aren't giving them enough
credit: remember, this only happens in the 2.0 BETA product (or, if it
also happens in earlier versions, they are definitely planning to
minimize the effects in the production version).

I didn't try to figure out what their exception detection mechanism is and
why they have this problem. I just uninstalled the IDE. So 10 days for the
trial was more than enough after all. ;-) I think I had it installed for
about 2 days. I can tolerate a bug and I wouldn't be so riled up, but what
got me was the marketing spin and how they were just excusing the bug, like
they would never fix it.

Well, I'm sure I remember reading somewhere that before they go into
production they plan to add a standard list of such exceptions for the
various Python versions, so that the standard libraries are far less
troublesome in this respect.

Plus, IIRC, all you have to do is check an "ignore this exception" box
to have a specific exception ignored everafter, which didn't seem like a
huge deal to me.

The "false exceptions" and the lack of a class browser made me decide not to
choose Wing IDE this time. Their attitude about the "false exceptions" made
me cut the trial short and never to consider Wing IDE again.

I find that it's very useful to be able to select from a dropdown list
of defined classes, and then select from a further drop-down list of
methods of that class. While it may not be a full class browser it's
certainly excellent functionality.

regards
Steve