Simple map/reduce utility function for data analysis

[Raymond Hettinger]

Here's a handy utility function for you guys to play with:

http://code.activestate.com/recipes/577676/


Raymond
twitter: @Raymondh

 

[Paul Rubin]

Here's a handy utility function for you guys to play with:
http://code.activestate.com/recipes/577676/

Cute, but why not use collections.defaultdict for the return dict?
Untested:

d = defaultdict(list)
for key,value in ifilter(bool,imap(mapper, data)):
d[key].append(value)
...

 

[Steven D'Aprano]

Here's a handy utility function for you guys to play with:

http://code.activestate.com/recipes/577676/

Nice.

That's similar to itertools.groupby except that it consolidates all the
equal key results into one list, instead of in consecutive runs.

Also groupby returns iterators instead of lists, which makes it a PITA to
work with. map_reduce is much more straightforward to use.

Example given in the code:
{0: [10, 12, 14, 16, 18, 20], 1: [11, 13, 15, 17, 19]}

[(key[0], list(group)) for key,group in groupby(range(30), even_odd)

if key is not None]
[(0, [10]), (1, [11]), (0, [12]), (1, [13]), (0, [14]), (1, [15]), (0,
[16]), (1, [17]), (0, [18]), (1, [19]), (0, [20])]


So... when can we expect map_reduce in the functools module?

 

[Raymond Hettinger]

Cute, but why not use collections.defaultdict for the return dict?
Untested:

My first draft had a defaultdict but that implementation detail would
get exposed to the user unless the return value was first coerced to a
regular dict. Also, I avoided modern python features so the code
would run well on psyco and so that it would make sense to beginning
users.

Untested:
d = defaultdict(list)
for key,value in ifilter(bool,imap(mapper, data)):
d[key].append(value)
...

Nice use of itertools. FWIW, ifilter() will accept None for the first
argument -- that's a bit faster than using bool().


Raymond