Google's MapReduce

B

bearophileHUGS

(This Google article suggestion comes from the CleverCS site):

http://labs.google.com/papers/mapreduce-osdi04.pdf

"MapReduce is a programming model and an associated implementation for
processing and generating large data sets. Users specify a map function
that processes a key/value pair to generate a set of intermediate
key/value pairs, and a reduce function that merges all intermediate
values associated with the same intermediate key."

This looks like something that can be (nicely) done with Python too.
Bye,
Bearophile
 
T

Terry Reedy

(This Google article suggestion comes from the CleverCS site):

http://labs.google.com/papers/mapreduce-osdi04.pdf

"MapReduce is a programming model and an associated implementation for
processing and generating large data sets. Users specify a map function
that processes a key/value pair to generate a set of intermediate
key/value pairs, and a reduce function that merges all intermediate
values associated with the same intermediate key."

Summarizing groups (and sometimes simultaneously by subgroups) has been a
standard operation for decades in both statistics packages and database
report generators. Python has various versions of the map and reduce
operations that they use. Its dicts can easily be used to group items with
the same key. What is somewhat specific to Google is the need to generate
*multiple* key-value pairs from each input document. What MapReduce does
that is somewhat innovative is automatically parallelize the computation to
run fault-tolerantly on a cluster of up to 1000s of machines with machine
slowdowns and failures 'common'. And indeed, that is the reason to use the
system with trivial map or reduce functions, as they sometimes do.

What this does show is that Google could be regarded as a cluster
supercomputing company, with Web search as the visible development
application.

Terry J. Reedy
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top