Andrew's implementation of MapReduce according to mapReduce paper
It is suppose to be running in distributed network. We simulate this by running multiple process on a single host.
/mr
folder contains the implmenetation of mapReduce framework.
/main
contains the programs that utilize the underlying mapReduce framework and some test cases.
- To run testing, run
./test-mr.sh
inside themain
folder.
/mrapps
contains bunch of user defined map reduce functions.
If the master or coordinator failed, the whole MapReduce job will be re-run.
If a worker fails, the master/coordinator will be responsible for re-running the task.
- If the task is a map task, we will re-run it even if it is in complete state because the underlying disk is not accessible.
- If the task is a reduce task and it has not been completed yet, we will re-run it.
There mechanism used for checking if a worker is still alive is not the perfect solution, we simply check if each task has been completed within 10 seconds. If the task has not been completed yet, we re-assign the job to other workers.
We can run the word count example described in the paper above for testing purpose.
- Go to the
main
directory, e.g.cd main
- To remove any previous outputs, run
rm mr-out*
- To start the coordinator, run
go run mrcoordinator.go pg-*.txt
- We need to build the
wc.go
program as plugin first, rungo build -buildmode=plugin ../mrapps/wc.go
- Open a few other terminals to run some workers, run
go run mrworker.go wc.so
- The output files are stored inside
main/mr-out-<partitionNumber>