
This describes the classic MapReduce model at a conceptual level. Hadoop YARN has different coordination and recovery behavior; the details depend on version and configuration.
How the Components Fit Together
The input files reside in a distributed file system, such as HDFS for Hadoop, or GFS as Google calls it.
Worker processes handle mapper or reducer tasks.
Mappers read data from HDFS, apply the mapping function, and save the output to their local disks.
Mappers process input splits and partition their intermediate output for reducers; completion and recovery are tracked by task attempts, not by individual reduce keys.
The scheduler assigns reducer tasks, each responsible for a partition of intermediate keys.
Each worker process can act as either a mapper or a reducer, as they are not specialized.
Reducers fetch their partitions of mapper output during the shuffle, using task-location information.
A reducer task must wait for all mapper processes to deliver data for a specific key before it can begin processing.
Task completion and shuffle progress coordinate the reduce phase; the master does not schedule a separate worker for every key.
Why Fault Tolerance Matters at This Scale
MapReduce offers superior fault tolerance compared to traditional database systems, which is one of the paradigm’s key advantages.
Consider a conventional database system such as Teradata.
Despite the various fault tolerance features in a Teradata system, it is difficult to imagine scaling it to hundreds of thousands of processes.
Failure probability must specify a time interval and a model of dependence between failures.
Let’s try a straightforward example:
If each of n processes independently has failure probability p over the same interval, the chance of at least one failure is 1-(1-p)^n. For many processes this can approach 100%, but near-certainty is not mathematical certainty.
How Map-Reduce Recovers from Failure
Map-Reduce implementations prioritize fault tolerance.
The same statistical rules apply to Map-Reduce. Consider a Map-Reduce job generating numerous mapper and reducer processes. Such a job can run for extended periods, processing large volumes of data while performing complex computations such as pattern recognition.
The coordinator tracks task attempts and worker status. Recovery behavior depends on the implementation and configured durable state.
A failed map attempt may be rerun for its input split, including regeneration of lost intermediate output.
A failed reduce attempt may be rerun; exactly which output is retained depends on task completion and the output-commit protocol.
The master process monitors all activities within the MapReduce job and handles failures gracefully, even in systems with many processes.
Speculative execution may launch another attempt of a slow task; it is not simply reassignment of individual key ranges.
The Master Process as a Single Point of Failure
A master outage does not universally destroy all work. Hadoop YARN supports work-preserving ResourceManager restart when appropriately configured; older MapReduce designs have different recovery limits.
Statistical probabilities must be considered here as well.
Assess fault tolerance using the actual replication, task recovery and master recovery configuration. Process counts alone do not establish superiority over a database system.
Deciding what to change in your data platform?
Compare optimisation, modernisation and migration against your workload and business requirements.

