DWHPro Insights

Understanding the Relationship Between RDBMS and Hadoop: A Map Reduce Example

By Roland Wenzlofsky ·

Understanding the Relationship Between RDBMS and Hadoop: A Map Reduce Example

This is a conceptual comparison of SQL aggregation and Hadoop MapReduce, not proof of actual Teradata-to-Hadoop delegation.

To illustrate how RDBMS logic is executed in Hadoop via the MapReduce algorithm, we use the example of a SQL aggregation statement involving the joining of two tables.

SELECT
Segment,
SUM(balance)
FROM
Client t01
INNER JOIN
ClientType t02
ON
t01.clienttype_id = t02.clienttype_id
GROUP BY Segment;

The above SQL statement summarizes the client balances for each client segment (private, business, etc.).

Let’s see how this can be implemented with a MapReduce algorithm.

A two-job reduce-side join followed by aggregation is one possible design. Map-side joins, combiners and other execution choices may change the required jobs.

Initially, the JOIN step is executed wherein the data is sorted according to clienttype_id. This enables each reducer to emit the corresponding balance for every segment. In simpler terms, the first Map-Reduce task performs the join operation between the Client and ClientType table, without carrying out any grouping.

join task
join task

The second Map-Reduce task sorts the data by segment for consolidation in a single Reducer. This step involves summarization akin to that in a GROUP BY statement.

grouping

The example shows how equivalent relational work can be expressed with MapReduce. The audited SQL returned Private=30 and Business=50; no integration/delegation behavior was demonstrated.

Deciding what to change in your data platform?

Compare optimisation, modernisation and migration against your workload and business requirements.

Explore independent platform assessments

Practical data engineering insights, regularly

Join more than 3,500 data professionals: a regular newsletter with practical insights on data engineering, new data platforms (Snowflake, Databricks, Teradata) and AI.

Follow DWHPro in Google to see our articles more often in Search.

Leave a Comment

Get more practical data engineering advice

Join the DWHPro newsletter for practical data engineering tips, troubleshooting examples and data platform insights.

Protected by Security by CleanTalk and CleanTalk Anti-Spam