DWHPro Insights

DWH Design

Data Warehouse of Horrors: 20 Years of Watching Smart People Build Stupid Things

In the projects I remember, a small team—sometimes five or six people—covered the whole warehouse. These staffing numbers and recalled conversations are personal observations, not an industry benchmark. This was the late 1990s. Data warehousing was young. Teradata was the dominant platform. And the people who built these systems understood, from end to end, what …

Read more

Real-World Map-Reduce Implementations: Design and Fault Tolerance

Illustration of a network graph with circular nodes joined by lines

This describes the classic MapReduce model at a conceptual level. Hadoop YARN has different coordination and recovery behavior; the details depend on version and configuration. How the Components Fit Together The input files reside in a distributed file system, such as HDFS for Hadoop, or GFS as Google calls it. Worker processes handle mapper or …

Read more

Understanding Teradata Load Isolation Most read

Illustration of a network graph with circular nodes joined by lines

Isolation Levels and their Impact on Performance & Concurrency Load isolation provides a defined committed view under its supported rules; it is not a blanket serializability or performance guarantee. Feature support and restrictions must be verified for the target deployment. Teradata provides varying degrees of isolation: In the next chapter, we will describe how these …

Read more

Building a Teradata Data Warehouse: Considerations for ETL Process, SQL Queries, and Physical Data Model Popular

Illustration of a network graph with circular nodes joined by lines

This post aims to compile all crucial aspects to be considered while constructing a Teradata Data Warehouse, including the ETL process and SQL queries. This list is just the beginning, and I anticipate receiving valuable feedback from my readers to expand it in the future. Initially, I have provided a few concepts, but I intend …

Read more

Understanding the Waterfall Model: Phases, Problems, and Solutions for Data Warehouse Projects Popular

Illustration of a dense isometric network of connected data nodes

What is the Waterfall Model? The waterfall model facilitates the sequential progression of a data warehouse project. Each phase must be concluded before the subsequent stage commences. The following stages will be navigated: What are the Problems with the Waterfall Model? Typically, these issues are identified solely during the verification stage of testing. Why do …

Read more

Teradata Referential Integrity: What it is and Why You Need it for Data Consistency and Performance Popular

Illustration of a dense isometric network of connected data nodes

Introduction to Teradata Referential Integrity Teradata implements 3 Types of Referential Integrity. 1. What is Standard Referential Integrity? The Standard Referential Integrity checks every Row INSERT, DELETE, or UPDATE immediately to ensure referential integrity. A reference index sub-table is required for referential integrity. Violation of referential integrity results in the failure of execution and generates …

Read more

The Importance of Teradata Surrogate Keys Popular

Illustration of a dense isometric network of connected data nodes

What are Teradata Surrogate Keys? A surrogate key is an artificial identifier mapped to a natural key, often INTEGER or BIGINT. Generated identity values need not be gapless or globally increasing in insertion or retrieval order. We can use the ROW_NUMBER() function in Teradata to programmatically generate the surrogate key or an IDENTITY column for …

Read more

Protected by Security by CleanTalk and CleanTalk Anti-Spam