Data Warehouse of Horrors: 20 Years of Watching Smart People Build Stupid Things

There was a time when a single team could build an entire data warehouse. Not a team of forty. Not a team of sixty distributed across three continents and coordinated by a project management office that had never seen an execution plan. A team of five. Perhaps six, if one counts the person from Controlling who appeared regularly to ask when the report would be ready.

This was the late 1990s. Data warehousing was young. Teradata was the dominant platform. And the people who built these systems understood, from end to end, what they were building and why.

The Generalist Era

In those early days, we did not call ourselves data engineers. The term did not exist, and it was not needed. We were the people who built the warehouse. Because there were so few of us, every team member was expected to understand the entire process — not in theory, not from a conference presentation, but in daily practice.

The work began with the business. One sat down with risk managers, controllers, or marketing analysts and determined what they were trying to achieve. This was not called “requirements engineering.” It was a conversation. In my experience, this direct interaction between people who understood the business and those who understood the technology was the single most important factor in the success of early data warehouse projects.

Then one went looking for the data. This meant crawling through source systems, legacy databases, and flat files of uncertain origin. One learned what the data actually looked like, which was rarely what the documentation claimed. The documentation, if it existed, had typically been written years earlier by someone who had since left the organization. This is a pattern that has not changed in twenty years.

Then one designed the data model. Wrote the ETL. Loaded the data. Tuned the queries when they performed poorly — which they inevitably did, because someone in the business had requested a join across three years of transaction history without filters. And one fixed it, because one understood every layer of the system. There was nobody else to escalate to.

The Value of End-to-End Ownership

Here is what is rarely discussed about those early projects: the work was deeply satisfying.

When a batch process failed at three in the morning — and this happened regularly, as the hardware of that era would make a modern smartphone appear overpowered — you knew exactly where to look. You had built it. You understood the data model because you had designed it. You understood the ETL because you had written it. You understood the business logic because you had sat in the room with the people who explained it.

There was no escalation path. There was no incident ticket that traveled between three teams for two weeks while production remained broken. There was you, your knowledge of the system, and a SQL session. By morning, the problem was resolved.

The teams were small enough that knowledge was shared by default. The architect understood the ETL. The ETL developer could explain the business logic to a new team member. The business analyst could, at a minimum, read an execution plan and understand why a report was slow. Knowledge did not live in organizational silos. It lived in people, and those people sat within immediate reach of one another.

In my experience, this is the single factor most responsible for the success of early data warehouse implementations: the absence of artificial boundaries between understanding the business problem and solving it technically.

A Formula That Required No Framework

Looking back, the approach was not complicated. One hired competent people with broad skills. One gave them responsibility across the full lifecycle. One let them communicate directly with the business. One did not place seventeen layers of governance, process, and methodology between the person who understood the problem and the person who could solve it.

It worked. Not because these teams were composed of exceptional individuals. It worked because the organizational structure had not yet evolved to the point of actively preventing competent people from doing their jobs. This observation is not cynicism. It is, in my experience, the most accurate summary of what changed in the decades that followed.

A team of five could deliver a data warehouse for an entire business unit of a large enterprise. Not a prototype. Not a minimum viable product. A working system in production, with real users running real reports that informed real business decisions. And the same five people who built the system were the ones who maintained it, tuned it, and extended it over the years that followed.

The Numbers in Perspective

Let me provide some context. In 1999, a team of five or six people could design, build, and deploy a production data warehouse in approximately nine to twelve months. The total cost — including hardware, software licenses, and the people in the room — was manageable and justifiable. The business received its reports. The data was reliable. The system met its service-level agreements.

Fast forward twenty years. The same company now employs, directly or through external contractors, between 60 and 120 people to achieve what amounts to the same functional outcome. The project duration has extended to two or three years. The total cost has increased by a significant multiple. And yet, the business users still cannot obtain their reports on time, because — and I cite this as a real example — someone changed the categorization scheme on the project management board, and nobody can determine which development sprint they are currently in.

What happened between then and now is the subject of the chapters yet to be written. But before proceeding, it is worth pausing to note that there was a time when building a data warehouse was a craft. It was practiced by a small number of skilled professionals who could move seamlessly between business analysis, data discovery, data modeling, development, and performance tuning. The work was hard. It was deeply technical. And it required people who understood that a data warehouse is, at its core, one interconnected system — not a collection of isolated components to be assembled by strangers.

It worked. And the people who did it took genuine pride in what they built.

Everything that came after was, to put it charitably, a series of increasingly expensive experiments in how to make simple things complicated.

Roland Wenzlofsky is the founder of DWHPro and a data warehouse consultant with over 20 years of experience helping major organizations manage and optimize their analytical platforms. He is the author of “Teradata Query Performance Tuning” and writes regularly about data warehousing, Teradata, Snowflake, and Databricks.