Welcome to the second part of the Teradata performance optimization series. Today we will analyze one of the leading causes of performance issues on a Teradata Data Warehouse system (although most of this probably is valid for any Data Warehouse).
A good data model will likely prevent future performance issues early in a Data Warehouse project.
I dare to argue that from a technical point of view, most projects fail or are in poor shape due to clients with a limited budget being saved. Creating a good data model does not immediately deliver visible results but directly causes costs.
Good data modelers are seldom. It is much cheaper to engage a developer educated in quickly moving data into the database than paying a data modeler to paint “boxes and lines” …
This approach may be tempting because the first results are soon available, and developers are the cheapest in the data warehouse job hierarchy. In a short time, you can bear fruit for your customer. He will be happy as he can assume his money is not wasted.
Some years ago, when the worldwide economic crisis struck daily rates, the data warehousing industry needed a marketing name for such a “reduced quality” approach.
Prototyping was the new buzzword.
Although, in principle, a good idea, most prototypes end up as the final solution. While initially prototypes were thought to be the communication link between the customer and base for further analysis and requirement specification, they often ended up being the final solution and blind alley at once.
I suppose the worst decision that can be taken during the modeling process is to take over the operational systems’ source definitions directly.
Integrating source systems 1:1 leads to operational data stores but has nothing to do with data warehousing. Combine this with the waiving of surrogate keys, and you can be sure that sooner or later, a source system will be replaced by another one. At this time, a cost-intensive redesign will wait for you.
Project costs have just been shifted to a later date, and they probably are many times higher now than the initial expenses an experienced data modeler would have caused.
Unfortunately, this is the time we live in, and we have to arrange somehow with this situation.
I decided to opt-out of this undesirable “low quality” data warehousing development. You can do it as I did: don’t compete against the crowd, fighting for projects in an environment of decreasing daily rates, but wait for the smashed projects to come to you.
The next part of this series will discuss our possibilities for fixing a broken data model.