GUEST POST BY ARTEMIY KOZYR
Today I will show you how to identify the source problem to tune the query’s performance utilizing the unnecessary amount of IO, what UII indicator is, and how the WHERE clause placed on a VIEW is evaluated.
This case shows that the WHERE clause placed on VIEW for 2 Partitioned tables resulted in excessive block reads and applying WHERE condition afterward. You need to be fully aware of the amount of data query reads and the amount of data they need.
Performance tuning boosted the performance five fold. The query is completed 4-5 times faster and utilizes 4-5 times fewer system resources:
1+ Billion rows INSERTED
Here is the target query for today, which populates a large fact table of the data mart.
– INSERT INTO … SELECT FROM …
– The query is run on a daily basis
– Over 1000M rows inserted (1 billion)
Poor performance detected
Query performance statistics stored in Teradata QueryLog signifies that the query is suspicious and performing badly in terms of unnecessary IO, excessive Spool Spool space and CPU utilization.
Take a look at Teradata performance metrics:
What knowledge can we discover from this data?
What is particularly wrong, and how can we tune and improve the query?
A closer look at IO, source view DDL
First of all, let us describe what UII (Unnecessary IO indicator) is and what it can tell us.
UII is calculated as SumIO / (SumCPU * 1000). The metric is used to make an idea about query efficiency for consuming CPU and IO resources. If UII is relatively high, then it could mean that many data blocks are read, but a relatively small proportion of them is processed.
Let us examine source View DDL through which the target table is populated.
A-ha! Here we see that the latest six months are extracted from the view.
The tables are partitioned by columns used in WHERE clause:
But only last month’s data is inserted in the target table! The INSERT statement is as:
So, the query simply reads 5 months data in vain with no particular reason. The thing is that before final query applies any WHERE clause, the whole data is read.
It means every data block under this VIEW is read first, and then non-relevant data is eliminated. But why read unnecessary data? Let us figure out how to force Teradata to extract only relevant partitions.
Here is the answer:
1. We might put the WHERE clause from the final query INSIDE the source VIEW DDL.
2. We might create an entirely new VIEW for this ETL process in case if some other users might not want any changes to the VIEW.
Improved query performance
This optimization gave us five times boost in performance. The query is completed 4-5 times faster and utilizes 4-5 times fewer system resources.
Detailed query metrics below:
Artemiy Kozyr is Data Engineer at Sberbank, Moscow, with Master Degree in CS.
He has five years of Experience in Data Warehousing, ETL, Visualization for Financial Institutions.