TPT Teradata – Introduction

Those of us working for many years with Teradata and its utilities know that the Teradata Parallel Transporter Utility (TPT) merges the functionalities of Fastload, Multiload, TPUMP, BTEQ, and Fastexport into one utility.

Teradata attempted to create a standard tool in the past, but I assume many of us did not even notice it, as it was an epic fail called “The Teradata Warehouse Builder”.

I can’t remember anybody ever using it at all. Nevertheless, scripts written in Teradata Warehouse Builder are executable with TPT without any changes.

While the standalone utilities’ syntax is inconsistent, TPT offers a uniform syntax for all loading, update, delete and extract tasks (and some other tasks, like DDL statements, execution of Linux shell scripts, etc.)

The base of TPT for Teradata is the concepts of data streams and operators.

Data streams are not directly accessible from your scripts. They are the pipelines between operators, and they are kept in memory. No data is written to the disks.

Operators read data from a source (a data stream or any other valid source like a flat-file or ODBC connection) or write data to a target (which can be a data stream or any valid target like a table or a flat file). Some operators take over more tasks, such as dropping and creating tables. We will describe each type of operator in detail later in this article.

We would probably use Linux pipes to make in-memory pipelining between files and the load utilities without TPT.

For example, one shell script could be writing a flat file into a named pipe, while at the same time, a Fastload script would be reading from this named pipe:

cat thefile > named_pipe &
cat named_pipe | fastload

Although TPT uses naming conventions and concepts different from the standalone tools, we can most easily find an equal TPT operator for each traditional tool; TPT combines all standalone tools and more features.

TPT Teradata – Operators Overview

TPT operators are grouped into producer operators (read operators), filter operators, and consumer operators (write operators).

Producer operators read data from various sources and make them available in a data stream for consumer operators (by reading from flat files, ODBC sources, SQL select statements, and export SQL).

Consumer operators read data from a data stream and write it to a target table or a flat-file.

Producer operators and consumer operators can use access modules. Access modules are software modules used to read from data stores such as CDs, DVDs, and tape drives.

The following access modules are available: Named pipes (for reading from Unix named pipes), WebSphere MQ (for reading from IBM message queues), and JMS. The user can implement additional access modules.

For a Teradata beginner, it isn’t easy to distinguish between consumers and producers. Maybe the easiest way to differentiate is to memorize the following:

Producer operators never write into the target, only into a data stream. Consumer operators never write into a data stream, only directly into a target.

The data streams connect operators (standalone operators we will cover later):

As you can see in the above picture, TPT covers the complete ETL chain.

The table below shows how TPT replaces the most used standalone utilities:

The most crucial difference between the standalone utilities and TPT is the level of parallelism. Traditionally the utilities have been strictly used sequentially. TPT offers parallelism by sharing the data streams and running several operators in parallel:

The level of parallelism is probably the most significant difference between the standalone utilities and TPT. Traditionally, the utilities have been strictly used sequentially. TPT offers parallelism by sharing the data streams and the possibility to run several operators in parallel:

In the above example, two producer operators run parallel, writing into a typical data stream. Simultaneously, two consumer operators read parallel from the data stream and write into the Teradata database table. Such a setup would need a lot of programming (Linux shell scripts, etc.) if implemented with the standalone utilities.
We will now get more into detail by showing you which operators exist and how the standalone tools made it into TPT operators:

The Producer Operators

Producer operators read data from a valid data source and make it available for consumer operators in a data stream.

The Data Connector Operator (DATACONNECTOR PRODUCER):

The Data Connector Operator is a two-way operator,  either a producer or consumer operator.

When the type is DATACONNECTOR PRODUCER, a producer operator is used to read data from flat files or an access module, pushing the data into a data stream.

It can read from a single flat file (similar to the file=” filename” statement in a Fastload). Furthermore, TPT can read all files of a specific directory matching a wildcard pattern simultaneously (i.e., treated as a single input file).

Apart from directly reading from flat files, we can use INMOD adapters to push the data to the consumer operator (Fastload INMOD and Multiload INMOD).

The Export Operator (EXPORT):

This operator replaces the Fastexport utility. It reads data from a Teradata table (using a SQL SELECT statement) and pushes it into a data stream: It’s a producer operator. It puts the data into a data stream, not directly into a flat-file!

The SQL Selector Operator (SELECTOR):

This operator produces data by executing a SQL SELECT statement. The data are written into a data stream. This operator is comparable to the BTEQ export.

The ODBC Operator (ODBC):

The ODBC operator produces data by reading from an ODBC data source and writing it into a data stream.

Consumer Operators

Consumer operators read data from a data stream and write it into a table or a flat-file target. Access modules are also valid targets. Consumer operators read from data streams and write to targets.

Consumer operators correspond with their standalone utilities.

The Data Connector Operator (CONSUMER):
When defined as a consumer type, the Data Connector Operator is used to write into a flat-file. Even access modules are utilized as the target.

The Load  Operator (LOAD):
This operator offers the block-level load functionality we can find in a Fastload.
The Update Operator (UPDATE):
This operator provides the enhanced block-level update functionality we can find in a Multiload.

The Stream Operator (STREAM):
This operator implements TPUMP functionality.

The SQL Inserter Operator (INSERTER):
This operator performs the transactional BTEQ INSERT functionality.

The Fastexport OUTMOD (FASTEXPORT OUTMOD):
This operator allows for the usage of the Fastexport OUTMOD adapter.

Filter Operators

We use this operator to apply filtering to the data stream.

The TPT scripts allow us to invoke user-written filters (C operator, C++), WHERE clauses, and CASE DML expressions in APPLY statements.

Standalone Operators

The OS Command Operator (OS COMMAND):

We operator to execute Linux commands. It replaces the functionality BTEQ offers with the “.OS” command.

The DDL Operator (DDL):

We use this operator to execute DDL statements. It’s useful for tasks such as dropping or creating tables and indexes before the data load occurs.

While this operator’s name might be misleading, it allows for any SQL statements that don’t return a result set.

We can use, for example, statements like INSERT…SELECT, UPDATE, and DELETE.

The Update Operator (UPDATE):
When used for deletion, it replaces an optimized Multiload DELETE.

TPT is tightly coupled to the standalone utilities, offering more functionality than the other tools, Fastload, Multiload, and Bteq.