Connecting two systems was once the work. It is now the part a machine can do. What remains is the part that was always the point.
Some time ago, I argued that the market does not pay for tool lists. The argument has aged in an unexpected direction. It is no longer only the list that has lost its value. It is the act of using the tools.
Mature connectors and generated boilerplate can reduce routine integration work, but connectors are not universally free or reliable. Authentication, compatibility, network limits, schema changes and operational support still require engineering.
Pipelines can fail in connectivity and integration as well as in downstream semantics. Successful transport does not prove correct business data.
Where Pipelines Actually Fail
Consider the most ordinary task in modern data work: read a stream of records, shape them, and load them into a warehouse. The demonstration takes an afternoon. The connector reads, the transformation runs, the rows arrive. Anyone can build it. A machine can build it. Then it goes to production, and the questions begin.
One record in the stream is malformed. Does the pipeline stop, or does the bad record step aside while the rest proceeds? Where does it go, and who decides what happens to it later? This is not a technical afterthought. It is a decision about whether a single bad row is permitted to halt a business process, and in a regulated environment, it is an audit question before it is an engineering one.
The load fails. Is the failure transient, in which case it should be retried, or is it permanent, in which case retrying it produces an infinite loop that quietly takes the whole pipeline down? The distinction is not visible in the wiring. It is a judgment about the nature of the error.
The system that delivers the data promises to deliver it at least once. At least once means one or more deliveries; duplicates are possible, not inevitable. More than once means duplicates, and duplicates mean the load must be designed so that loading the same record twice changes nothing. Nobody who only wired the connection would know this was required. The pipeline that ignores it does not announce the problem. It simply reports the wrong numbers.
A node restarts in the middle of a load and re-reads what it had already read. The warehouse now holds some rows twice. Whether this matters depends entirely on a decision made long before the restart.
The batch is sized for throughput and is therefore too large for the promised latency. Or it is sized for latency and is therefore too small to load efficiently. The correct size is not a setting. It is a position on a curve that has to be found, against a service level that someone must first define.
None of this is wiring. All of it is judgment.
The Part That Does Not Generate
Models can generate error-handling and validation paths when requirements and examples are supplied. They may still miss domain-specific failures. Engineers must identify the relevant risks, test generated behavior and verify that operational controls work.
What the market pays for, then, was never the connection. It is knowing what happens after the connection succeeds. It is knowing which failures matter and which can be ignored. It is knowing what not to build, which is the most undervalued skill in the field and the one least likely to appear on any tool. Judgment of this kind is the residue of having been wrong before, in production, with consequences. It is not on a certification, and it is not in the model.
The engineer who is anxious about the tools being commoditized is anxious about the wrong asset. The tools were never the asset. The decision about when, whether, and how to use them was, and that decision still requires someone who has carried the cost of getting it wrong.
Routine connection boilerplate can be inexpensive to generate, but integration still has licensing, testing, security and operating costs. The economic value shifts toward verified reliability rather than disappearing entirely.
Written by Roland Wenzlofsky, founder of DWHPro and author of Teradata Query Performance Tuning. DWHPro has helped data warehouse practitioners for 15+ years.
Deciding what to change in your data platform?
Compare optimisation, modernisation and migration against your workload and business requirements.
