Data Engineering Is Not Dying. But Management Is Doing Its Best to Kill It.

How corporate cost-cutting is creating the exact problem AI was supposed to solve.

Let me describe a scene playing out in companies across Europe and North America right now.

A management team sits in a meeting room. The CFO presents a slide. It shows the cost of the onshore data engineering team — experienced people who understand the business, know the systems, and can both write code and design architecture. Next slide: a proposal to replace them with an offshore team at one-third the cost. Per head. On paper. Everyone nods. The decision is made before the coffee gets cold.

Six months later, the same management team sits in the same room, wondering why their Databricks costs tripled, why nobody understands how the pipelines work, and why the new offshore team keeps asking questions that the people who were fired could have answered in thirty seconds.

I wish this were exaggerated. It’s not. I see it happening everywhere. And the timing makes it particularly absurd, because these companies are doubling down on an offshoring model at the exact moment AI is about to make that model obsolete.

Let me explain why.

The Model That Worked — Past Tense

For twenty years, the economics of data engineering were simple. You had expensive work (writing ETL pipelines, SQL transformations, Informatica workflows, PySpark jobs) and you had expensive people in expensive countries doing it. A senior data engineer in Germany or the US costs €/$150,000–200,000 per year. The same skill set in India costs $30,000–60,000. The work was structured enough to offshore: an architect writes specifications, and an offshore team implements them.

It worked. Not perfectly — time zones, communication overhead, context loss, the never-ending cycle of “that’s not what I meant” in the specification review — but well enough. Companies saved money. Consultancies like TCS, Infosys, and Wipro built empires on this arbitrage. Millions of engineers in Bangalore, Hyderabad, and Pune built careers translating specifications into code.

Here’s the thing, though. This model worked because human labor was the only way to write code. The offshore engineer was cheaper than the onshore engineer, but both were dramatically more expensive than the next option.

Enter AI, Stage Left

An AI coding assistant costs roughly $20–200 per month. Per developer. Not per hour. Per month.

It generates a working PySpark in seconds. It converts Teradata SQL to Spark SQL while you wait. It builds Airflow DAGs from natural language descriptions. It writes DBT models. It documents your pipeline. It suggests optimizations. It doesn’t need onboarding, doesn’t have visa complications, doesn’t take holidays, and doesn’t resign for a 30% raise at a competitor after eighteen months.

Is it perfect? No. Does it replace all human judgment? Absolutely not. We’ll get to what it can’t do.

But here’s the math that should be keeping every outsourcing executive awake at night:

The work that was offshored — translating specifications into code — is exactly the work AI automates best. Structured. Pattern-based. Well-documented. Clear inputs, clear outputs. This is not a coincidence. This is the same reason it was offshorable in the first place. If a task is structured enough to be put in a specification document and handed to someone who doesn’t know your business, it’s structured enough for AI to handle.

The offshore model competed on cost against onshore labor. AI competes on cost against all labor. And AI wins.

Now Watch What Management Does Next

So here we are. AI is automating the code layer. The smart move would be to recognize this, retain the people who understand architecture and business context, and let AI absorb the implementation work. Smaller teams, more senior, more capable.

That’s not what’s happening.

What’s happening instead — and I genuinely cannot believe I have to write this — is that management teams are laying off their experienced generalists and replacing them with more offshore coders. They’re cutting the architecture layer and doubling down on the code layer. The exact layer AI is about to eat.

Let me walk you through the decision process, because it’s a masterclass in optimizing for the wrong decade.

The management team sees economic pressure. They look at headcount. They see a senior data engineer in Europe who costs €170,000. She’s been with the company for ten years. She knows every pipeline, every business rule, every workaround, every reason why that one table has a weird column name from a merger in 2019. She can write code and sit down with the business to explain why the proposed architecture won’t work.

Management sees: €170,000.

They hire three offshore engineers at €40,000 each. Total: €120,000. Savings: €50,000. The CFO puts this on a slide labeled “efficiency.”

Here’s what actually happens.

Months 1–6: The offshore team takes over. Things more or less work. The documentation left by the fired generalist is enough to keep the lights on. Management feels vindicated. “See? Same output, lower cost.”

Months 7–12: The questions start. The offshore team encounters a pipeline they don’t understand. They can’t find documentation for the business logic. They make assumptions. Some assumptions are wrong. Data quality issues start appearing in reports. The business team notices. The offshore team asks for more specifications. Nobody onshore has the context to write them, because that person was fired.

Months 12–18: AI coding tools have matured further. The offshore team’s implementation work — the very work they were hired for — is increasingly automatable. Management starts wondering why they’re paying €120,000 for three people to do work that an AI tool could do for €600 per year. Meanwhile, architectural problems are piling up. Databricks costs are rising. Nobody understands which pipelines can be consolidated. Nobody knows why the cluster auto-scaling is behaving strangely. There’s no one left who understands the system at an architectural level.

Months 18–24: Management realizes they need a senior architect. They go to market. The same profile they fired 18 months ago now costs €200,000 because everyone else fired their generalists too, and the good ones are locked into contracts at companies that were smarter. The new hire needs six months to rebuild the institutional context that was walked out the door. Total cost of the “efficiency” decision: more than if they’d simply kept the original person and given them AI tools.

This is not a hypothetical scenario. I’m watching it happen at multiple companies simultaneously. It’s the corporate equivalent of selling your roof because the sun is shining, then acting surprised when it rains.

What Management Gets Wrong About AI and Offshoring

The root error is treating AI as just another reason to cut headcount, rather than recognizing it as a force that changes which headcount matters.

Management hears “AI can write code” and translates it to “we need fewer people.” That’s half right. You do need fewer people writing code. But you need more people — or at least the same people — who understand architecture, evaluate AI output, and make design decisions.

The generalist they fired wasn’t expensive because she wrote code. She was expensive because she understood the system well enough to make decisions that saved the company millions in computing costs, data quality incidents, and architectural dead ends. The code was 30% of her value. The architecture was 70%.

By firing her, management kept the 30% (offshore coders) and discarded the 70% (architectural judgment). Then AI automated the 30%. Now they have neither.

There’s also a deeper misunderstanding about what offshoring and AI optimize for. Offshoring optimizes for labor cost — same work, cheaper humans. AI optimizes for labor elimination — same work, no humans. These are not the same strategy. You cannot combine them by saying “let’s offshore AND use AI.” If AI does the work, there’s nothing left to offshore.

Companies that offshore their implementation teams and then adopt AI tools end up paying for two solutions to the same problem. The offshore team writes code. The AI tool also writes code. Someone onshore has to review both. The “efficiency” is now three layers of cost, where one would do.

The Split That Actually Matters

Let’s set aside management decisions for a moment and look at what’s structurally happening to the profession.

Data engineering is splitting into two layers. This isn’t a forecast. It’s already happening.

The code layer: Writing transformations. Building pipelines. Debugging SQL. Wiring up orchestration. Translating requirements into working code. This layer was commoditized first by offshoring, now by AI. Its economic value is approaching zero — not because the work doesn’t matter, but because the cost of performing it is collapsing.

The architecture layer: Deciding how data flows. Choosing platforms. Designing for cost, performance, and reliability. Understanding trade-offs between materialization and virtualization, between streaming and batch, between lakehouse and warehouse. Knowing why a system was designed a certain way, not just how it was built. This layer is becoming more valuable because AI can’t do it, offshoring can’t commoditize it, and every organization desperately needs it.

For 20 years, these layers were bundled into a single job description. The same person who designed the architecture also wrote the code. That made sense when code was expensive to produce. Now that code is cheap (via AI), the bundle breaks. And the two layers have very different economic futures.

What AI Can Actually Do — Honestly

Let’s be precise, because the hype isn’t helping anyone.

Code generation works. AI assistants generate working PySpark, SQL, dbt models, and Airflow DAGs from natural language descriptions. Not always perfect. Often needs adjustment. But correct often enough that the human role shifts from writing to reviewing.

Data profiling is automated. AI examines a dataset and describes its structure, types, null patterns, and statistical distributions in seconds. Work that used to take an afternoon.

Documentation is generated. AI reads a pipeline and produces documentation. It also reviews existing documentation and flags outdated sections.

Platform translation is emerging. AI converts Teradata SQL to PySpark, translates Informatica mappings into Airflow DAGs, and migrates stored procedures between SQL dialects. Not perfectly. But well enough that a migration project that used to need 50 offshore engineers for 6 months might now need 10 engineers for 3 months. The outsourcing industry should be terrified by this.

Testing and monitoring are improving. AI generates data quality checks, suggests edge cases, and spots anomalies in pipeline behavior.

All of this is real. All of this is in production today. None of it is hype.

What AI Cannot Do — Also Honestly

AI cannot decide whether you need Spark or Snowflake. It can list the differences. The actual decision depends on your workload patterns, team skills, vendor relationships, budget, compliance requirements, and strategic direction. These are judgment calls that require organizational context.

AI cannot design your data architecture. Batch or streaming? Materialize or virtualize? Lakehouse or warehouse? These are trade-offs that depend on cost, latency, team capability, and business requirements. AI proposes options. Humans weigh them.

AI cannot understand your business context. It can calculate revenue per region. It can’t tell you the region definitions changed because of a merger, the finance team counts deferred revenue differently, and the CEO only cares about year-over-year using the old methodology.

AI cannot evaluate its own output. This is the critical one. AI generates a syntactically correct Spark job that triggers a massive, unnecessary shuffle. The code compiles. The tests pass. The architecture is terrible. Spotting this requires understanding how distributed systems actually work.

Which brings us back to the management problem. The person who spots the architectural flaw in AI-generated code is the generalist they just fired.

The Cost Equation Nobody Wants to Hear

Let’s compare three models with real numbers.

The old onshore model: Six senior engineers x €170,000 = €1,020,000/year. Expensive. But they know the systems, the business, and the architecture.

The offshoring model (what management is choosing now): Two onshore leads x €170,000 + eight offshore engineers x €40,000 = €660,000/year. Looks cheaper. But add communication overhead, specification writing, review cycles, quality issues, and knowledge loss. Real cost: probably €800,000+ when you include rework and incidents. And the offshore team is doing work that AI will automate within two years.

The AI-augmented model (what smart companies are choosing): Three senior engineers x €180,000 + AI tools x €2,000/year = €542,000/year. Fewer people, more senior, all architecturally capable, all using AI for the code layer. No specification documents, no handoff delays, no time zone management. Higher output than either of the previous models.

The AI-augmented model is cheaper than offshoring and produces better results. Let that sink in. The “cost-saving” decision to offshore is actually more expensive than the alternative.

But it gets worse. The offshoring model requires ongoing management overhead (coordination, specification, review). The AI-augmented model doesn’t. The offshoring model loses architectural knowledge when onshore people leave. The AI-augmented model retains it because the architects are doing the work directly. The offshoring model will face another cost restructuring when AI fully automates the offshore team’s work. The AI-augmented model is already adapted.

Management teams choosing offshoring right now are selecting the most expensive option and calling it efficiency.

What This Means for Different People

If You’re an Engineer in India or Eastern Europe

The implementation role is shrinking. This is not your fault, and it’s not about your talent — it’s about economics. The strategic response is to move toward architectural understanding, domain expertise, and AI-adjacent skills. An engineer in Bangalore who deeply understands Spark’s execution model and can design data architectures is just as valuable as one in Berlin. The value has shifted from volume (many people writing code) to depth (fewer people making architectural decisions). The depth is geography-independent.

If You’re an Engineer in the US or Western Europe

You’re not safe just because you’re onshore. If your work is primarily implementation, you’re as replaceable as anyone. Your protection isn’t your passport — it’s your ability to work at the intersection of technology, architecture, and business context. Invest in understanding how platforms work, not just how to write code for them.

If You’re in Management

Stop. Think about what you’re actually cutting.

When you lay off a generalist who understands both code and architecture, you’re not removing a cost line. You’re removing a capability that took years to develop and cannot be replaced by hiring three people who lack it. AI will handle the code. You need the architecture. If you fire the architect and keep the coders, you’ve chosen to keep the part that’s being automated and discard the part that isn’t.

Retain your generalists. Give them AI tools. Let the AI absorb the implementation work. Reduce your offshore team gradually as AI takes over that workload. Your total cost will be lower, your quality will be higher, and your architectural knowledge stays in-house. Or ignore this, offshore everything, and budget for the emergency architect hiring in 18 months. Your choice.

If You’re in the Outsourcing Industry

The model of deploying large teams of implementation engineers at a cost advantage is structurally challenged. You cannot compete on cost with software that works for €50/month. The firms that survive will pivot from selling implementation labor to selling architectural expertise and AI integration services. This is a higher-value but lower-volume business. Fewer people, different skills, different margins. The industry will be smaller but more specialized.

Where Does This All Land

The data engineering profession isn’t dying. It’s being compressed. Fewer people, doing higher-value work, augmented by AI that handles the mechanical parts.

The tragedy is that, in many organizations, management is accelerating the wrong transition. Instead of embracing the compression — smaller teams, more architectural, AI-augmented — they’re clinging to the last decade’s playbook. Cut onshore costs. Offshore the implementation. Celebrate the savings. Ignore the fact that the implementation layer they’ve offshored is the same layer AI is about to eliminate.

In two years, these companies will be hiring expensive architects to fix problems that didn’t need to exist. The fired generalists will be happily employed elsewhere. The offshore teams will struggle against AI tools that can do their work in seconds. And management will be in the same meeting room, wondering what went wrong.

I could tell them. But they probably won’t ask — because asking costs €170,000 a year and they found someone cheaper.

Originally published on Medium.