Don’t Cut the Expertise That Makes AI Valuable Just to Please Shareholders

How AI multiplies engineering productivity, and why cutting experience can erase the benefit

Companies that understand AI will use it to multiply expertise, not eliminate it. The others will see only the dollar sign, replace experienced engineers with cheaper developers plus an LLM, celebrate the quarterly savings, and proudly call it transformation.

Artificial intelligence is often discussed in software and data engineering in terms of code generation. The usual argument is that AI can create SQL, Python, Spark code, configuration files, tests, documentation, deployment scripts, and many other implementation artifacts faster than a human can produce them manually.

This is certainly useful, but it does not describe the most important change in my own work.

The biggest productivity gain comes from something else: the cost of trying an idea has dropped significantly.

In the past, implementation effort was an important factor in deciding which ideas were worth pursuing. A diagnostic utility, a different ingestion pattern, an additional validation step, a parallel test environment, an alternative orchestration design, or a small internal tool could easily require several hours or even days of work before one could determine whether the idea was useful at all.

This meant that many technically sensible ideas were never tested.

The reason was not that they were poor ideas. The expected benefit was too small compared with the implementation effort, the project budget, and the available time.

AI has changed this relationship considerably for me.

Today, I can describe an idea to a coding agent, create a first implementation within minutes, execute it, observe the result, and decide whether the approach is worth continuing.

If the idea turns out to be wrong, the cost of discovering this is small.

That changes the engineering process much more fundamentally than faster code generation alone.

From Selecting One Idea to Testing Several

Consider a typical data engineering problem for which several technically valid solutions exist.

A transformation could run in SQL or Spark. Processing could happen in Snowflake, Databricks, a relational database, Python, or somewhere else in the architecture. A flow could be implemented as batch, micro-batch, or streaming. A task could use a platform-native feature or an external service.

In the past, it was often necessary to decide which option was most promising before implementation began.

The process looked approximately as follows:

Problem
   ↓
Possible approaches
   ↓
Select one
   ↓
Implement
   ↓
Test
   ↓
Adjust

The reason for selecting one approach early was simple: implementation was expensive.

With AI-assisted development, another process becomes possible:

Problem
   ↓
Several possible approaches
   ↓
Prototype several of them
   ↓
Measure and compare
   ↓
Select the most suitable design
   ↓
Complete the implementation

This reduces the need to predict the behavior of a solution before any implementation exists.

Instead, more decisions can be based on observation.

For me, this has become one of the most important changes in daily engineering work.

The Number of Ideas I Can Afford to Test Has Increased

Before AI, I often had ideas that I simply did not implement.

I might have thought that a small tool would be useful, that an additional validation would reduce risk, that another processing strategy might behave better, or that a temporary environment would help reproduce a production problem safely.

Then came the second thought:

Is this worth half a day or a full day of implementation?

Very often, the answer was no.

This created an invisible filter on engineering work.

Many useful things remained manual.

Many alternative designs were discussed but never prototyped.

Many small improvements were never implemented because the cost of creating them was higher than the expected benefit.

Today, I can often try those ideas immediately.

I describe what I want, let AI create a first version, run it, and see what happens.

Sometimes the result is useful.

Sometimes it is wrong.

Sometimes the whole idea turns out to be unnecessary.

But I usually know that after minutes instead of hours.

This makes me much more productive because I can explore more alternatives before committing to a final design.

Testing Is One of the Clearest Examples

One of the best examples from my own work is testing.

In many projects, I knew exactly what should ideally be tested.

The limitation was time.

If a project budget was tight and a deadline was approaching, the test scope was often reduced to what could realistically be implemented within the available effort.

Instead of creating twenty or thirty useful test scenarios, I might create five.

Instead of building a separate parallel environment to reproduce a production problem safely, I might decide that the setup effort was too high.

Instead of running a large number of automated tests against a database, I might limit the exercise to a few representative cases because writing the scripts, preparing the test data, executing the tests, collecting the results, and comparing them would have consumed too much of the remaining project budget.

This is not unusual.

Engineering has always been constrained by economics.

There is a difference between what would technically make sense and what a project can afford to implement.

AI has changed this boundary considerably for me.

Today, if I need a large number of test scripts, I can generate the first versions within minutes.

If I want to create a parallel test environment, AI can produce much of the required configuration, scripts, deployment definitions, data generators, and supporting tooling.

If I want to run hundreds or thousands of tests against a database, AI can help create the test cases, execution framework, logging, result comparison, and reporting.

Tasks that previously would have consumed several days can sometimes be reduced to hours, and individual parts of them to minutes.

This means that I no longer have to reduce the quality of an engineering exercise simply because producing the supporting implementation would be too expensive.

That is a much more important productivity improvement than generating one SQL statement or one Python function faster.

From Budget-Limited Testing to Risk-Driven Testing

The important change is not that AI knows how to test a system better than an experienced engineer.

It can certainly suggest test cases, and sometimes those suggestions are useful.

The bigger benefit is that it dramatically lowers the implementation cost of testing ideas that I already know would be valuable.

Before AI, the reasoning might have been:

These 30 tests would make sense.

Available time allows us to build 5.

Choose the most important 5.

Today, it can increasingly become:

These 30 tests would make sense.

Generate the framework and the test cases.

Review them.

Run all 30.

Add another 20 based on what we learned.

That changes the quality of the engineering process.

The better question changes from:

Which tests can we afford to build?

to:

Which risks should we test?

For me, this is a real improvement in engineering quality, not just productivity.

My Role Has Shifted

For many years, my normal workflow consisted of understanding a problem, designing the architecture, implementing the solution, testing it, and then correcting or optimizing the result.

Implementation consumed a significant part of the available time.

This has changed.

The current workflow is more often:

Understand the problem
        ↓
Design the solution
        ↓
Define implementation constraints
        ↓
Generate a first implementation
        ↓
Review
        ↓
Test
        ↓
Correct
        ↓
Validate the final design

The implementation has not disappeared.

What has changed is the amount of manual effort required to create the first version.

As a consequence, I spend less time writing code from scratch and more time reviewing, testing, comparing alternatives, and thinking about the architecture.

This is important because the most difficult questions in data engineering were never syntax questions.

They are questions such as:

• Where should processing take place?
• Which component should own the state?
• Should data be moved, or should computation move to the data?
• Is streaming really required, or is batch processing sufficient?
• Where should transaction boundaries be placed?
• How should failures and retries be handled?
• Which dependencies are really necessary?
• How will the solution behave when volume or concurrency increases?
• Is a more complex architecture justified by the business requirement?

AI can help implement the answer.

It does not remove the need to answer these questions.

I Spend More Time Thinking About How to Solve the Problem

This is probably the biggest benefit for me personally.

AI gives me more time to think.

Previously, architectural thinking and implementation competed for the same working time. Even when the design was clear, several hours could still be required to build the necessary scripts, transformations, tests, deployment files, or integration code.

Reducing this implementation effort creates additional time for design.

This allows me to investigate alternatives that I would previously have discarded because of cost.

It also allows me to spend more time simplifying a solution before it reaches production.

In this sense, AI does not merely accelerate an existing workflow.

It changes the distribution of effort within the workflow.

I spend less time thinking about how long it will take to implement something and more time thinking about how the problem should actually be solved.

Being Wrong Has Become Less Expensive

Engineering problems are rarely solved by selecting the perfect approach immediately.

Many decisions involve uncertainty.

A design may appear elegant but become unnecessarily complex during implementation. A Spark transformation may cause more data movement than expected. A database-native feature may be simpler operationally than an external service. A streaming solution may provide little business value compared with a much simpler micro-batch design.

Such observations have always been part of engineering.

The difference is the cost of reaching them.

If a prototype requires a full day of work, engineers naturally become conservative about which alternatives they test.

If the same prototype can be created in minutes, the cost of being wrong becomes much smaller.

This encourages experimentation.

For me, this is one of the most useful characteristics of AI-assisted development.

It allows a larger number of assumptions to be tested instead of discussed theoretically.

More Things Become Worth Automating

The same effect applies to smaller tasks.

Before AI, automation always had an economic threshold.

Assume a repetitive activity requires twenty minutes every week. If automating the task requires an entire day of development, manual execution may remain the more economical solution for a long time.

The calculation changes when the first implementation can be created in a few minutes.

Tasks that were previously too small to justify automation suddenly become reasonable candidates.

This applies to schema comparisons, metadata extraction, log analysis, data validation, test generation, file conversion, deployment checks, monitoring helpers, SQL generation, data migration utilities, and many other small engineering tasks.

Each improvement may save only a limited amount of time.

However, the cumulative effect of many such improvements is substantial.

This is one of the reasons why the productivity gain I experience is much larger than the simple reduction in time required to write a single piece of code.

AI Can Also Make a Bad Process Faster

The productivity gains are real, but I have also seen the opposite effect.

AI does not automatically improve the quality of a process. In many cases, it simply makes the existing process faster.

I have seen this quite clearly in the way some companies now create business requirements.

The idea sounds attractive.

Instead of having an analyst spend hours interviewing business users and manually writing a requirements document, the business is presented with a sequence of questions. The answers are entered into a system, AI processes them, and a professionally structured business requirements document is produced almost immediately.

I have seen this approach implemented.

It certainly reduces the effort required to write the document.

What I have not seen is the same improvement in the quality of the requirements.

The result is often the same poorly specified requirement that existed before, only written more quickly and presented more professionally.

If an important question was not asked, the generated document will still miss the answer.

If the business does not understand an exception, the exception will still be missing.

If two requirements contradict each other, AI may even make the problem less visible by expressing both in fluent and consistent language.

The resulting document can therefore look considerably better without the underlying requirement becoming considerably better (Some of you may recognize your own organization in this description).

The quality of the document improves much faster than the quality of the requirement.

The difficult part was never writing the document.

The difficult part was asking the right questions.

An experienced analyst knows when an answer is too vague, when another question is necessary, when two statements contradict each other, and when a business requirement has technical consequences that the business user may not initially consider.

AI can certainly assist with this process. It can summarize discussions, identify open points, suggest questions, and produce documentation.

But if AI is mainly used to automate the production of the document, the organization risks automating the least difficult part of requirements engineering.

The same applies to software development.

Generating code is becoming cheaper.

Understanding what should be built is not.

Code Review Has Become More Important

There is another consequence.

When more implementation is generated automatically, more implementation must be reviewed.

Generated code can be syntactically correct, well structured, and fully documented while still representing a poor architectural decision.

A data pipeline can execute successfully and still move unnecessary amounts of data.

A SQL statement can return the correct result and still consume excessive resources.

A Spark application can be technically correct while introducing avoidable shuffles.

A cloud-native implementation can work as designed while creating unnecessary operational complexity or cost.

A streaming design can process events correctly while solving a problem that did not require streaming in the first place.

The implementation can therefore be correct while the solution is wrong.

This distinction has become more important in my work since I started using AI more intensively.

Experience Becomes the Filter

This is also why I am skeptical when companies conclude that AI makes experienced engineers less important.

I can understand where the idea comes from.

If one engineer can create considerably more implementation with AI assistance, the required amount of manual development work can decrease.

I see this effect in my own work.

However, the conclusion that experience therefore becomes less important does not follow.

My experience is rather the opposite.

The more implementation I generate with AI, the more I depend on experience to evaluate it.

Someone still has to recognize that an apparently elegant design will create operational problems later.

Someone has to understand whether data should be processed in Snowflake, Databricks, a relational database, Spark, Python, or somewhere else.

Someone has to understand the difference between a solution that works for a test dataset and one that remains stable with much larger volumes.

Someone has to recognize when an architecture contains unnecessary components.

And someone has to decide whether a technically valid solution is worth operating for the next five or ten years.

AI can generate alternatives very quickly.

Experience is needed to determine which of those alternatives should survive.

The Dangerous Cost-Cutting Conclusion

One development I consider particularly short-sighted, and one I know from my own project experience, is the idea that AI makes it possible to replace experienced data engineers with much cheaper, less experienced developers.

The calculation is attractive.

An experienced engineer may have a relatively high daily rate.

A lower-cost offshore developer may cost 30% of that rate.

Give the cheaper developer access to an AI coding tool, and the difference in experience is supposedly compensated by the model.

On a spreadsheet, this can look convincing.

In practice, I think it misunderstands where the value of AI actually comes from.

The productivity gain I experience does not come from AI replacing my knowledge.

It comes from AI multiplying it.

I can ask AI to create a large test framework quickly because I already know which risks I want to test.

I can ask it to prototype several architectures because I understand the trade-offs I want to compare.

I can reject generated code quickly because I recognize when an apparently reasonable implementation does not fit the system.

I can ask useful follow-up questions because I know which questions matter.

Remove that experience and the situation changes substantially.

An Experienced Engineer Does Not Only Know More Answers

An inexperienced developer can certainly ask an LLM to generate a pipeline, optimize a query, create a Spark job, design a data model, or propose a cloud architecture.

The result may even look impressive.

But the difficult part is not producing an answer.

The difficult part is recognizing which questions should have been asked before the answer was produced.

If nobody asks about concurrency, the proposed solution may ignore concurrency.

If nobody describes the expected data volume, the architecture may be designed for the wrong scale.

If nobody asks about partial failure, the happy path may be implemented perfectly while recovery remains undefined.

If nobody asks about cost, the generated architecture may work technically while becoming economically unattractive.

If nobody recognizes that data distribution, transaction scope, network movement, security, or operational complexity could become the dominant problem, the model may never be pushed in that direction.

An experienced engineer does not only know more answers.

Experience changes the questions.

After enough years working with production systems, one develops a habit of questioning assumptions that are not visible in the original requirement.

What happens when the volume increases?

What happens when two processes run at the same time?

What happens when a message arrives twice?

What happens when a schema changes?

What happens when one component succeeds, and another fails?

What happens when a retry executes after a partial commit?

How expensive will this become?

Who operates it when it fails?

Can the same problem be solved with one component instead of four?

AI can help answer these questions.

Someone still has to ask them.

Cutting Experience Removes the Part AI Needs Most

This is why replacing experienced engineers with much cheaper, less experienced developers equipped with AI can remove exactly the expertise that makes AI valuable.

The organization retains implementation capacity.

It removes judgment.

This may work for clearly defined implementation tasks.

If the requirement is precise, the architecture already exists, the interfaces are stable, the expected behavior is well documented, and the task consists mainly of implementation, lower-cost development supported by AI can certainly be economical.

But many data engineering problems are not like that.

The requirement is incomplete.

The data are inconsistent.

Several systems are involved.

The correct architecture is not obvious.

Operational constraints are discovered during implementation.

Performance depends on workload characteristics that were never included in the requirement.

In such situations, reducing the decision to cost per developer-day is misleading.

The cheapest implementation resource is not necessarily the cheapest way to build the system.

The Savings Are Immediate, the Cost Often Comes Later

I can understand why this strategy is attractive to management.

The savings are visible immediately.

Headcount costs fall.

Average daily rates decline.

The financial effect appears in the current reporting period.

The consequences of weaker engineering judgment are much harder to place on the same spreadsheet.

They may appear months or years later as higher cloud consumption, unnecessary data movement, unreliable pipelines, production incidents, duplicated functionality, difficult migrations, technical debt, performance problems, or systems that need to be redesigned much earlier than expected.

I suspect that this is one reason such decisions can be attractive in organizations under pressure to demonstrate short-term efficiency to investors and shareholders.

The savings are measurable now.

The architectural cost belongs to the future.

This is not an argument against offshore development.

Excellent engineers work everywhere, and geography says little about engineering quality.

The problem is treating a lower daily rate as a substitute for experience and assuming that access to AI automatically closes the gap.

It does not.

AI Changes Team Economics, but Perhaps in the Opposite Direction

AI certainly changes the economics of engineering teams.

I can already see that one experienced engineer can accomplish more implementation work than before.

Teams may become smaller.

Some repetitive activities will require fewer people.

But this leads me to a different conclusion.

If AI multiplies the productivity of an experienced engineer, then experience may become more economically valuable rather than less.

The engineer’s architectural decisions can now be turned into working implementations much faster.

More ideas can be tested.

More risks can be covered.

More generated code can be reviewed.

More repetitive work can be delegated.

The leverage of experience increases.

The model I see in my own work is therefore not:

Expensive senior engineer
        ↓
replace with
        ↓
cheap developer + AI

It is closer to:

Experienced engineer
        +
AI implementation capacity
        ↓
more experimentation
broader testing
faster implementation
more architecture time
more review capacity

AI reduces the cost of implementation.

It does not reduce the value of knowing what should be implemented.

In many cases, it increases it.

When AI Is Reduced to a Chatbot

There is another limitation I currently see in many large organizations.

The issue is not the capability of AI itself, but the way access to it is restricted.

The reasons are understandable.

Data protection, confidentiality, intellectual property, regulatory requirements, and security all matter. Companies cannot simply allow employees to send arbitrary source code, customer data, production logs, or internal documents to any external AI service.

The problem starts when the response to these risks is to reduce AI to a very narrow set of approved use cases.

I have seen environments where employees technically have access to AI, but only in a heavily restricted form.

They can ask questions, summarize text, or perform a few predefined tasks.

What they cannot do is let AI work with real project files, modify code across a repository, generate and execute test frameworks, create temporary environments, inspect logs, build tooling, or iterate quickly on an implementation.

At that point, much of the productivity gain disappears.

AI becomes a chatbot.

That is useful, but it is not where I experience the largest benefits.

The biggest gains come when AI can participate in the engineering process itself.

If I have an idea for a test framework, I can try it.

If I want to create a temporary parallel environment, I can create it.

If I want to compare several implementation approaches, I can prototype them.

If I see a repetitive task that could be automated, I can automate it without first having to justify days of development effort.

When AI is restricted to predefined prompts and isolated tasks, this exploratory part disappears.

The employee is technically allowed to use AI, but not to use it as an extension of their engineering capacity.

The result can be paradoxical.

A company may invest heavily in AI, announce an AI strategy, provide employees with approved tools, and still capture only a small fraction of the potential productivity gain.

The technology is available.

The creative use of it is not.

This does not mean that data protection should be ignored.

Quite the opposite.

Companies need controlled environments, clear rules, proper access separation, approved models, auditability, and protection of sensitive information.

But there is a large difference between controlling AI usage and making useful AI-assisted engineering practically impossible.

In my experience, the strongest productivity gains come when experienced engineers are given enough freedom to experiment within safe boundaries.

We May Need to Rethink What Engineers Are Taught

The increasing abstraction created by AI also raises a broader question about computer science and informatics education.

Software engineering has continuously moved away from the bare metal.

Assembler exposed processors and memory directly.

Languages such as C provided a higher level of abstraction while still leaving many hardware and memory concepts visible.

Object-oriented languages raised the abstraction further.

Libraries, frameworks, managed runtimes, databases, distributed systems, and cloud platforms continued the same development.

Each step made engineers more productive by hiding details that no longer needed to be handled manually.

Conceptually, the progression looks something like this:

Machine code
    ↓
Assembler
    ↓
C and similar languages
    ↓
Object-oriented languages
    ↓
Libraries and frameworks
    ↓
Managed platforms and cloud services
    ↓
AI-generated implementation

AI is therefore not the first abstraction layer in computing.

But there is an important difference.

A framework may hide its internal implementation, but the rules of the system are normally documented.

A database optimizer is extremely complex, but we can inspect execution plans, collect statistics, measure runtime behavior, and understand the principles that influence its decisions.

A compiler performs transformations that a developer did not explicitly request, but the relationship between source code and generated machine code remains based on a well-defined technical process.

Large language models are different.

We can understand transformers, tokenization, embeddings, attention, training, and inference.

But as users of a particular model, we generally do not know the complete training material on which it was built.

We do not know which examples influenced a particular answer.

We do not know which patterns were overrepresented.

We do not know what relevant material may have been absent.

And in many situations we cannot reconstruct why one implementation was generated instead of another equally plausible one.

This makes the abstraction qualitatively different from many of the abstractions that came before it.

The Black Box Is Now Part of the Development Process

When I ask an AI agent to produce a data pipeline, I receive an implementation that may look entirely reasonable.

What I usually do not know is where the underlying pattern came from.

Was it influenced by official product documentation?

A commonly used open-source implementation?

A five-year-old blog post?

A Stack Overflow answer?

A repository containing poor but frequently copied code?

Some combination of all of them?

The generated answer may represent a very common solution.

That does not mean it is the correct solution for my system.

This is why I increasingly treat AI output as a proposal rather than an answer.

It has to be checked against architecture, documentation, measurements, expected workloads, and the actual behavior of the target platform.

The more implementation AI produces for me, the more important this validation becomes.

Does Traditional Education Still Put the Weight in the Right Places?

Traditional computer science and informatics curricula invest considerable effort in mathematics, statistics, algorithms, data structures, computer architecture, programming languages, and theoretical foundations.

I do not think these subjects become unnecessary.

The opposite may be true.

But the way they are weighted may need to change.

If future engineers spend less time manually implementing algorithms and more time directing systems that generate implementations, then education must also prepare them to evaluate those implementations.

System architecture becomes important.

Distributed-system behavior becomes important.

Data modeling remains important.

Performance measurement remains important.

Experimental design becomes important.

Security remains important.

Failure analysis remains important.

And statistics may become even more relevant.

When engineers increasingly work with probabilistic systems, they need to understand uncertainty, sampling, bias, confidence, measurement, and the difference between a plausible result and a well-supported one.

Mathematics remains valuable as well, not necessarily because every engineer will manually implement the underlying algorithms, but because mathematical reasoning helps determine whether generated results make sense.

The required knowledge does not disappear.

It moves toward validation and judgment.

Knowing Less About the Implementation Requires Knowing More About the System

This sounds contradictory, but I think it describes the direction well.

As implementation becomes increasingly abstracted away, engineers may need fewer low-level implementation details for everyday work.

At the same time, they need a stronger understanding of the system as a whole.

If AI writes most of a transformation, the engineer may not need to type every line.

But somebody still needs to understand data distribution, concurrency, consistency, performance, failure behavior, security, cost, and the business meaning of the data.

If AI creates a statistical test, somebody still has to know whether the test is appropriate.

If AI proposes an architecture, somebody still has to recognize whether the components fit together.

If AI generates code using a particular framework, somebody still has to understand the operational consequences of that framework.

The higher the abstraction moves, the less useful it becomes to judge an engineer only by how quickly they can produce code manually.

Judgment becomes more important.

The Experienced Engineer Remains the Brain

The easiest way for me to describe the relationship is that AI provides implementation capacity while the experienced engineer provides direction.

AI can produce alternatives.

It can write code.

It can create tests.

It can build supporting tools.

It can generate documentation.

It can suggest approaches that I had not considered.

But somebody still has to understand the complete system.

Somebody has to recognize which risks matter.

Somebody has to understand the business context.

Somebody has to decide whether a technically attractive solution will remain maintainable and economically reasonable.

Somebody has to know when an AI-generated answer is plausible but wrong.

This is where experience becomes more important rather than less.

The more implementation capacity one engineer controls, the greater the impact of that engineer’s decisions.

In that sense, I do not see AI replacing experienced data engineers.

I see it moving their work upward.

Less time is spent producing every implementation detail manually.

More time is spent on architecture, review, experimentation, validation, and deciding what should actually be built.

What Has Changed Most for Me

When I compare my work today with the way I worked before AI-assisted development, the biggest difference is not how fast I can write code.

It is how many ideas I can afford to test.

Ideas that would previously have remained unimplemented because the expected benefit was too small can now become working prototypes.

Alternative architectures can be compared instead of only discussed.

Small repetitive activities can be automated.

One-time diagnostic tools become economical to build.

Parallel test environments become feasible where previously the setup effort might have prevented them.

Test scopes no longer have to be reduced as aggressively simply because creating the necessary scripts would consume too much of the project budget.

Approaches that turn out to be wrong can be discarded without losing a large amount of time.

This has made me considerably more productive.

At the same time, it has moved my work further toward the part of engineering that I consider most important: understanding the problem, designing the solution, evaluating alternatives, testing the relevant risks, and deciding what should finally become part of the architecture.

The danger is therefore not that AI makes experienced engineers obsolete.

The danger is that organizations see the reduction in implementation effort, remove the experience that gives the implementation direction, and then use AI to produce poor decisions faster and at lower apparent cost.

AI can make engineering substantially more productive.

Whether it makes the resulting systems better still depends on the people steering it.