What is the Waterfall Model?
In the waterfall model, the data warehouse project proceeds one phase after another. Only when one step is completed is the next phase started.
The following phases will be passed through:
- Requirements Gathering (more about why I’m not too fond of this term later)
- Verification (Testing, User Acceptance Test)
- Deployment, Operation, and Maintenance
What are the Problems with the Waterfall Model?
- The requirements must be predicted into the future (prediction of requirements up to some years ahead)
- Due to the phased implementation, the designer or developer might misinterpret the requirements, and it takes a long time before this is noticed.
Usually, these problems are only detected in the verification phase (during testing).
Why do these problems exist?
The waterfall method assumes that the requirements are perfect.
They are usually not, except for small projects. Therefore it is no wonder that most data warehouse projects fail or cause much more effort than planned.
The phased approach of the predictive model allows the responsibility for the project’s faulty result to be given to the last link in the chain, usually the developer.
From my experience, I can say that this usually always happens.
The Waterfall method further assumes that the translation from one phase to the next is perfect. This is also usually not given.
Misinterpretations often remain undetected until the verification phase and only become visible during testing.
The later it is discovered in the project that the implemented solution does not meet the requirements, the higher the costs:
Typical errors in the requirement gathering phase
- User requirements are accepted uncritically
A successful data warehouse project requires that user requirements are critically analyzed.
It is not about what the business user wants but what he needs:
“Whatever you want, we’ll make it happen!”
Therefore, the task of a good analyst is to work this out with the business users and lead them in the right direction.
The rude awakening usually comes at a later stage of the project, when it turns out that what was specified has been implemented but that this does not correspond to what is needed.
My experience from more than 20 years in various roles is that the full responsibility for the implemented business requirements is shifted exclusively to the business users:
“Business wanted it like this”
No, dear project managers and analysts! You are the experts who should know how a data warehouse project is handled. Our task is to support the business user in working out the desired requirements.
- Mixing of functional requirements and non-functional requirements
This is a common mistake. I often see only functional requirements have to be formulated with business users.
Non-functional requirements are, in most cases, part of a Data Warehouse project but only to be discussed with the business user if this is a mandatory requirement with high importance, making it a functional requirement. Here is one example:
A simple “the report that delivers the number of customers per month must be fast” is too vague and has nothing to be discussed with business users. It’s non-functional.
Suppose the availability of the report must be given at a particular time. In that case, it must be specified in detail and mandatory to be a functional requirement: “The report must be available daily at 8 a.m.”. n this case, the software’s functionality is only given if we can deliver at 8 a.m. We are dealing with a functional requirement.
- Unnecessary consideration of the design in the requirement phase
Considering possible design constraints already in the requirement phase may be necessary. No question.
However, I often see that the design dominates the requirements. This restricts the requirements for no reason and may not consider options that would be good.
“We will use the data model that we already have. Then it will be easier and faster to implement the project. And it’ll be cheaper too. We adapt the requirements to fit this design, and that’s it! e can improve design later.”
- Design and implementation are started using a waterfall model before the requirements are complete.
Dear project managers:
This is not a waterfall model. his is not Agile, either. hat’s just crap. We consider this “Climbing up the waterfall”. This is what you will have to do at the end of the project to bring it somehow to an end.
If the project fails, it’s your fault, not the developer’s fault, and not the responsibility of the business users!
The framework and responsibilities are still those of a waterfall model. It may look agile to you, but it’s not. This can only lead to problems. The project will almost certainly fail or involve extra work and money.
If you can’t guarantee that the requirements are complete and correct, consider doing that project agile. But don’t insist on a waterfall model in which responsibilities are passed on like in a relay race and where the waterfall framework is wrong.
- Leave design decisions to the business.
Not taking responsibility for design decisions is never a good idea. The rule is: Business says what they want, but not how it should be done.
- Playing “Chinese Whispers” between business, project managers, and developers.
Many project managers see themselves as mediators between business users and developers—a completely wrong approach.
The exchange has to take place directly between the people who are business experts and development experts.
I see again and again how valuable information is lost from business users to the development team because project management is passing information back and forth.
As seen in the previous section, many things can go wrong already in the requirement phase. Now let’s see how it goes until we finally arrive at the “failed project”.
Typical errors in the further phases
If errors such as those mentioned above occur in the requirement phase, we are already on the sidetrack with our project.
The probability is that your project will fail (you are not alone. According to research, up to 70% of data warehouse projects implemented using the waterfall method die).
Passing on the requirements to the design team with simultaneous transfer of responsibility is a standard procedure. Business users are usually no longer involved. They only come into play when the User Acceptance Test takes place, and it is too late.
The relay race continues. Now it’s the developers’ turn. They are trying to deliver a result based on an incomplete specification and a design that does not meet the requirements.
Finally, we have reached the testing stage after 3, 6, 9, or 12 months. Only here, it often turns out that the implementation does not meet the requirements:
Business to Project Lead:
“That’s now what we wanted!”
Project Lead to Developer:
“Please check what the specification says about this topic!”
“These details were never specified”
From now on, it becomes expensive to finish the project successfully. Usually, the next mistake is made right away:
Workarounds are implemented to keep the costs low.
Workarounds seem to be cheaper to implement but only shift the problems to the final phase of the project:
Operation & Maintenance
The Operations Team will be grateful…
From Waterfall to Agile
If you have recognized in my remarks that your projects are also running this way, it would be appropriate to consider carrying out Agile projects.
By this, I mean to acquire an agile mindset. It is not about using specific tools or methods but the handling of the project in close cooperation between the teams and the business users.
Which it certainly is not:
Ridiculous methods like the “Scaled Agile Framework” which has completely misunderstood the original idea of the agile mindset and is only one way to get money out of the pockets of solvent companies (keyword: “Certified SAFe® Agilist“).
Adding additional layers of management and processes and removing team autonomy is not the original idea of agile development.
But certification is big business. Therefore we will continue to see creative “agile” solutions.
I’m talking about dividing the project into small subtasks verified in close cooperation with the business users to detect undesirable developments early on. Pieces that go live step by step.
I am astonished by how agile methods are currently modern again. I can still remember the beginnings of data warehousing: projects were handled with an agile mindset, even if it wasn’t called “agile”.
The waterfall becomes more and more dominant in data warehousing through the enormous cost pressure. At the same time, an extreme specialization of the project team members went along with it, whereby “seemingly” costs can be saved (developers are usually cheaper than project managers, etc.). We all know the story of outsourcing development to cheap locations and the disappointments that typically accompany it.
Decades later later companies realize that the execution of a data warehouse project on the assembly line is usually doomed to failure…and run in droves into the arms of the snake oil sellers who offer “agile overall concepts” which have nothing to do with the idea of an agile approach.
The waterfall method works best for predictive projects. Otherwise, consider going agile (but right).
What Is The Failure Rate Of Agile?
It is said that agile has a lower failure rate (as always, there are many different numbers and opinions). It is tough to determine the success rate because much of what is advertised as “agile” is just a marketing scam.
I think one thing can be said for sure if done in the right way :
Because a project is done in small pieces and close cooperation with business, the customer’s perception of project success is different from that of the waterfall method.