What is the Waterfall Model?
In the waterfall model, the data warehouse project is proceeding one phase after the other. Only when one phase is completed is the next phase started.
The following phases will be passed through:
- Requirements Gathering (more about why I don't like this term later)
- Verification (Testing, User Acceptance Test)
- Deployment, Operation, and Maintenance
What are the Problems with the Waterfall Model?
- The requirements must be predicted into the future (prediction of requirements up to some years ahead)
- The designer or developer might misinterpret the requirements, and it takes a long time before this is noticed due to the phased implementation
Usually, these problems are only detected in the verification phase (during testing).
Why do these problems exist?
The waterfall method assumes that the requirements are perfect.
They are usually not, except for very small projects. Therefore it is no wonder that most data warehouse projects fail or cause much more effort than planned.
The phased approach of the predictive model allows the responsibility for the faulty end result of the project to be given to the last link in the chain, which is usually the developer.
From my experience, I can say that this usually always happens.
The Waterfall method further assumes that the translation from one phase to the next is perfect. This is also usually not given.
Misinterpretations often remain undetected until the verification phase and often only become visible during testing.
The later it is discovered in the project that the implemented solution does not meet the requirements, the higher the costs:
Typical errors in the requirement gathering phase
- User requirements are accepted uncritically
A successful data warehouse project requires that user requirements are critically analyzed.
It is not about what the business user wants, but what he really needs:
“Whatever you want, we'll make it happen!”
The task of a good analyst is, therefore, to work this out together with the business users and to lead them in the right direction.
The rude awakening usually comes at a later stage of the project, when it turns out that what was specified has been implemented, but that this does not correspond to what is really needed.
My experience from more than 20 years in various roles is that the full responsibility for the implemented business requirements is shifted exclusively to the business users:
“Business wanted it like this”
No, dear project managers and analysts! You are the experts who should know how a data warehouse project is handled. Your task is to support the business user in working out the desired requirements.
- Mixing of functional requirements and non-functional requirements
This is a common mistake I see quite often. Only functional requirements have to be formulated with business users.
Non-functional requirements are in most cases part of a Data Warehouse project, but only to be discussed with the business user if this is a mandatory requirement with high importance which actually makes it a functional requirement. Here is one example:
A simple “the report which delivers the number of customers per month must be fast” is too vague, and nothing to be discussed therefore with business users. It's non-functional.
If the availability of the report must be given at a certain time, it must be specified in detail and mandatory in order to be a functional requirement: “The report must be available daily at 8 a.m.”. In this case, the functionality of the software is only given if we can deliver at 8 a.m. and we are dealing with a functional requirement.
- Unnecessary consideration of the design in the requirement phase
It may be necessary to consider possible design constraints already in the requirement phase. No question.
However, I often see that the design dominates the requirements. This restricts the requirements for no reason, and may not take into account options that would be good options.
“We will use the data model that we already have. Then it will be easier and faster to implement the project. And it'll be cheaper too. We adapt the requirements to fit this design and that's it! We can improve design later.”
- Using a waterfall model, but design and implementation are started before the requirements are complete
Dear project managers:
This is not a waterfall model. This is not Agile either. That's just crap. I consider this “Climbing up the waterfall”. This is what you will have to do at the end of the project to bring it somehow to an end.
If the project fails it's your fault, not the fault of the developer and also not the fault of the business users!
The framework and responsibilities are still those of a waterfall model. It may look agile to you, but it's not. This can only lead to problems. The project will almost certainly fail or involve a lot of extra work and money.
If you can't guarantee that the requirements are complete and correct, then consider doing that project agile. But don't insist on a waterfall model in which responsibilities are passed on like in a relay race and where the waterfall framework definitely is the wrong one.
- Leave design decisions to the business
Not taking responsibility for design decisions is never a good idea. The rule is: Business says what they want, but not how it should be done.
- Playing “Chinese Whispers” between business, project manager and developers
Many project managers see themselves as mediators between business users and developers. A completely wrong approach.
The exchange has to take place directly between the people who are business experts and development experts.
I see again and again how valuable information is lost from business users to the development team because project management is passing information back and forth.
As we have seen in the previous section, many things can go wrong already in the requirement phase. Now let's see how it goes on until we finally arrive at the “failed project”.
Typical errors in the further phases
If errors such as those mentioned above occur in the requirement phase, we are already on the sidetrack with our project.
The probability is high that your project will fail (you are not alone. According to research, up to 70% of data warehouse projects implemented using the waterfall method fail).
Passing on the requirements to the design team with simultaneous transfer of responsibility is a common procedure. Business users are usually no longer involved. They only come back into play when the User Acceptance Test takes place and it is too late.
The relay race continues, now it's the developers' turn. They are trying to deliver a result based on an incomplete specification and a design that does not meet the requirements.
Finally, after 3, 6, 9, or 12 months we have reached the testing stage. Only here it often turns out that the implementation does not meet the requirements:
Business to Project Lead:
“That's now what we wanted!”
Project Lead to Developer:
“Please check what the specification says about this topic!”
“These details were never specified”
From now on it becomes expensive to finish the project successfully. Often the next mistake is made right away:
Workarounds are implemented to keep the costs low.
Workarounds seem to be cheaper to implement but only shift the problems to the final phase of the project:
Operation & Maintenance
The Operations Team will be grateful…
From Waterfall to Agile
If you have recognized in my remarks that your projects are also running in this way, it would be appropriate to think about carrying out Agile projects.
By this, I mean to acquire an agile mindset. It is not about the use of certain tools or methods, but the handling of the project in close cooperation between the teams and the business users.
Which it certainly is not:
Ridiculous methods like the “Scaled Agile Framework” which has completely misunderstood the original idea of the agile mindset and is only one way to get money out of the pockets of solvent companies (keyword: “Certified SAFe® Agilist“).
Adding additional layers of management, processes, and removing autonomy from teams is not really the original idea of agile development.
But certification is big business. Therefore we will continue to see creative “agile” solutions.
I'm talking about dividing the project into small subtasks that are verified in close cooperation with the business users in order to detect undesirable developments early on. Pieces that go live step by step.
I am very surprised by how agile methods are currently modern again. I can still remember the beginnings of data warehousing: even if it wasn't called “agile” at the time, but projects were handled with an agile mindset.
Only through the enormous cost pressure waterfall became more and more dominant in data warehousing, because at the same time an extreme specialization of the project team members went along with it, whereby “seemingly” costs can be saved (developers are usually cheaper than project managers, etc.). We all know the story of outsourcing development to cheap locations and the disappointments that usually accompany it.
Decades later later companies realize that the execution of a data warehouse project on the assembly line is usually doomed to failure…and run in droves into the arms of the snake oil sellers who offer “agile overall concepts” which have nothing to do with the idea of an agile approach.
The waterfall method works best for predictive projects. Otherwise, consider going agile (but right).
What Is The Failure Rate Of Agile?
It is said that agile has a lower failure rate (as always there are many different numbers and opinions). It is extremely difficult to determine what the actual success rate is because much of what is advertised as “agile” is just a marketing scam.
I think one thing can be said for sure if done in the right way :
Because a project is done in small pieces and in close cooperation with business, the perception of project success by the customer is different than with the waterfall method.