What is the Waterfall Model?
The waterfall model facilitates the sequential progression of a data warehouse project. Each phase must be concluded before the subsequent stage commences.
The following stages will be navigated:
- Requirements Gathering (more about why I’m not too fond of this term later)
- Design
- Implementation
- Verification (Testing, User Acceptance Test)
- Deployment, Operation, and Maintenance
What are the Problems with the Waterfall Model?
- The requirements must be predicted into the future (prediction of requirements up to some years ahead)
- Due to the phased implementation, the designer or developer might misinterpret the requirements, which takes a long time to notice.
Typically, these issues are identified solely during the verification stage of testing.
Why do these problems exist?
The waterfall approach presupposes flawless requirements, which is seldom the case except for minor undertakings. Hence, it is not surprising that a majority of data warehouse ventures either flounder or necessitate greater exertion than anticipated.
The predictive model’s phased approach assigns responsibility for a faulty project outcome to the developer, who typically represents the final link in the chain.
In my experience, this is typically the case.
The Waterfall methodology assumes flawless progression between phases, which is typically unrealistic.
Misinterpretations may go unnoticed until the verification stage and only manifest during testing.
The later a project discovers that the implemented solution falls short of the requirements, the greater the associated costs.
Typical errors in the requirement-gathering phase
- User requirements are accepted uncritically.
A successful data warehouse project requires that user requirements are critically analyzed.
It is not about what the business user wants but what he needs:
“Whatever you want, we’ll make it happen!”
Therefore, the task of a good analyst is to work this out with the business users and lead them in the right direction.
The rude awakening usually comes later in the project when it turns out that what was specified has been implemented but does not correspond to what is needed.
My experience from more than 20 years in various roles is that the full responsibility for the implemented business requirements is shifted exclusively to the business users:
“Business wanted it like this”
No, dear project managers and analysts! You are the experts who should know how a data warehouse project is handled. Our task is to support the business user in achieving the desired requirements. - Mixing of functional requirements and non-functional requirements
This is a common mistake. I often see only functional requirements have to be formulated with business users.
Non-functional requirements are, in most cases, part of a Data Warehouse project but are only to be discussed with the business user if this is a mandatory requirement with high importance, making it a functional requirement. Here is one example:
A simple “the report that delivers the number of customers per month must be fast” is too vague and has nothing to be discussed with business users. It’s non-functional.
Suppose the availability of the report must be given at a particular time. In that case, it must be specified in detail and mandatory to be a functional requirement: “The report must be available daily at 8 a.m.”. n this case, the software’s functionality is only given if we can deliver at 8 a.m. We are dealing with a functional requirement. - Unnecessary consideration of the design in the requirement phase
Considering possible design constraints already in the requirement phase may be necessary. No question.
However, I often see that the design dominates the requirements. This restricts the requirements for no reason and may not consider options that would be good.
“We will use the data model that we already have. Then it will be easier and faster to implement the project. And it’ll be cheaper too. We adapt the requirements to fit this design, and that’s it! e can improve design later.” - Design and implementation are started using a waterfall model before the requirements are complete.
Dear project managers:
This is not a waterfall model. his is not Agile, either. That’s just crap. We consider this “Climbing up the waterfall”. You will have to do this at the end of the project to bring it to an end.
If the project fails, it’s your fault, not the developer’s fault, and not the responsibility of the business users!
The framework and responsibilities are still those of a waterfall model. It may look agile to you, but it’s not. This can only lead to problems. The project will almost certainly fail or involve extra work and money.
If you can’t guarantee the requirements are complete and correct, consider doing that project agile. But don’t insist on a waterfall model in which responsibilities are passed on, like in a relay race and where the waterfall framework is wrong. - Leave design decisions to the business.
Not taking responsibility for design decisions is never a good idea. The rule is: Business says what they want, but not how it should be done. - Playing “Chinese Whispers” between businesses, project managers, and developers.
Many project managers see themselves as mediators between business users and developers—a completely wrong approach.
The exchange has to take place directly between the people who are business experts and development experts.
I repeatedly see how valuable information is lost from business users to the development team because project management passes information back and forth.
“You don’t like surrogate keys? No problem, we like to leave them out for you.”
“You like storing your data in a different data model? You decide, of course!”
As seen in the previous section, many things can go wrong already in the requirement phase. Now let’s see how it goes until we finally arrive at the “failed project”.
Typical errors in the further phases
As aforementioned, project derailment may occur due to errors in the requirement stage.
The probability is that your project will fail (you are not alone. According to research, up to 70% of data warehouse projects implemented using the waterfall method die).
Delegating responsibility and requirements to the design team is a customary practice. Usually, business users are excluded until the User Acceptance Test, which can be tardy.
The developers are currently participating in the relay race. They face the challenge of producing a deliverable based on insufficient specification and a design that fails to fulfill the necessary criteria.
After a period of 3, 6, 9, or 12 months, we have arrived at the testing phase. At this juncture, it frequently becomes apparent that the implementation fails to meet the necessary standards.
Business to Project Lead:
“That’s now what we wanted!”
Project Lead to Developer:
“Please check what the specification says about this topic!”
Developer:
“These details were never specified”
Henceforth, successfully completing the project will require a significant financial investment. Unfortunately, it is common for workarounds to be employed immediately to minimize costs. Such measures may appear cost-effective but defer problems to the project’s final phase: Operation and Maintenance.
The Operations Team will be grateful…
From Waterfall to Agile
If my comments have made you aware that your projects follow a similar pattern, it may be prudent to consider implementing Agile methodology.
To clarify, the goal is to cultivate an agile mindset. This involves collaborating closely with both the project teams and business users rather than relying solely on particular tools or methods.
The “Scaled Agile Framework” is a flawed approach that fails to align with the core principles of agile ideology. It appears to be a tool for capitalizing on the financial stability of corporations, as evidenced by the use of the term “Certified SAFe® Agilist.”
Deviation from the original concept of agile development involves management layering and process implementation that diminishes team autonomy.
But certification is a big business. Therefore, we will continue to see creative “agile” solutions.
Dividing the project into smaller subtasks in close collaboration with business users can help detect any undesirable developments early on. These subtasks can be implemented step by step and released gradually.
I am impressed by the resurgence of agile methods. I recall the early days of data warehousing, where projects were approached with an agile mindset, albeit without the label “agile.”
Due to significant cost pressures, the waterfall approach has gained greater prominence in data warehousing. This has led to a high degree of specialization among project team members in order to reduce costs (as developers are typically less expensive than project managers). We are all familiar with the risks associated with outsourcing development to low-cost locations and the predictable disappointments that often result.
Decades later later companies realize that the execution of a data warehouse project on the assembly line is usually doomed to failure…and run in droves into the arms of the snake oil sellers who offer “agile overall concepts” which have nothing to do with the idea of an agile approach.
For projects that require predictability, the waterfall method is the optimal approach. Otherwise, it is advisable to opt for agile methodology, but with careful consideration.
What Is The Failure Rate Of Agile?
Agile is reputed to have a lower failure rate, although there is a range of figures and viewpoints. Establishing its success rate is challenging because numerous instances labeled “agile” are merely a form of deceitful promotion.
One thing is certain if executed correctly:
The customer’s perception of project success varies between the traditional waterfall method and the iterative approach, as the latter involves working collaboratively with the business and completing the project in incremental stages.
