The test pyramid misconception #1: “We have to build the pyramid”

Recently I have a few conversations about the possibility of TDD implementation in projects with rich waterfall history. I was surprised by how much the test pyramid’s popular explanations are misleading people, putting everything upside-down. In this article, I want to clarify the key ideas of the test pyramid. I hope it would help to create a better understanding.

Mike Cohn[1] introduced the test pyramid idea. This idea was popularized by many XP prophets, including Martin Fowler. If you google what the test pyramid is, the top result will probably be Fowler’s “The Practical Test Pyramid” article [2]. Both are excellent sources of information. 

However, way too many people think we have to build the pyramid to improve its quality. It is unbelievably wrong. We do not build the pyramid. Sounds weird, huh? But this is true. Just stacking test layers one upon the other, making each following smaller, would not improve anything. 

It is critically important to understand that we do not make the test pyramid intentionally. The test suite becomes shaped naturally if we properly apply just two Agile practices to the testing. I speak of “fail early” and “simplicity is the art of maximizing work not done.”

Let’s discuss the main issue we are trying to address. Every single book about software project management discusses “the cost of the defect.” Every book reiterates the idea that the later we find the defect, the more it will cost. 

Every person who cares about the project economy would be passionate about detecting errors as soon as possible. Fail early!

What is the earliest moment when can validate at least part of the requirements? When the code is written! What is the most effective way to test the requirements for an isolated piece of code? Unit tests! 

It is how the base of our pyramid appears. Because we do not test a piece of code, we test the conformance to the requirements (sic!). To do that, we have to enforce one vision of how we test from the very beginning of the development process. 

Now we build more code. We have multiple pieces of code, and each of them is proven to be correct. Now the second agile principle comes to play. How much testing can we skip because we trust the level below yet be sure that the solution is in conformance with the requirements? It is crucial! We do not make the next level smaller because Mike Cohn said it should be a pyramid! We do not answer the question: “well, how much testing we can get within the limits of the pyramid shape?” We just benefit from the single testing strategy applied from bottom to top, from having reliable coverage at the levels below. And we do not need that many tests now. Plain and simple, this is how the next level became smaller naturally.

If we failed to provide a trustworthy test at the levels below, each next level would be broader. It happens because it is four times more complex to test a system two times larger. In other words, the complexity of the testing depends exponentially on the size of the code tested. It is how you get to the ice-cone antipattern. 

If we see an ice cone – we ultimately failed to reach the goal. Most of the testing is done way too late to reduce the cost of the defect. 

If we failed to enforce one strategy to test, we could not establish connections between the levels, even if they were perfect. It is how we get an hourglass antipattern. We have multiple trustworthy levels that exist in their own separate universes. Most probably, we would have some improvements in early defect validation. However, because we couldn’t benefit from relying on the levels below, we ultimately failed to “maximize work undone.” We simply wasted many resources that might be used to create business value for our stakeholders.

So, the critical elements of building the working test pyramid are:

1) Enforce one test strategy from the top to the bottom

2) Make each level trustworthy so upper levels can rely on its foundation

3) Benefit from having a part of the testing job already done on the levels below.

The pyramid shape is not the tool. It is a great visual metric that shows us that we applied the TDD approach correctly and helps to diagnose the mistakes. 

[1] Mke Cohn “Succeeding with Agile” ISBN 978-0321579362


Tell your friends
Thank you! Your submission has been received!