There has always been something about performance testing that has bugged me over the years. It’s the standard approach of ramping up the amount of load on each test execution cycle.
The typical order of executions for a performance test is:
- Smoke Test
- 25% Load
- 50% Load
- 75% Load
- 100% Load
The flaw, as I see it, lies in the order of execution. I think we an improve on this concept by aiming to ramp down instead.
The current ramp-up is no doubt derived from industrial engineering techniques, and has inherently become the standard practice amongst software projects. The problem is that while testing a physical object by “easing” on the load makes logical sense, this does not translate to the software world. The fear of something catastrophically breaking beyond repair under load does not exist in the software world. This makes starting with 25% load the least logical choice for performance testing.
Unfortunately, what makes even less sense is that after 100%, increments are often increased exponentially to 200%, 400%, etc. So why do we ease into software load tests the same was as in the engineering world then run enormous numbers at the later stages of the test? The answer is confidence. The results obtained at 100% load can give us the confidence to try 200% load and beyond, or in many cases not. If we are seeing low resource utilisation on our SUT at 100% load, we have the confidence to try exponentially increased load.
It could therefore be argued that it is more efficient to execute the 100% load test first. This test will provide you with significantly more information about the AUT and the performance test exercise itself than any other load level. Let’s look at some advantages.
1. Provide a sanity check of your planned load.
If the response times are atrocious or resource utilization maxes out, we know after only 1 test run that we have an issue. If response times are well below expected, we might have underestimated the load we should be simulating. Both of these scenarios are important to know as early as possible.
2. Prove your test generator setup.
Another advantage of the 100% test is that we are also able to determine if our load generation technique is adequate. Imagine executing 3 load tests from 25% to 75%, performed test analysis and your on the final night of execution only to find you have maxed out your virtual user licenses or load generator boxes an are unable to reach 100% load. Disaster.
3. Knowing if we go up or down.
What is the point of a load report that shows up to 100% load when response times are excellent and server resource utilization is very low? Wouldn’t it be better to use our 5 runs to go up from 100% showing how many transactions the AUT can handle? Instead, by sticking to our ramp-up philosophy, we would have to increase the number of executions to achieve this same result. Extra executions aren’t good for anyone involved.
By aiming to ramp down from the 100% test, we can benefit from these advantages without negatively effecting our test cycle. So next time your planning your test executions, put some thought towards running your 100% load test first, it may just save you some time in the long run.
I totally agree. You should aim to make it “fail fast” and let everyone know there is a problem as early as possible. I always like to run some early tests before I have even finished scripting. If the application fails under the load from a single business process, then you might have given the project team an extra couple of weeks compared to if you had waited until all the scripts were complete before running any load.
I have seen the “fail fast” method is often found by accident by Junior testers who decide to validate their data with think time and pacing turned off, inadvertently loading the system up significantly more than was planned in the official executions. It’s always great to see their reaction when they realise they have brought the systems down with VUGen.
I have always followed the ramp-up method, based on my training and what seemed to be industry practice. It’s not something that I have thought of before. Reading this and Stuart’s comments definitely has inspired me to take on my next project with a new approach. Thanks guys.