TechRepublic: Continuous Testing: Surveys show contradictions in how much is actually happening in the enterprise

May 2020

Are companies doing Continuous Testing? Two recent surveys provide a few answers … and some caution.

While Continuous Testing is becoming a de facto standard for software groups in Silicon Valley, barriers to adoption persist, especially outside of the West Coast and for legacy software. Two groups, Capgemini and QA Supermarket, recently released surveys that paint startling–and contradictory–pictures about the state of the industry. 

Capgemini’s 2020 Continuous Testing Report draws an optimistic view, stating that 55% of organizations they surveyed have “adopted a continuous testing approach”, with 37% using containers to automate the creation of virtual test machines, and 42% using artificial intelligencefor predictive analytics. The QA Supermarket data was flip-flopped, with 44% of respondents doing actual-device (“real”) testing as their primary mechanism for all forms of software testing, and 13% reporting they were not testing at all. That has one survey saying 55% of groups are doing Continuous Testing, a relatively mature process that requires a fair bit of infrastructure or tooling, while another says that 57% are doing nothing or doing real end-to-end simulations with humans.

I spoke with Mark Buenen, the leader of quality engineering for Capgemini, and Paul Belevich, CEO of QA Supermarket, to understand how these surveys worked, who they surveyed, and to integrate the two perspectives. 

Survey methods 

The Cagemini survey first identified 500 larger organizations, then sent the questionnaire to one single leader in the organization responsible for software. That might be a vice president, a CTO, or general manager. Instead of someone responsible for software, the QA Supermarket survey went out to 140 people involved in testing in some way, which could be a test lead, project manager, programmer, or development manager. The Supermarket data was mostly based on smaller companies, with only 40% working at a company with more than 100 employees, and much more fine-grained, asking what kind of software the person tests, what types of testing, and if the respondent felt the organization did enough. The first disconnect was in that data: While 82% of tech managers thought there was “definitely” or “probably” enough testing, that number dropped to 67% for QA engineers and QA managers. One of the common reasons for not enough testing: “The decision makers at my organization believe we do enough testing.” 

But there are 13% who are doing no testing. Belevich said the primary reason to make such a claim was that testing was happening outside of the team. For example, the customer might do some sort of formal user acceptance testing. Thus, “we” don’t do testing. Of course, programmers could click through screens, write unit tests, and do debugging as part of their work process, but testing might not exist as a formal, “external” role inside of the team. There might be no slot for it in the workflow, or there might not be a “tester” role. Given that explanation, I would expect the 13% number is artificially low. 

The Capgemini Survey, on the other hand, was incredibly optimistic. It stated that 16% of the survey respondents were using “predictive test selection and optimization,” 14% were doing “release risk [AI] prediction,” 12% were doing “automatic defect remediation,” and 9% were running “self-healing test scripts.” To be frank, I cannot understand where these numbers came from. Adding up the numbers in figure 8 of the survey, I see they total 100%. It turns out the readers were forced to pick one. In context, the actual percentages were one thing the survey respondents, in 2019, expected to look into doing in 2020. On the plus side, I found AppSurify, a company that can analyze some types of code changes to run a subset of just the automated checks that could be impacted by the change. These tools are starting to emerge, slowly, but I am very skeptical of over-hyped solutions. 

Explaining the disconnect 

Buenen acknowledged a real gap between rhetoric and practice. “When we see what people are actually doing, the adoption at implementation is really very slow. Strangely with the adoption of DevOps, which should require automation, in many cases the opposite is true—the amount of automation is actually going down.” 

In my own work, I split functional testing into two major categories. There is the testing of an individual feature, usually best done by a human. Then, there is an exploration of the entire system prior to release, sometimes called “regression testing.” By breaking the application down into different pieces, by rolling out only the module that changed, strong monitoring and fast-rollback, it is often possible for some teams to change the risk picture to eliminate the need for most regression testing. 

Also, if you send a survey to one person in a very large organization, especially a decision maker disconnected from the work, they are likely to give the response for their single best-performing team or business unit. So, don’t feel bad that your team isn’t performing as well as some vice president believes the best performing team is of some Fortune 500 company. At the same time, many organizations don’t see testing as a documented, formalized process being done by someone with the title “tester.” That doesn’t mean it isn’t happening. 

Take what you can from the surveys, but do your own thinking. If you want to pursue continuous testing, start with automating the build and delivery pipeline, including automating the creation of test data and test environments.

Why QA Supermarket?

  • Expert Testing on Demand
  • Upfront NDA
  • Transparent Pricing
  • Free Test Plan
  • No Obligations
  • No Overhead
  • No Sales Pitch
  • 100% Satisfaction Guaranteed

Why do you need professional testing:

  • About 60% of apps fail because of bad quality.
  • Automated testing will not show most of the bugs witch human may get.
  • Most developers are not qualified and not efficient to perform testing of your app.
  • The application must be tested in full – unit test can work, but integration of the all parts of the system can bring lots of surprises (see examples).

Why choose QA Supermarket?

  • Cost effective: Third-party testing on demand reduces expenses for internal QA team and hardware
  • Experienced: All testing is performed manually by highly-trained, knowledgeable experts
  • Transparent: Pricing is provided upfront so you know exactly what you’ll get and what you’ll pay
  • Easy: QA’s interactive dashboard guides you through the process, providing real-time status updates from start to finish
  • Secure: With an upfront NDA, you can be confident we’ll find your bugs while you keep your code

It takes only 3 steps to test your app

Log In
Type your link
Get free testplan & start testing

QA Supermarket’s full range of testing services for software, websites and applications include:

Functional testing

Does your product work like it’s supposed to? >

Installation testing

Can people install and load your product quickly and easily? >

Usability testing

Do the interface and other components of the product offer users an optimal experience? >

Interrupt testing

Will interacting with device components or other software cause your product to malfunction? >

Performance testing

Does your product behave like you want under different load conditions? >

Compatibility testing

Will your product look good and work properly across different devices and platforms? >

Stability testing

Will your product continue to perform well over time and through a full range of uses? >

Why QA

Why QA Supermarket?

You no longer need to maintain your own testing department and pay salaries, insurance and taxes. No need to spend money on facilitating tasks and purchasing equipment for testing. QA Supermarket provides an NDA upfront. We provide workflow to guide you through the process from a task definition to final bug list. Get in touch and rest assured that your software will be bug-free soon.

How can QA Supermarket help you?

Business owner

QA Supermarket helps you deliver the a highest-quality product to keep customers happy and your reputation intact

Read more

Project manager

QA Supermarket acts as an extension of your team so you can reach your goals on the timelines you need

Read more

Software developer

QA Supermarket enables you to focus on creating a great product while we keep the bugs out of your code

Read more