Neil C. Hughes: Testing With QA Supermarket

Aug 2020

QA Supermarket provides comprehensive QA testing services on-demand and with competitive, completely transparent prices. The company is on a mission to offer Manual Testing as SaaS. The startup was founded by current CEO Paul Belevich, whose previous roles include Product Development Director at Blue Lithium (acquired by Yahoo!) and GM of Wargaming Research and Development Center in Austin. I wanted to learn more about how they found a way to change a long, boring, and difficult application testing to an easy, transparent, and hassle-free one. We talk about the differences between Automated and Manual testing. We discuss what trends they are noticing in the industry. We also bust a few myths and misconceptions in the world of application testing

Q: Who are you and what do you do?
A: I was born in Europe, Belarus. The software development industry is pretty wide there. I started my career as a software developer more than 20 years ago (as a Java and C++ engineer). Then I moved to team leading and project management, then to a higher management position (C-level). Since 2009 I was working in the gaming development industry in Wargaming. There is a project World of Tanks, which I developed from the beginning. It’s a multinational company now.
2 years ago I decided to start my own business which is connected to software development. Now I am working on the project.

Q: I heard you were working on World of Tanks!
A: Yeah, we started in 2009 and we launched in 2010. Soon we started to earn money and it is now quite successful.

Q: What kind of problems are you solving with QASupermarket?
A: I understand that quality control (QA), which is needed especially in startups, is on a pretty low level. And even brilliant ideas of startups can be refused because of that. It can fail because of the bad quality of the application, and startups can just be closed. Even worse, if you are the person to bring the idea to people, but without the appropriate quality, the next one could do it in the right way and bring the idea to life. Someone else gets the benefits from the idea.

I know a few reasons why decision makers do not test their applications. (see also our survey on this subject). In most of the cases they don’t have time, budget or experience for management or planning. Some developers think that automated testing tools are doing a good job or they can find bugs by themselves. Developers love their “baby” and know how users should work with it, and developers click the right button. But the behavior of real users is different, and test engineers can emulate it.
Finally we got that in applications, the average quantity of bugs is one [bug] per 25 lines of code. And these applications were already tested by programmers.

Q: What are the differences between automated testing and manual?
A: Automated testing is a testing process run by software which is tuned and scripted by test engineers or developers. Manual testing is a test that is performed by engineers. They use both logic and all possible testing tools. By selecting automated methods, they can use the suitable methods for application testing.
The main advantage of manual testing is quality. A real person can find the most unexpected bug, even the most advanced automated testing tools cannot check what it wasn’t tuned for. So for automated tests you need to develop a script, and, at the same time the automated testing scripts could have their own bugs. But testing engineers can interact in the natural “user’s” ways with the applications and devices, and perform some gestures that are impossible to replicate digitally. They [testing engineers] can also describe bugs in a professional way so it makes it easier to understand, reproduce, and finally fix, because developers can easily find a reason for these bugs.

Q: Do you have any use cases, or examples of the value and how it would work in others worlds?
A: There were many things that inspired me, but 2 of them I would like to describe:
1) One project leader has asked a shareholder to install a cell phone emulator from an Android phone on her Apple MacBook. It took her 2 days just to find and install it, and after that she was spending many hours to test the application and the result was really bad. She found some bugs which cannot be reproduced on real devices. She is actually not a professional, and can you imagine how many hours she’s spent on that for this not such a good result?

2) There was a client that walked up to me and asked to fix his application because in a few weeks he needed to present it to real users at the conference. I said it needed to be tested before fixes to know what needs to be fixed and what not. He wanted me to skip testing to save money. He said he did testing by automated tools already, and it didn’t show any of bugs.I asked him why he came to me, and he said “we have bugs in logic and automated tests couldn’t find it, but I tested the app with my wife”. We have fixed everything he found. And the presentation failed. He lost more than he saved on testing.
Q: What trends are you noticing in this industry?
A: Moving to remote work and actually that was how I developed the QA supermarket project. I set up the work remotely in the Netherlands, Slovakia, Belarus, and the United States. We are working online, it helps us at the time of quarantine. Now, we fit greatly into this “trend”. The idea behind QA Supermarket is Self Serving. Like “Uber” or “Walmart” for testing. So people can come and choose online what they need. They don’t need to interact with sales and other staff. We just eliminated all the stoppers and overheads, and we made an online, hassle-free testing service. So startups and app developers can test with our services very quickly without any hassle.

Q: Are there any myths in the world of testing or anything …?
A: Definitely! I mentioned them already. First is automated tools, out of the box it can test perfectly. That’s not true. You need to spend a lot of effort to tune it up, to do a good job. Also to adjust or update it in the future. So basically it’s very expensive to test automatically in the beginning. You need to spend a lot of money for the testing tool to be tuned up after installation. If the project becomes really really big, you can use automated tools. But in the beginning, I would suggest using manual testing. It would be cheaper and faster, in comparison to automated.

The other thing [the other myth] is that friends and family testing is good. In comparison to automated tools, it brings the real user experience, but usually non-professionals cannot describe bugs well. They don’t have experience like “what could happen with applications in real life”. For example: they can say just: “it crashes there” and that’s it. But a professional will describe steps to reproduce this bug, what was expected to be, what a test case we can apply to this app’s behavior, what a user gets and so on. Finally for developers it would be easier to find the reason for the bug and fix it.

Another myth I would like to mention is that programmers are also good testers. Usually it’s not true. In order to be a good test engineer, you need to have some kind of special mind set, to be able to find issues. It could sound boring for someone, but people love this job. Programmers know what the application should do and what users are supposed to click in their application. In this case they cannot find a way to find a bug and weak points of the app.

One last thing is that anyone can test. In order to be a good tester, you have to have a “special” mindset to work on other applications and experience where bugs can be found.

Q: What’s the best way of contacting you?
A: It’s really easy! All you have to do is go to the website, and find all the info. Like i said before, we have articles about testing, explanation of the differences of automated and manual testing and what you need to choose in your particular case. Also you can contact us directly.

Related Topics

Continuous Testing: Surveys Show Contradictions in How Much is Actually Happening in the Enterprise The World’s Biggest Bugs and How They Could Have Been Avoided