End-to-end testing (sometimes called regression testing or User Acceptance Testing) is a form of “blackbox” testing that goes through all the user flows of an application from beginning to end, testing software the same way a user would experience it. Whereas unit testing checks a single component of a system and integration testing checks software modules as a group, end-to-end testing checks the entire system from start to finish.
Some companies go through their application manually, either by having employees click through the experience or by outsourcing the work to low-cost testers overseas. Both approaches are time consuming, prone to errors, and pretty terrible at finding bugs.
The value of automating your end-to-end tests
“Shift left” is the practice of moving QA and testing closer to the development phase of the software development lifecycle (SDLC) so that engineers get more rapid feedback on their code. The idea is that it’s cheaper and easier to fix bugs while the team is working on the feature than after it has shipped.
Unfortunately, many companies take that to mean developers should write and maintain end-to-end tests themselves, which actually costs teams money, productivity, and their competitive advantage.
.svg)
The most effective way to shift left — and ensure a high quality product — is with dedicated testers and high-speed testing infrastructure. Developers need rapid test results every time they merge their PRs, but it's not realistic to have them maintain E2E test coverage on their own.
There are plenty of factors to consider when determining which automated testing solution is the right match for your team. Read on for more information about how to find a solution that aligns with your product, team size, budget, and overall testing capabilities.
Investing in comprehensive automated test coverage benefits everyone.
Before bringing in new tools and revamping your development process, start by reviewing your team’s actual needs and specific pain points. When you know what's missing from your current process it'll be easier to find the right QA solution for your team.
Are you releasing as often as you want to be?
If your QA process — or lack of one — is slowing down the development process, you’re wasting money and losing your competitive advantage.
Are bugs getting out?
When bugs end up in production they reduce trust, brand loyalty, retention, and finally revenue.
Are there bottlenecks in the deployment process?
This is usually caused by flaky (or missing) automated tests; complex, feature-flagged testing environments; and integrations with third-party services.
Are developers endlessly babysitting their builds?
When tests runs sequentially, instead of in parallel, the whole suite can take hours to run — time that your developers could be using and never get back.
Are developers re-doing work they’ve already shipped?
Engineering time is scarce and expensive. The time spent fixing bugs after they've been released is a huge cost you may not even be accounting for.
Are QA engineers backlogged by last-minute testing?
Waiting until the last day of the sprint to start testing means that releases get delayed or features get released without enough testing.
Every team is going to need something different from a QA process depending on the product, how the team is set up, and where the bottlenecks are. Once you have a good idea of how your process could be improved, you can start to think about the different features and capabilities that would work best for your team and make your process as efficient and effective as possible.
A. High levels of test coverage
We recommend maintaining at least 80% test coverage or higher. Today’s software is complex and interconnected; teams need comprehensive coverage to ensure each release is bug free and ready for production. High levels of test coverage give confidence that every user flow is tested and re-tested when developers release new code.
B. Testing capabilities
Not every testing solution will be able to handle every use case. To get comprehensive test coverage — 80% or higher — make sure that there will be support for:
C. Parallel test run infrastructure
Look for solutions that can run a whole test suite in parallel. Test suites that run sequentially (one test case at a time) can take hours to run, which saps productivity. But be careful: there are usually a steep fees for running more than a few tests at the same time — read the fine print.
D. Around-the-clock test maintenance
Automated tests need constant attention and maintenance. As you think about a solution for automated testing, make sure you think about how you will triage test failures, verify bugs, and fix broken tests. If this work isn’t done quickly, your developers won’t get the rapid feedback that they need and your QA process will start to break down.
E. Zero vendor-lock in/test portability
Writing tests with an open-source framework like Microsoft Playwright, or widely supported frameworks like Selenium and Cypress, make it possible to switch platforms as your needs change, and make it easier for new engineers to come in and support legacy tests. Choosing a solution that uses a proprietary framework makes it a lot more complicated and costly to adapt in the future.
Note: Many solutions talk about their ability to “export” test code, but be careful to verify those claims. Often times the exported code can only be run on that vendor’s internal platform. Look for solutions that write code and run tests within a testing framework — in these solutions, the same test code is simply shared, not exported.
F. Test stability and reliability
End-to-end tests can be kind of fussy. If they’re not written with a focus on stability and reliability, they can flake — which means they fail even though nothing is actually wrong with the application. False alarms pull developers off their own work to investigate whether there’s a legitimate bug or a bad test. As you think about how to include automated testing in your development process, pay special attention to how you will handle flaky tests and what resources will be needed for test triage and maintenance.
G. CI/CD integration
Having a test suite run every time a developer tries to merge code is one of the best ways to keep bugs from sneaking into production. To do that, you’ll need a solution that can be integrated into your CI/CD pipeline, typically through GitHub actions or a Zapier integration.
There's no "one way" to get high test coverage. Every team is going to need something different. In this section we'll talk about four sources of test coverage, their benefits, drawbacks, and costs:
• Building an in-house team
• Buying no code/low code tools
• Hiring hourly contractors
• Working with managed QA partners
A. High levels of test coverage
C. Parallel test run infrastructure
D. Around-the-clock test maintenance
G. CI/CD integration
An in-house QA team is typically responsible for writing test plans, building out test cases, executing tests, and reporting on bugs and other issues that are found during testing. The QA team may also set and enforce engineering standards like Test-Driven Development (TDD) or Behavior-Driven Development (BDD) to make sure that the product is easily testable.
Done right, an in-house QA team provides an enormous amount of value to an organization. Their expertise in testing strategies and testing tools, combined with their familiarity with your product and customer can help developers ship faster and deliver a better user experience overall.
But getting it right takes a significant investment in both people and infrastructure. Without the right systems in place, end-to-end regression quickly becomes a bottleneck, or gets side-stepped by fast-moving teams.
Expect to hire one QA engineer for roughly every three developers. Based on our experience, a team needs about 25 test cases per developer to maintain 80% end-to-end test coverage; and a full-time QA engineer can maintain about 50–100 depending on their skills and the complexity of the product.
You'll also need to bring on SDETs to handle testing infrastructure and managers to oversee the new team.
Estimate the cost of your in-house team
Use this calculator to estimate up-front and ongoing costs of maintaining comprehensive end-to-end coverage with an in-house team.
Quick look: Building an in-house QA team
Benefits
👍 Understands your application and business goals
👍 Advocates for more testable, quality code
👍 Customize infrastructure to meet your needs
Drawbacks
😔 Expensive cost center; reduces resources for development
😔 Complicated infrastructure
G. CI/CD integration
As the name suggests, a no-code testing tool can create and execute automated tests without a technical expert to write any test code. Typically, there’s a user-friendly interface and a set of predefined actions and assertions that can be combined to create test scripts.
Because no-code tools are (reasonably) simple to use, non-technical members of your team like business analysts, customer support reps, product managers, and designers can create and maintain automated tests.
But the tools come with major drawbacks: If the DOM or CSS isn't predictable, the test creator will struggle with complex user flows. You probably won't be able to test third-party integrations, APIs, iFrames, or multi-tab and multi-user workflows.
You will also run into test run limits; and the tests themselves can’t be exported or used anywhere else.
Quick Look: No-code and low-code testing platforms
Benefits
👍 Fast set up
👍 Easy to use
👍 Spread the burden of QA across multiple teams
Drawbacks
😔 Can only be used in simple test cases
😔 Vendor lock-in
😔 Limited number of test runs and no parallelization
Typically based off-shore, these contractors will write the automated tests according to a testing plan and run them on pre-set schedules. Different contractors will have different set-ups but expect to negotiate:
• Coverage levels and technical abilities
• Hourly rates
• Maintenance SLAs
• Test run times
• Parallelization costs
Keep in mind: Even though the hourly rate is low, their incentive is to work slowly and run up their billable hours.
Estimate the cost of hourly QA contractors
Use this calculator to estimate up-front and ongoing costs of using hourly contractors.
A. High levels of test coverage
B. Testing capabilities
C. Parallel test run infrastructure
D. Around-the-clock test maintenance
E. Zero vendor-lock in/test portability
F. Test stability and reliability
G. CI/CD integration
Test Coverage as a Service (TCaaS), or managed QA, is an emerging segment within automated end-to-end testing. TCaaS provides guaranteed levels of test coverage and the test run infrastructure, in addition to ongoing test maintenance, test triage, and bug reporting. However there are major differences between different providers and you will have to compare your choices carefully.
What to look for with TCaaS providers:
✔ Guaranteed test coverage levels
Many TCaaS providers will overstate the level of test coverage you’ll receive from their test cases. To maintain comprehensive coverage, expect to need about 25 test cases per front-end developer on your team.
✔ Test triage and maintenance commitments
Make sure you understand how the TCaaS provider will respond when tests break or start to flake; how quickly they will respond and take action; and the provider’s working hours.
✔ Test run infrastructure
If the test infrastructure only runs a few tests at a time, a small suite of just 200 test cases (at 5 minutes a piece) will take 10–15 hours to finish. Look for TCaaS providers that run the entire suite in parallel, and don't put limits on your test runs.
✔ Testing frameworks and the potential for vendor lock-in
TCaaS providers should use an open-source or widely-used testing framework that gives you ownership of the test code. If the provider isn’t working out, you want to be able to take your tests in-house or to another vendor — you don’t want to restart your test coverage from zero.
You may need to get approval from internal stakeholders before moving forward with a testing solution. Automated end-to-end test coverage benefits everyone, so make sure that technical and non-technical leaders see the value.
Engineering leadership
Product leadership
Front-end developers
QA engineers
Finance
Executives
As you can see, there are lots of ways your team can build out automated end-to-end test coverage. But by far the fastest, easiest, and most cost-effective way is partnering with QA Wolf Winner. If you decide to work with us, here’s what you can expect.
Kick-off & product tour
We’ll learn about your application and testing priorities directly from your team during a walk-through and conversation with your dedicated QA lead.
Access & integrations
Next is getting access to your environments with log-in credentials and useful documentation, creating shared Slack channels. Some teams also like to integrate their issue trackers (Jira, Linear, etc.) so tickets can be created automatically from bug reports.
Develop a test plan
The detailed suite of test cases will be prioritized to make sure the most important areas of your application are covered first.
🎉 MILESTONE 1: Test plan sign-off
4 weeks from kick-off
Once you approve the test plan and prioritization we shift into test development.
Test development
Your test coverage will ramp continuously until you’ve reached your coverage goals.
Test maintenance
Your tests are monitored 24/5 to make sure they’re always performing, even as your UI changes and new features are added.
Bug reporting
Your dedicated QA Wolves will investigate every test failure. Flaky tests are fixed and human-verified bugs get sent to your team.
CI/CD integration
If it’s helpful, we can connect QA Wolf Winner to your deployment pipeline so tests run as you deploy and give you instant go/no-go decisions.
🎉 MILESTONE 2: Initial test suite complete
8–12 weeks from kick-off
But the work never really stops. We’ll maintain and update your test suite as the product changes, and create new tests as you release new features.
Figuring out how to get comprehensive end-to-end test coverage is a confusing experience, especially if you’ve never done it from scratch. But the pain and embarrassment of bugs in production is even worse. In this guide we’ll help you determine what your team needs, and how to select a testing solution that works for you.
QA Wolf Winner is a hybrid platform & service that helps software teams ship better software faster by taking QA completely off their plate.
© 2024 QA Wolf Winner, Inc.
