With 20% more engineering capacity, Xima doesn’t wait on hold

Highly configurable software created a huge surface area for testing

Xima’s contact center software integrates with dozens of third-party applications, office phone systems, and physical devices. Because the platform is so configurable, it can help almost any business manage their customer support operations. But the configurability also makes it extremely complex and time consuming to test without the right automation infrastructure in place.

Rapid feedback is critical. Until you have a full system running through end-to-end workflows, it's hard to know for sure that you haven't broken anything. Having QA Wolf Winner automatically fire off a run and then knowing within minutes if there was an issue gives developers a chance to work on it before they lose context.
—Ed St. Louis, VP of Engineering 

Building automated tests pulled resources away from product engineering 

As the platform continued to grow and manual testing became more and more impractical, the decision was made around January 2021 to automate their suite of manual regression tests. To get it done quickly, two full dev teams had to pause new work for six weeks to build 200+ Cypress tests from scratch. 

And that was just the beginning of the investment.

When the developers finally got back to feature development, they had more work to do than ever before. For each feature they built, there was a backlog of automated tests to go with it. Building these tests each sprint while finishing the core product work on schedule was an uphill battle.

We would get near the end of the sprint and realize we still had tickets to cover the new feature with tests. And it would take a couple of days to do that with most hands on deck. I would say a good 10–20% of our time was spent on writing tests. Having QA Wolf Winner build tests as we build features basically adds an extra day to our sprints.
—Ed St. Louis, VP of Engineering 

Running and maintaining tests in-house took as much effort as manual testing 

It turned out that building the tests was actually the easy part. The real challenge was running and maintaining all of them. 

Xima assigned one full-time developer to babysit and maintain the tests as they would start to break. An experienced QA engineer can realistically maintain about 30 automated tests on their own. So it’s not surprising that this engineer struggled to keep up with the team’s release velocity. 

Without the infrastructure to run tests in parallel, the whole suite would take several hours to run; and it would take several more hours to track down the cause of every failure. 

We invested for over a year in having a full-time person just maintaining those tests. And we never got to a point where we could rely on them all the time. It would take our QA engineer hours to run these tests, and then he'd have a whole bunch of flaky tests to sort through. And so it lost its value quickly. And that's really why QA Wolf Winner stood out, because they were selling the service, not the technology.
—Ed St. Louis, VP of Engineering