Post History

I was discussing this problem again with some coworkers, and one of them said "QA has that problem too -- here's what we're doing about it".

At its core, having everybody add lots of tests to the test suite in the server repository isn't manageable.  In addition to the volume -- and *number* of files (to dig through looking for something) can be as big a challenge as *volume* -- we really don't want to open that repository for writes from a larger group of people.  And members of the doc and QA teams don't want to have to go through what is, for them, a pretty heavyweight process for committing changes to the server code repository.

QA has, as you might expect, a large body of tests that are independent of the developer tests.  They've been using some of their own tooling to manage them (in a separate repository), but everyone involved wanted those tests to run using the same test-running framework that we use for dev tests -- but run *separately*.  This is the problem QA was tackling recently, and it's analogous to the doc problem.

The solution we implemented was to modify the test-running framework to, optionally, link in a separate repository.  The tests *run* from the test directory in the server repo, like the existing tests already do, but additional arguments to the invocation can now specify whether to use the QA, or doc, repo.  If this is specified, then after checking out the server repo as usual for test runs, the test run also checks out the additional repository and creates a sym link in the server test directory.  It then runs the tests (which test suite(s) to run is also an input) as usual.

To the test runner, it looks like the tests are in the server test directory.  But the server repo doesn't need to care about them, and QA and doc don't need to be able to modify that repo.

When we're ready to integrate doc tests (example scripts) into the doc, we can have the doc build check out the additional repo in a similar fashion and then do its build.  We haven't gotten that far yet.

In the end, we did something similar to a suggestion made in a comment months ago.