The number of tests we have for MySQL Server are constantly growing. There is a need to group them in different ways so we can select what and where to run. We do this by using suites, either the default suite that we call "main" in mysql-test/t or one of the subdirs of mysql-test/suite.
As each test becomes more advanced it's also necessary to use different configurations for a particular test or suite. For example all the replication tests in suite/rpl need to be run with the server started in three different ways(three different configurations) to get full coverage. To avoid that the individual developer or the "one" running tests have to remember different parameters to run the tests with, we have made it possible to make each suite special. That means it can have different number of MySQL Servers and different settings of each server.
The current mysql-test-run.pl in MySQL 5.1 has:
- support for running tests from suite/ directory
- each suite can have some special setting with the use of a suite.opt file.
- each suite can make each test to be run more different configurations by adding a combinations file(see for example suite/rpl/). That file will cause all .test to be "multiplied" by the number of combinations and thus run several times with different server settings.
- All the standard suites to look for are added to mysql-test-run.pl to makerit as easy as possible for a developer to run the default set of tests.
We also have a new mysql-test-run.pl that is going into 5.1 and up very soon. it has been extended to also support:
- allowing each suite to have it's own my.cnf file(s), this way we can run tests with various settings of the server.
- that each .test has it's own
- to find suites in storage/
- to find a suite in any directory by simply giving the full path to where that suite is.
In the future we'll continue to make it possible for each suite to be even more special(in a pluggable way) as we find the need for it.