Thursday, June 5, 2008

Grouping .test files into suites

Please, enlighten me on what being in a separate suite means! I see there is a suite directory in mysql-test. Will the suites be run by default when I use 'mtr' to run tests? Or do I have to add them manually?

The number of tests we have for MySQL Server are constantly growing. There is a need to group them in different ways so we can select what and where to run. We do this by using suites, either the default suite that we call "main" in mysql-test/t or one of the subdirs of mysql-test/suite.

As each test becomes more advanced it's also necessary to use different configurations for a particular test or suite. For example all the replication tests in suite/rpl need to be run with the server started in three different ways(three different configurations) to get full coverage. To avoid that the individual developer or the "one" running tests have to remember different parameters to run the tests with, we have made it possible to make each suite special. That means it can have different number of MySQL Servers and different settings of each server.

The current in MySQL 5.1 has:
- support for running tests from suite/ directory
- each suite can have some special setting with the use of a suite.opt file.
- each suite can make each test to be run more different configurations by adding a combinations file(see for example suite/rpl/). That file will cause all .test to be "multiplied" by the number of combinations and thus run several times with different server settings.
- All the standard suites to look for are added to to makerit as easy as possible for a developer to run the default set of tests.

We also have a new that is going into 5.1 and up very soon. it has been extended to also support:
- allowing each suite to have it's own my.cnf file(s), this way we can run tests with various settings of the server.
- that each .test has it's own .cnf file. For example rpl_circular_for_4_hosts.test(4 way circular replication) need a config file with 4 MySQL Server's in it, thus it adds rpl_circular_for_4_hosts.cnf along side the .test file.
- to find suites in storage/ to make it possible to put storage engine specific tests along side the code for that storage engine
- to find a suite in any directory by simply giving the full path to where that suite is.

In the future we'll continue to make it possible for each suite to be even more special(in a pluggable way) as we find the need for it.


Malcolm Cook said...

Timely reading for me as I've just been trying to get to work with a set of unit tests that I am devising to test some triggers and stored functions/procedures in a recent MySQL custom database.

While casting around for a unit testing framework, when I found mysqltest and, I thought "What could be better than to use what the developers of the engine are using themselves?"

My problem has been the paucity of examples of how to use for anything other than the tests of the engine that reside in the same directory structure. It seems that the perl script MUST run in the directory in which it is installed, and that you can not point it at your own directory of tests.

I just started looking at the Perl source to see how to create a new --base option to pass down to the call to mysqltest. I think this is the way to go. Do you agree? Can you advise?

Magnus said...

1. Yes, it should be supported to point to your own directory(suite) with .test and .result files. This is included in the next version.

2. I've more or less resolved the issue with that you have to cd to mysql-test/ before you run, but it still has a check that you run it from mysql-test/ since we don't have time to really test any other ways to run it.

Don't think a --base is needed, but maybe you can describe what it should do?

Hope you find useful.