Grouping .test files into suites

Please, enlighten me on what being in a separate suite means! I see there is a suite directory in mysql-test. Will the suites be run by default when I use 'mtr' to run tests? Or do I have to add them manually?
=========================================

The number of tests we have for MySQL Server are constantly growing. There is a need to group them in different ways so we can select what and where to run. We do this by using suites, either the default suite that we call "main" in mysql-test/t or one of the subdirs of mysql-test/suite.

As each test becomes more advanced it's also necessary to use different configurations for a particular test or suite. For example all the replication tests in suite/rpl need to be run with the server started in three different ways(three different configurations) to get full coverage. To avoid that the individual developer or the "one" running tests have to remember different parameters to run the tests with, we have made it possible to make each suite special. That means it can have different number of MySQL Servers and different settings of each server.

The current mysql-test-run.pl in MySQL 5.1 has:
- support for running tests from suite/ directory
- each suite can have some special setting with the use of a suite.opt file.
- each suite can make each test to be run more different configurations by adding a combinations file(see for example suite/rpl/). That file will cause all .test to be "multiplied" by the number of combinations and thus run several times with different server settings.
- All the standard suites to look for are added to mysql-test-run.pl to makerit as easy as possible for a developer to run the default set of tests.

We also have a new mysql-test-run.pl that is going into 5.1 and up very soon. it has been extended to also support:
- allowing each suite to have it's own my.cnf file(s), this way we can run tests with various settings of the server.
- that each .test has it's own .cnf file. For example rpl_circular_for_4_hosts.test(4 way circular replication) need a config file with 4 MySQL Server's in it, thus it adds rpl_circular_for_4_hosts.cnf along side the .test file.
- to find suites in storage/ to make it possible to put storage engine specific tests along side the code for that storage engine
- to find a suite in any directory by simply giving the full path to where that suite is.

In the future we'll continue to make it possible for each suite to be even more special(in a pluggable way) as we find the need for it.

How to concatenate strings in a mysqltest testcase

Hi Magnus, is there a way to concatenate strings in a mysqltest?
=================================================================


Yes, you can "easily" create more dynamic strings using let and a while loop. For example like this:

let $c= 254;
let $str= t255;

while ($c)
{
     let $str= t$c,$str;
     dec $c;
 }
 echo $str;

This will printout t1, t2, t3, ... t255.


You can then use the $str variable in an eval
to CREATE TABLE, VIEW or SELECT

eval CREATE TABLE t0 (a INT) ENGINE=MERGE UNION($str);

How to restart a server after an expected crash or shutdown in a test case?

There are now two ways how to do this:

1. Using "set debug=d,flag" and adding some DBUG_EXECUTE_IF("flag", abort()) we can trigger the server to crash at a specific place in the code. There is an example of this in crash_commit_before.test
This requires a debug compiled server so you need to add a "source include/have_debug.inc" at the beginning of the test.

2. There is also a new way to do it without using DBUG_. By using the new command in mysqltest called "shutdown_server"[1], we will tell the server to stop, wait a while(60 seconds) and finally kill it off. This way we get a reliable shutdown. The first testcase that uses this is events_restart.test

In both of these methods, the testcase has to write a small file before the "crash/shutdown", that file tells mysql-test-run.pl that it was an expected crash/shutdown. The server will be started up again with the same settings as before crash/shutdown.


[1] "shutdown_server" is a new command in mysqltest, not yet pushed to the main trees. It will appear in 5.1 and up.
The syntax is:
shutdown_server [timeout]
if the server's pid file is not gone after timeout seconds, the process will be killed. Default timeout value is 60 seconds.