Debugging MySQL Cluster data nodes(ndbd) using ndbout

One basic way of debugging the MySQL Cluster data nodes(ndbd) is with good old printf-style debugging. Ie. adding printouts of the variable you want to see, recompile, start the node(s) and run your tests. This is a bit tedious but very universal.

Support for printing is available using "ndbout", which has both C++ style and printf style. This functionality is implemented in storage/ndb/include/util/NdbOut.hpp

Here is an example from debugging BUG#37592, where I wanted to add printouts to "Dbtup::rebuild_page_free_list" to see how the pageId increases while the rebuild is running:

void
Dbtup::rebuild_page_free_list(Signal* signal)
{
  Ptr fragOpPtr;
  fragOpPtr.i = signal->theData[1];
  Uint32 pageId = signal->theData[2];
  Uint32 tail = signal->theData[3];
  ptrCheckGuard(fragOpPtr, cnoOfFragoprec, fragoperrec);

Using "ndbout" with printf style:

 ndbout_c("Dbtup::rebuild_page_free_list, pageId: %u", pageId);

In this case you have to think about what datatype "pageId" is to print it out correctly.
Newline is always added to the end of the printout.

Using C++ style:

 ndbout << "Dbtup::rebuild_page_free_list, pageId: " << pageId << endl;

In this case you need not worry about the datatype of pageId it will automatically select the correct conversion.

The stdout from ndbd will when I run it look like this:

Dbtup::rebuild_page_free_list, pageId: 65
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 67
Dbtup::rebuild_page_free_list, pageId: 67

Using the C++ style it is also possible to add a custom printer to print a whole class or struct easily. An example of that is the one for Restore::Column which in ndb/src/kernel/blocks/restore.cpp looks likes this:

NdbOut& 
operator << (NdbOut& ndbout, const Restore::Column& col)
{
  ndbout << "[ Col: id: " << col.m_id 
  << " size: " << col.m_size 
  << " key: " << (Uint32)(col.m_flags & Restore::Column::COL_KEY)
  << " variable: " << (Uint32)(col.m_flags & Restore::Column::COL_VAR)
  << " null: " << (Uint32)(col.m_flags & Restore::Column::COL_NULL)
  << " disk: " << (Uint32)(col.m_flags & Restore::Column::COL_DISK) 
  << "]";

  return ndbout;
}

Hopefully this give some idea about how to start working with a new feature or how to see what is going on in the black box. Will write more about other ways in the future.

Grouping .test files into suites

Please, enlighten me on what being in a separate suite means! I see there is a suite directory in mysql-test. Will the suites be run by default when I use 'mtr' to run tests? Or do I have to add them manually?
=========================================

The number of tests we have for MySQL Server are constantly growing. There is a need to group them in different ways so we can select what and where to run. We do this by using suites, either the default suite that we call "main" in mysql-test/t or one of the subdirs of mysql-test/suite.

As each test becomes more advanced it's also necessary to use different configurations for a particular test or suite. For example all the replication tests in suite/rpl need to be run with the server started in three different ways(three different configurations) to get full coverage. To avoid that the individual developer or the "one" running tests have to remember different parameters to run the tests with, we have made it possible to make each suite special. That means it can have different number of MySQL Servers and different settings of each server.

The current mysql-test-run.pl in MySQL 5.1 has:
- support for running tests from suite/ directory
- each suite can have some special setting with the use of a suite.opt file.
- each suite can make each test to be run more different configurations by adding a combinations file(see for example suite/rpl/). That file will cause all .test to be "multiplied" by the number of combinations and thus run several times with different server settings.
- All the standard suites to look for are added to mysql-test-run.pl to makerit as easy as possible for a developer to run the default set of tests.

We also have a new mysql-test-run.pl that is going into 5.1 and up very soon. it has been extended to also support:
- allowing each suite to have it's own my.cnf file(s), this way we can run tests with various settings of the server.
- that each .test has it's own .cnf file. For example rpl_circular_for_4_hosts.test(4 way circular replication) need a config file with 4 MySQL Server's in it, thus it adds rpl_circular_for_4_hosts.cnf along side the .test file.
- to find suites in storage/ to make it possible to put storage engine specific tests along side the code for that storage engine
- to find a suite in any directory by simply giving the full path to where that suite is.

In the future we'll continue to make it possible for each suite to be even more special(in a pluggable way) as we find the need for it.

How to concatenate strings in a mysqltest testcase

Hi Magnus, is there a way to concatenate strings in a mysqltest?
=================================================================


Yes, you can "easily" create more dynamic strings using let and a while loop. For example like this:

let $c= 254;
let $str= t255;

while ($c)
{
     let $str= t$c,$str;
     dec $c;
 }
 echo $str;

This will printout t1, t2, t3, ... t255.


You can then use the $str variable in an eval
to CREATE TABLE, VIEW or SELECT

eval CREATE TABLE t0 (a INT) ENGINE=MERGE UNION($str);

How to restart a server after an expected crash or shutdown in a test case?

There are now two ways how to do this:

1. Using "set debug=d,flag" and adding some DBUG_EXECUTE_IF("flag", abort()) we can trigger the server to crash at a specific place in the code. There is an example of this in crash_commit_before.test
This requires a debug compiled server so you need to add a "source include/have_debug.inc" at the beginning of the test.

2. There is also a new way to do it without using DBUG_. By using the new command in mysqltest called "shutdown_server"[1], we will tell the server to stop, wait a while(60 seconds) and finally kill it off. This way we get a reliable shutdown. The first testcase that uses this is events_restart.test

In both of these methods, the testcase has to write a small file before the "crash/shutdown", that file tells mysql-test-run.pl that it was an expected crash/shutdown. The server will be started up again with the same settings as before crash/shutdown.


[1] "shutdown_server" is a new command in mysqltest, not yet pushed to the main trees. It will appear in 5.1 and up.
The syntax is:
shutdown_server [timeout]
if the server's pid file is not gone after timeout seconds, the process will be killed. Default timeout value is 60 seconds.