Finding the source for MySQL 5.4

In order to really check what currently has made it into 5.4(there has been rumors...) I wanted to branch the code and see for myself.

Couldn't find it at first in our internal code repository since it's actually not named 5.4 there yet. Fortunately it's also pushed out to launchpad, that means you can easily get a copy of the code using bazaar

$> bzr branch lp:mysql-server/5.4

There is of course also tar balls available from our download page
http://dev.mysql.com/downloads/mysql/5.4.html#source

Anyone curious about what our internal name is? :)

Stutter in MediPortal streaming server

Reading the Mediaportal forum thread and seeing that others have similar stutter as I have been annoyed about for the time I've been using MediaPortal in multiseat mode with RTSP. After having compiled the DirectShowFilters solution, I modified the StreamingServer example application to stream a 2.5GB recording I had on my disk. This made it fairly easy to debug by both attaching the debugger and also adding printouts(my favourite).

My analyze shows that(at least my) stutter occurs when TsBuffer.cpp reads more data from the file using 'FileReader::read'. The time to read from file is normally below 50 usec, but sometimes I can see it takes one second or more.

Will attempt to try and fix the problem by one of:
* Add a thread to TSBuffer that keeps the buffer full by reading the following part's of the file asynchronously in the background. Since TSBuffer is using either MultiFileReader or FileReader to read from file, it's probably better to use a thread then to change both of those classes to use asynch IO with Overlapped IO. Well, that coudl be discussed, some what of reading ahead should be done anyway.
* Seems like there is less stutter when using MultiFileReader(could be because the file is "hot" since it's being recorded to). Could be a bug with FileReader, Need to be examined. For example, why are all those SetFilePointer done when the file is already positioned at the correct place? Maybe they are expensive calls?
* Start to read from file as part of putting the "frame" into the queue rather than doing the read when taking it out of the there. This could give some additional time for the read(if I have read the code correct).
* Buy new hardware :)


Have also downloaded and compiled latest version of liveMedia555 on linux, configured one test program to stream "my" file and that shows hardly any stutter. I'm not saying that the Mediaportal version need to b e upgraded, since that was on a much more powerful computer with RAID disks, but anyway good to know that it can be ,made to work!

Looking at the liveMedia code there are remarks about how the read from file are being done synchronously on Windows. And although Mediaportal have two special classes(MultiFileReader and FileReader) they are still reading from file synchronously as part of serving the packet to the client - which thus causes this stutter.

Anyway, the good thing is that I realized that the streaming part of TvServer is quite self contained and not that much code(if you exclude the actual setup of the streaming). Comparing it againt latest liveMedia555 source and incorporate any bugfixes should be quite easy. I see _one_ thing, but hard to say if it affects this at all.

Also looked at Windows socket 2 to understand the "we need Winsock 2 rumor" and would say that is not the problem in this case. The code already initializes winsock to use version 2(which has been around since Windows 2000). And of course there are some "new" features but I have a hard time to see which one would benefit the streaming server. Of course Scatter I/O and QOS sound nice but how does it fit into the current architecture?

Comments?

Funny quote about bzr

"Merging in Bazaar is easy, as the implementation is able to avoid many
spurious conflicts, deals well with repeated merges between branches,
and is able to handle modifications to renamed files correctly."

Debugging MySQL Cluster data nodes(ndbd) using ndbout

One basic way of debugging the MySQL Cluster data nodes(ndbd) is with good old printf-style debugging. Ie. adding printouts of the variable you want to see, recompile, start the node(s) and run your tests. This is a bit tedious but very universal.

Support for printing is available using "ndbout", which has both C++ style and printf style. This functionality is implemented in storage/ndb/include/util/NdbOut.hpp

Here is an example from debugging BUG#37592, where I wanted to add printouts to "Dbtup::rebuild_page_free_list" to see how the pageId increases while the rebuild is running:

void
Dbtup::rebuild_page_free_list(Signal* signal)
{
  Ptr fragOpPtr;
  fragOpPtr.i = signal->theData[1];
  Uint32 pageId = signal->theData[2];
  Uint32 tail = signal->theData[3];
  ptrCheckGuard(fragOpPtr, cnoOfFragoprec, fragoperrec);

Using "ndbout" with printf style:

 ndbout_c("Dbtup::rebuild_page_free_list, pageId: %u", pageId);

In this case you have to think about what datatype "pageId" is to print it out correctly.
Newline is always added to the end of the printout.

Using C++ style:

 ndbout << "Dbtup::rebuild_page_free_list, pageId: " << pageId << endl;

In this case you need not worry about the datatype of pageId it will automatically select the correct conversion.

The stdout from ndbd will when I run it look like this:

Dbtup::rebuild_page_free_list, pageId: 65
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 67
Dbtup::rebuild_page_free_list, pageId: 67

Using the C++ style it is also possible to add a custom printer to print a whole class or struct easily. An example of that is the one for Restore::Column which in ndb/src/kernel/blocks/restore.cpp looks likes this:

NdbOut& 
operator << (NdbOut& ndbout, const Restore::Column& col)
{
  ndbout << "[ Col: id: " << col.m_id 
  << " size: " << col.m_size 
  << " key: " << (Uint32)(col.m_flags & Restore::Column::COL_KEY)
  << " variable: " << (Uint32)(col.m_flags & Restore::Column::COL_VAR)
  << " null: " << (Uint32)(col.m_flags & Restore::Column::COL_NULL)
  << " disk: " << (Uint32)(col.m_flags & Restore::Column::COL_DISK) 
  << "]";

  return ndbout;
}

Hopefully this give some idea about how to start working with a new feature or how to see what is going on in the black box. Will write more about other ways in the future.

Grouping .test files into suites

Please, enlighten me on what being in a separate suite means! I see there is a suite directory in mysql-test. Will the suites be run by default when I use 'mtr' to run tests? Or do I have to add them manually?
=========================================

The number of tests we have for MySQL Server are constantly growing. There is a need to group them in different ways so we can select what and where to run. We do this by using suites, either the default suite that we call "main" in mysql-test/t or one of the subdirs of mysql-test/suite.

As each test becomes more advanced it's also necessary to use different configurations for a particular test or suite. For example all the replication tests in suite/rpl need to be run with the server started in three different ways(three different configurations) to get full coverage. To avoid that the individual developer or the "one" running tests have to remember different parameters to run the tests with, we have made it possible to make each suite special. That means it can have different number of MySQL Servers and different settings of each server.

The current mysql-test-run.pl in MySQL 5.1 has:
- support for running tests from suite/ directory
- each suite can have some special setting with the use of a suite.opt file.
- each suite can make each test to be run more different configurations by adding a combinations file(see for example suite/rpl/). That file will cause all .test to be "multiplied" by the number of combinations and thus run several times with different server settings.
- All the standard suites to look for are added to mysql-test-run.pl to makerit as easy as possible for a developer to run the default set of tests.

We also have a new mysql-test-run.pl that is going into 5.1 and up very soon. it has been extended to also support:
- allowing each suite to have it's own my.cnf file(s), this way we can run tests with various settings of the server.
- that each .test has it's own .cnf file. For example rpl_circular_for_4_hosts.test(4 way circular replication) need a config file with 4 MySQL Server's in it, thus it adds rpl_circular_for_4_hosts.cnf along side the .test file.
- to find suites in storage/ to make it possible to put storage engine specific tests along side the code for that storage engine
- to find a suite in any directory by simply giving the full path to where that suite is.

In the future we'll continue to make it possible for each suite to be even more special(in a pluggable way) as we find the need for it.

How to concatenate strings in a mysqltest testcase

Hi Magnus, is there a way to concatenate strings in a mysqltest?
=================================================================


Yes, you can "easily" create more dynamic strings using let and a while loop. For example like this:

let $c= 254;
let $str= t255;

while ($c)
{
     let $str= t$c,$str;
     dec $c;
 }
 echo $str;

This will printout t1, t2, t3, ... t255.


You can then use the $str variable in an eval
to CREATE TABLE, VIEW or SELECT

eval CREATE TABLE t0 (a INT) ENGINE=MERGE UNION($str);

How to restart a server after an expected crash or shutdown in a test case?

There are now two ways how to do this:

1. Using "set debug=d,flag" and adding some DBUG_EXECUTE_IF("flag", abort()) we can trigger the server to crash at a specific place in the code. There is an example of this in crash_commit_before.test
This requires a debug compiled server so you need to add a "source include/have_debug.inc" at the beginning of the test.

2. There is also a new way to do it without using DBUG_. By using the new command in mysqltest called "shutdown_server"[1], we will tell the server to stop, wait a while(60 seconds) and finally kill it off. This way we get a reliable shutdown. The first testcase that uses this is events_restart.test

In both of these methods, the testcase has to write a small file before the "crash/shutdown", that file tells mysql-test-run.pl that it was an expected crash/shutdown. The server will be started up again with the same settings as before crash/shutdown.


[1] "shutdown_server" is a new command in mysqltest, not yet pushed to the main trees. It will appear in 5.1 and up.
The syntax is:
shutdown_server [timeout]
if the server's pid file is not gone after timeout seconds, the process will be killed. Default timeout value is 60 seconds.