Building MySQL Cluster 7.0 on Windows

This walkthrough will show you how to configure your more or less clean install of Windows(you can use XP, Vista or 7) in order to build MySQL Cluster on Windows. The setup also works for the vanilla MySQL Server since the only real difference is the configure switches used to activate build of the NDB storage engine and the NDBCLUSTER handler.

Installing the necessary tools

• Install bison from GnuWin32 - http://gnuwin32.sourceforge.net/packages/bison.htm, this program is used generate the yacc parser in MySQL Server and is invoked automatically by a build rule. To avoid problem's with spaces in the path name, I recommend installing it to non default location c:\GnuWin32(i.e remove the "Program Files (x86)" part) - this is luckily the only tools that seem to have problem with spaces in the path.
  
• Install diff from GnuWin32 - http://gnuwin32.sourceforge.net/packages/diffutils.htm, this program is used by the test programs (to show the difference between the actual and expected result if a .test file run by mysql-test-run.pl should fail) and also to generate a somewhat more advanced diff for commit mails. This package will also go into c:\GnuWin32(the installer will autodetect it).
• Add the bin directory of your GnuWin32 installation to your PATH to make both of the above programs executable from command line. (i.e append ";c:\GnuWin32\bin")
• Install Visual Studio Express 2008 - http://www.microsoft.com/Express/VC/, normally you can skip installing any extras like Silverlight and Microsoft SQL Server.
• Install Cmake from http://www.cmake.org/, there is a brand new version 2.8 and it seems to work fine. Tell installer to add cmake to your PATH.
• Install the version control tool Bazaar -http://wiki.bazaar.canonical.com/WindowsDownloads - using the standalone package installer. Use default settings to make it add itself to PATH.
  

Getting the source using bzr
NOTE! You can also download a "source release" aka. "source dist" from http://dev.mysql.com/downloads/cluster/ and select "Source Code" as your platform and then download the "Genric Linux (Architecture independent), Compressed TAR Archive", unpack and build. But since this is the hardcore instructions I'm using Bazaar to do some real development.

The public branch of MySQL Cluster 7.0 is hosted on launchpad, it's a readonly mirror of our internal repository and the details about the branch can be found at https://code.launchpad.net/~mysql/mysql-server/mysql-cluster-7.0 . For example it's possible to see the latest pushes and check the diff and commit messages for each push.

• Open a command prompt and change to the directory where you like to keep your MySQL clones. Normally you use one clone for each task you work on so it's good to initialize a shared repository. It's very important to use the option --pack-0.92 to create a repository using the same format as the MySQL repositories, otherwise bzr will try to convert from our "old" format to the latest and that will use up all your RAM and CPU for quite some time(haven't heard anyone that been patient enough to wait for it to complete).
  
  

  M:\> bzr init-repo --pack-0.92 .
  

• The branch location is lp:~mysql/mysql-server/mysql-cluster-7.0 and it may unfortunately take some time to do the first branch over the internet so be patient. Unless you are logged in to launchpad, you will get a complaint from bzr - just ignore it(although rumors say that something called "streaming" will be turned on if you do login and that should speed up the cloning).
  
  

  M:\> bzr branch lp:~mysql/mysql-server/mysql-cluster-7.0 7.0

• Change into the newly branched tree and run the "windows configure", a small javascript that will generate a file that tells cmake which parts to include into the build. I'm showing tree different configure lines below and you should only select one of them. The first one is just MySQL Server with NDB, the second one adds all our test programs for MySQL Cluster and finally the third is the configure line we're curently using to build the test builds in our continous integrion system.
  

  
  M:\> cd 7.0
  M:\7.0> cscript win\configure.js WITH_NDBCLUSTER_STORAGE_ENGINE
  M:\7.0> cscript win\configure.js WITH_NDBCLUSTER_STORAGE_ENGINE WITH_NDB_TEST
  M:\7.0> cscript win\configure.js WITH_INNOBASE_STORAGE_ENGINE WITH_PARTITION_STORAGE_ENGINE WITH_ARCHIVE_STORAGE_ENGINE WITH_BLACKHOLE_STORAGE_ENGINE WITH_EXAMPLE_STORAGE_ENGINE WITH_FEDERATED_STORAGE_ENGINE WITH_EMBEDDED_SERVER WITH_NDBCLUSTER_STORAGE_ENGINE WITH_NDB_TEST __NT__ "COMPILATION_COMMENT=MySQL Cluster test build XXX"
  

• Now it's time to make Cmake do it's magic and generate the Visual Studio project files. It will create a solution file called MySQL.sln and one project for every binary and lib we have selected to build.

M:\7.0> cmake -G "Visual Studio 9 2008"

• Open the resulting MySQL.sln in Visual Studio. By default the "Active Configuration" will be "Debug" and although it now works to run the binaries generated with debug, my recomendation is still to switch to "RelWithDebinfo". Hopefully everything works and the build succeeds. The most common problem here is that it does not find bison and you get a build failure for sql_yacc.cc, in that case just check bison is in your path.
  

The MySQL Cluster patches published

Spent some time last night to split apart the big patch of improvements that we have added to MySQL Server 5.1 over the years. It's now 8 patches, and looks like there going to be at least three more on top of this.

The patches are organized as a series of quilt patches which I pushed to launchpad with bazaar. They can be browsed online http://bazaar.launchpad.net/~msvensson/+junk/mysql-cluster-patches/files or use "bzr branch lp:~msvensson/+junk/mysql-cluster-patches" . Currently they apply to mysql-5.1.34, but I'll try to keep them updated as we merged in future versions of MySQL Server

1. already_in_next_mysql - Fortunately I could quickly isolate all the changes from MySQL Cluster 6.2 into one patch since they have already been merged back up into the next version of MySQL Server. That shaved almost a third of the changes off.
2. wl4775_dynamic_ndb_variables - This is new for 7.1 and moves all variables and options for ha_ndbcluster, from the MySQL Server source code. Using the dynamic variable support. The patch simply removes the stuff from mysqld.cc, set_var.cc and sql_class.h.
   
3. wl3126_bind_adress_for_client - This is WL#3126 which add the possibility for all the mysql clients as well as programs using libmysql to specify which interface to bind to. It's described more detailed in the WL. I think it's a very interesting improvement, only thing that is bad is that it modifies the ABI of libmsyql since it add a new field to "struct st_mysql_option". This has to be rewritten so that it instead put this information in the mysql_option.extension pointer(a new field we added last time the ABI version was changed) to avoid breaking the ABI. Also like to point out to users of MySQL Cluster that due to this change you can't just take any mysql program with dynamic linking and use it with a libmysqlclient from MySQL Cluster. This is actually quite bad and I think we should actually try to fix the ABI in MySQL Cluster to be the same as of the base MySQL Server version that has been sourced. Filed BUG#45344 for this problem, let's see if we can fix it somehow. Until then, beware! This was added in MySQL Cluster 6.3.4.
   
4. backport_dbug_win_perf - This is a backport from MySQL Server 6.0 added as part of porting MySQL Cluster 7.0 to windows.
5. always_use_tls.patch - Another backport that changes the pthread implementation for windows to always use TLS. This is a nice cleanup and avoid libmysqlclient to be compiled differently for .lib and .dll on windows.
6. joinable_pthreads_win - Extend mysys to make it possible to use pthread_join on Windows. All threads in MySQL Server are created detached - this is by historical reasons since there was a limit on number of joinable threads in linux - and thus support for joinable threads was never implemented when MySQL Server was ported to Windows. Since MySQL Cluster always use joinable threads, this is a requirement to make it run on windows. There aren't so many threads and normally all threads are shutdown in an organized fashion before the program ends, using pthread_join. This is not available in next version of MYSQL Server so it need to be added. (It looks like I have put some of the always_use_tls patches in here by mistake).
7. mysqltest_win_replace_fix - Fixes a problem in mysqltest where it performed replace on strings too many times. This code was only run on windows, it was detected when porting MySQL Cluster to windows. Should probably file a bug and backport it to 5.1
8. my_socket - Refactoring of the socket code in MySQL Server to make it impossible to write code that is not portable. The idea is to use a "struct my_socket" to encapsulate the differences between *nix and Windows. This patch has been around and reviewed over and over again, but never made it into the main MySQL Server - it should.

Remaining things to split out is

• heartbeat for replication (backport)
• IPV6(backport)
• some new field flags
• and a couple of bug fixes

But at least my original diff has shrunk significantly and diffstat tells me "31 files changed, 1164 insertions(+), 384 deletions(-)"

Hope too sort the remaining parts out soon.

Introducing the MySQL Cluster patch(es)

The current release of MySQL Cluster 7.0 is based on MySQL Server 5.1.34, normally we update the MySQL Server version as soon as a new one has been released. That is an almost automated process since it's just another branch in bazaar - ie "bzr pull", resolve any conflicts and commit.

The cluster team mainly work with the files in storage/ndb/ where the source for ndbd, ndb_mgmd and all the ndb_* tools for working with MySQL Cluster is kept. We have also produced improvements to the MySQL Server itself. While most of them have been merged back up - either to 5.1 or 6.0 - some hasn't. This means that when someone touch an are that has been improved we get a few conflicts when merging down the latest MySQL Server version. Fortunately that is quite rare now when 5.1 is GA.

Some of the improvements are generic and simply improves MySQL Server's portability, while some are specific and useful only for ha_ndbcluster(the MySQL Cluster handler). The intention has always been to merge the improvements back into an upcoming release of MySQL Server, unfortunately that has been rare lately.

That's why I started to examine the differences between 5.1.34 and latest 7.0 and produced a patch. Currently the unified diff is 11207 lines(not that much) and diffstat shows 77 files changed with 4082 insertions and 1868 deletions. Note that this does not include anything from storage/ndb. My plan is to split that big patch into a series of smaller patches so it get manageable, can be described what they do and we can determine which parts have been merged back and which hasn't. I have also spotted a few changes that we're just going to revert since they simply change formatting or "my_bool->bool".

Will get back with more detailed information what the improvements are when after the patch has been split up.

In the long run, this will make it possible to take any version of MySQL Server, copy in the latest from storage/ndb/, compile and have a MySQL Server that works with MySQL Cluster. There are of course more changes required to make sure that everything that is part of MySQL Cluster is kept in storage/ndb and some of them are already in the works for the next version. For example moving all the ndb_ system variables away from mysqld.cc and into ha_ndbcluster.cc.


You can view the full patch with diffstat here

Finding the source for MySQL 5.4

In order to really check what currently has made it into 5.4(there has been rumors...) I wanted to branch the code and see for myself.

Couldn't find it at first in our internal code repository since it's actually not named 5.4 there yet. Fortunately it's also pushed out to launchpad, that means you can easily get a copy of the code using bazaar

$> bzr branch lp:mysql-server/5.4

There is of course also tar balls available from our download page
http://dev.mysql.com/downloads/mysql/5.4.html#source

Anyone curious about what our internal name is? :)

Stutter in MediPortal streaming server

Reading the Mediaportal forum thread and seeing that others have similar stutter as I have been annoyed about for the time I've been using MediaPortal in multiseat mode with RTSP. After having compiled the DirectShowFilters solution, I modified the StreamingServer example application to stream a 2.5GB recording I had on my disk. This made it fairly easy to debug by both attaching the debugger and also adding printouts(my favourite).

My analyze shows that(at least my) stutter occurs when TsBuffer.cpp reads more data from the file using 'FileReader::read'. The time to read from file is normally below 50 usec, but sometimes I can see it takes one second or more.

Will attempt to try and fix the problem by one of:
* Add a thread to TSBuffer that keeps the buffer full by reading the following part's of the file asynchronously in the background. Since TSBuffer is using either MultiFileReader or FileReader to read from file, it's probably better to use a thread then to change both of those classes to use asynch IO with Overlapped IO. Well, that coudl be discussed, some what of reading ahead should be done anyway.
* Seems like there is less stutter when using MultiFileReader(could be because the file is "hot" since it's being recorded to). Could be a bug with FileReader, Need to be examined. For example, why are all those SetFilePointer done when the file is already positioned at the correct place? Maybe they are expensive calls?
* Start to read from file as part of putting the "frame" into the queue rather than doing the read when taking it out of the there. This could give some additional time for the read(if I have read the code correct).
* Buy new hardware :)


Have also downloaded and compiled latest version of liveMedia555 on linux, configured one test program to stream "my" file and that shows hardly any stutter. I'm not saying that the Mediaportal version need to b e upgraded, since that was on a much more powerful computer with RAID disks, but anyway good to know that it can be ,made to work!

Looking at the liveMedia code there are remarks about how the read from file are being done synchronously on Windows. And although Mediaportal have two special classes(MultiFileReader and FileReader) they are still reading from file synchronously as part of serving the packet to the client - which thus causes this stutter.

Anyway, the good thing is that I realized that the streaming part of TvServer is quite self contained and not that much code(if you exclude the actual setup of the streaming). Comparing it againt latest liveMedia555 source and incorporate any bugfixes should be quite easy. I see _one_ thing, but hard to say if it affects this at all.

Also looked at Windows socket 2 to understand the "we need Winsock 2 rumor" and would say that is not the problem in this case. The code already initializes winsock to use version 2(which has been around since Windows 2000). And of course there are some "new" features but I have a hard time to see which one would benefit the streaming server. Of course Scatter I/O and QOS sound nice but how does it fit into the current architecture?

Comments?

Funny quote about bzr

"Merging in Bazaar is easy, as the implementation is able to avoid many
spurious conflicts, deals well with repeated merges between branches,
and is able to handle modifications to renamed files correctly."

Debugging MySQL Cluster data nodes(ndbd) using ndbout

One basic way of debugging the MySQL Cluster data nodes(ndbd) is with good old printf-style debugging. Ie. adding printouts of the variable you want to see, recompile, start the node(s) and run your tests. This is a bit tedious but very universal.

Support for printing is available using "ndbout", which has both C++ style and printf style. This functionality is implemented in storage/ndb/include/util/NdbOut.hpp

Here is an example from debugging BUG#37592, where I wanted to add printouts to "Dbtup::rebuild_page_free_list" to see how the pageId increases while the rebuild is running:

void
Dbtup::rebuild_page_free_list(Signal* signal)
{
  Ptr fragOpPtr;
  fragOpPtr.i = signal->theData[1];
  Uint32 pageId = signal->theData[2];
  Uint32 tail = signal->theData[3];
  ptrCheckGuard(fragOpPtr, cnoOfFragoprec, fragoperrec);

Using "ndbout" with printf style:

 ndbout_c("Dbtup::rebuild_page_free_list, pageId: %u", pageId);

In this case you have to think about what datatype "pageId" is to print it out correctly.
Newline is always added to the end of the printout.

Using C++ style:

 ndbout << "Dbtup::rebuild_page_free_list, pageId: " << pageId << endl;

In this case you need not worry about the datatype of pageId it will automatically select the correct conversion.

The stdout from ndbd will when I run it look like this:

Dbtup::rebuild_page_free_list, pageId: 65
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 67
Dbtup::rebuild_page_free_list, pageId: 67

Using the C++ style it is also possible to add a custom printer to print a whole class or struct easily. An example of that is the one for Restore::Column which in ndb/src/kernel/blocks/restore.cpp looks likes this:

NdbOut& 
operator << (NdbOut& ndbout, const Restore::Column& col)
{
  ndbout << "[ Col: id: " << col.m_id 
  << " size: " << col.m_size 
  << " key: " << (Uint32)(col.m_flags & Restore::Column::COL_KEY)
  << " variable: " << (Uint32)(col.m_flags & Restore::Column::COL_VAR)
  << " null: " << (Uint32)(col.m_flags & Restore::Column::COL_NULL)
  << " disk: " << (Uint32)(col.m_flags & Restore::Column::COL_DISK) 
  << "]";

  return ndbout;
}

Hopefully this give some idea about how to start working with a new feature or how to see what is going on in the black box. Will write more about other ways in the future.