The MySQL Cluster patches published

Spent some time last night to split apart the big patch of improvements that we have added to MySQL Server 5.1 over the years. It's now 8 patches, and looks like there going to be at least three more on top of this.

The patches are organized as a series of quilt patches which I pushed to launchpad with bazaar. They can be browsed online http://bazaar.launchpad.net/~msvensson/+junk/mysql-cluster-patches/files or use "bzr branch lp:~msvensson/+junk/mysql-cluster-patches" . Currently they apply to mysql-5.1.34, but I'll try to keep them updated as we merged in future versions of MySQL Server

1. already_in_next_mysql - Fortunately I could quickly isolate all the changes from MySQL Cluster 6.2 into one patch since they have already been merged back up into the next version of MySQL Server. That shaved almost a third of the changes off.
2. wl4775_dynamic_ndb_variables - This is new for 7.1 and moves all variables and options for ha_ndbcluster, from the MySQL Server source code. Using the dynamic variable support. The patch simply removes the stuff from mysqld.cc, set_var.cc and sql_class.h.
   
3. wl3126_bind_adress_for_client - This is WL#3126 which add the possibility for all the mysql clients as well as programs using libmysql to specify which interface to bind to. It's described more detailed in the WL. I think it's a very interesting improvement, only thing that is bad is that it modifies the ABI of libmsyql since it add a new field to "struct st_mysql_option". This has to be rewritten so that it instead put this information in the mysql_option.extension pointer(a new field we added last time the ABI version was changed) to avoid breaking the ABI. Also like to point out to users of MySQL Cluster that due to this change you can't just take any mysql program with dynamic linking and use it with a libmysqlclient from MySQL Cluster. This is actually quite bad and I think we should actually try to fix the ABI in MySQL Cluster to be the same as of the base MySQL Server version that has been sourced. Filed BUG#45344 for this problem, let's see if we can fix it somehow. Until then, beware! This was added in MySQL Cluster 6.3.4.
   
4. backport_dbug_win_perf - This is a backport from MySQL Server 6.0 added as part of porting MySQL Cluster 7.0 to windows.
5. always_use_tls.patch - Another backport that changes the pthread implementation for windows to always use TLS. This is a nice cleanup and avoid libmysqlclient to be compiled differently for .lib and .dll on windows.
6. joinable_pthreads_win - Extend mysys to make it possible to use pthread_join on Windows. All threads in MySQL Server are created detached - this is by historical reasons since there was a limit on number of joinable threads in linux - and thus support for joinable threads was never implemented when MySQL Server was ported to Windows. Since MySQL Cluster always use joinable threads, this is a requirement to make it run on windows. There aren't so many threads and normally all threads are shutdown in an organized fashion before the program ends, using pthread_join. This is not available in next version of MYSQL Server so it need to be added. (It looks like I have put some of the always_use_tls patches in here by mistake).
7. mysqltest_win_replace_fix - Fixes a problem in mysqltest where it performed replace on strings too many times. This code was only run on windows, it was detected when porting MySQL Cluster to windows. Should probably file a bug and backport it to 5.1
8. my_socket - Refactoring of the socket code in MySQL Server to make it impossible to write code that is not portable. The idea is to use a "struct my_socket" to encapsulate the differences between *nix and Windows. This patch has been around and reviewed over and over again, but never made it into the main MySQL Server - it should.

Remaining things to split out is

• heartbeat for replication (backport)
• IPV6(backport)
• some new field flags
• and a couple of bug fixes

But at least my original diff has shrunk significantly and diffstat tells me "31 files changed, 1164 insertions(+), 384 deletions(-)"

Hope too sort the remaining parts out soon.

Introducing the MySQL Cluster patch(es)

The current release of MySQL Cluster 7.0 is based on MySQL Server 5.1.34, normally we update the MySQL Server version as soon as a new one has been released. That is an almost automated process since it's just another branch in bazaar - ie "bzr pull", resolve any conflicts and commit.

The cluster team mainly work with the files in storage/ndb/ where the source for ndbd, ndb_mgmd and all the ndb_* tools for working with MySQL Cluster is kept. We have also produced improvements to the MySQL Server itself. While most of them have been merged back up - either to 5.1 or 6.0 - some hasn't. This means that when someone touch an are that has been improved we get a few conflicts when merging down the latest MySQL Server version. Fortunately that is quite rare now when 5.1 is GA.

Some of the improvements are generic and simply improves MySQL Server's portability, while some are specific and useful only for ha_ndbcluster(the MySQL Cluster handler). The intention has always been to merge the improvements back into an upcoming release of MySQL Server, unfortunately that has been rare lately.

That's why I started to examine the differences between 5.1.34 and latest 7.0 and produced a patch. Currently the unified diff is 11207 lines(not that much) and diffstat shows 77 files changed with 4082 insertions and 1868 deletions. Note that this does not include anything from storage/ndb. My plan is to split that big patch into a series of smaller patches so it get manageable, can be described what they do and we can determine which parts have been merged back and which hasn't. I have also spotted a few changes that we're just going to revert since they simply change formatting or "my_bool->bool".

Will get back with more detailed information what the improvements are when after the patch has been split up.

In the long run, this will make it possible to take any version of MySQL Server, copy in the latest from storage/ndb/, compile and have a MySQL Server that works with MySQL Cluster. There are of course more changes required to make sure that everything that is part of MySQL Cluster is kept in storage/ndb and some of them are already in the works for the next version. For example moving all the ndb_ system variables away from mysqld.cc and into ha_ndbcluster.cc.


You can view the full patch with diffstat here

Finding the source for MySQL 5.4

In order to really check what currently has made it into 5.4(there has been rumors...) I wanted to branch the code and see for myself.

Couldn't find it at first in our internal code repository since it's actually not named 5.4 there yet. Fortunately it's also pushed out to launchpad, that means you can easily get a copy of the code using bazaar

$> bzr branch lp:mysql-server/5.4

There is of course also tar balls available from our download page
http://dev.mysql.com/downloads/mysql/5.4.html#source

Anyone curious about what our internal name is? :)

Stutter in MediPortal streaming server

Reading the Mediaportal forum thread and seeing that others have similar stutter as I have been annoyed about for the time I've been using MediaPortal in multiseat mode with RTSP. After having compiled the DirectShowFilters solution, I modified the StreamingServer example application to stream a 2.5GB recording I had on my disk. This made it fairly easy to debug by both attaching the debugger and also adding printouts(my favourite).

My analyze shows that(at least my) stutter occurs when TsBuffer.cpp reads more data from the file using 'FileReader::read'. The time to read from file is normally below 50 usec, but sometimes I can see it takes one second or more.

Will attempt to try and fix the problem by one of:
* Add a thread to TSBuffer that keeps the buffer full by reading the following part's of the file asynchronously in the background. Since TSBuffer is using either MultiFileReader or FileReader to read from file, it's probably better to use a thread then to change both of those classes to use asynch IO with Overlapped IO. Well, that coudl be discussed, some what of reading ahead should be done anyway.
* Seems like there is less stutter when using MultiFileReader(could be because the file is "hot" since it's being recorded to). Could be a bug with FileReader, Need to be examined. For example, why are all those SetFilePointer done when the file is already positioned at the correct place? Maybe they are expensive calls?
* Start to read from file as part of putting the "frame" into the queue rather than doing the read when taking it out of the there. This could give some additional time for the read(if I have read the code correct).
* Buy new hardware :)


Have also downloaded and compiled latest version of liveMedia555 on linux, configured one test program to stream "my" file and that shows hardly any stutter. I'm not saying that the Mediaportal version need to b e upgraded, since that was on a much more powerful computer with RAID disks, but anyway good to know that it can be ,made to work!

Looking at the liveMedia code there are remarks about how the read from file are being done synchronously on Windows. And although Mediaportal have two special classes(MultiFileReader and FileReader) they are still reading from file synchronously as part of serving the packet to the client - which thus causes this stutter.

Anyway, the good thing is that I realized that the streaming part of TvServer is quite self contained and not that much code(if you exclude the actual setup of the streaming). Comparing it againt latest liveMedia555 source and incorporate any bugfixes should be quite easy. I see _one_ thing, but hard to say if it affects this at all.

Also looked at Windows socket 2 to understand the "we need Winsock 2 rumor" and would say that is not the problem in this case. The code already initializes winsock to use version 2(which has been around since Windows 2000). And of course there are some "new" features but I have a hard time to see which one would benefit the streaming server. Of course Scatter I/O and QOS sound nice but how does it fit into the current architecture?

Comments?