The MySQL Cluster patches published

Spent some time last night to split apart the big patch of improvements that we have added to MySQL Server 5.1 over the years. It's now 8 patches, and looks like there going to be at least three more on top of this.

The patches are organized as a series of quilt patches which I pushed to launchpad with bazaar. They can be browsed online http://bazaar.launchpad.net/~msvensson/+junk/mysql-cluster-patches/files or use "bzr branch lp:~msvensson/+junk/mysql-cluster-patches" . Currently they apply to mysql-5.1.34, but I'll try to keep them updated as we merged in future versions of MySQL Server

1. already_in_next_mysql - Fortunately I could quickly isolate all the changes from MySQL Cluster 6.2 into one patch since they have already been merged back up into the next version of MySQL Server. That shaved almost a third of the changes off.
2. wl4775_dynamic_ndb_variables - This is new for 7.1 and moves all variables and options for ha_ndbcluster, from the MySQL Server source code. Using the dynamic variable support. The patch simply removes the stuff from mysqld.cc, set_var.cc and sql_class.h.
   
3. wl3126_bind_adress_for_client - This is WL#3126 which add the possibility for all the mysql clients as well as programs using libmysql to specify which interface to bind to. It's described more detailed in the WL. I think it's a very interesting improvement, only thing that is bad is that it modifies the ABI of libmsyql since it add a new field to "struct st_mysql_option". This has to be rewritten so that it instead put this information in the mysql_option.extension pointer(a new field we added last time the ABI version was changed) to avoid breaking the ABI. Also like to point out to users of MySQL Cluster that due to this change you can't just take any mysql program with dynamic linking and use it with a libmysqlclient from MySQL Cluster. This is actually quite bad and I think we should actually try to fix the ABI in MySQL Cluster to be the same as of the base MySQL Server version that has been sourced. Filed BUG#45344 for this problem, let's see if we can fix it somehow. Until then, beware! This was added in MySQL Cluster 6.3.4.
   
4. backport_dbug_win_perf - This is a backport from MySQL Server 6.0 added as part of porting MySQL Cluster 7.0 to windows.
5. always_use_tls.patch - Another backport that changes the pthread implementation for windows to always use TLS. This is a nice cleanup and avoid libmysqlclient to be compiled differently for .lib and .dll on windows.
6. joinable_pthreads_win - Extend mysys to make it possible to use pthread_join on Windows. All threads in MySQL Server are created detached - this is by historical reasons since there was a limit on number of joinable threads in linux - and thus support for joinable threads was never implemented when MySQL Server was ported to Windows. Since MySQL Cluster always use joinable threads, this is a requirement to make it run on windows. There aren't so many threads and normally all threads are shutdown in an organized fashion before the program ends, using pthread_join. This is not available in next version of MYSQL Server so it need to be added. (It looks like I have put some of the always_use_tls patches in here by mistake).
7. mysqltest_win_replace_fix - Fixes a problem in mysqltest where it performed replace on strings too many times. This code was only run on windows, it was detected when porting MySQL Cluster to windows. Should probably file a bug and backport it to 5.1
8. my_socket - Refactoring of the socket code in MySQL Server to make it impossible to write code that is not portable. The idea is to use a "struct my_socket" to encapsulate the differences between *nix and Windows. This patch has been around and reviewed over and over again, but never made it into the main MySQL Server - it should.

Remaining things to split out is

• heartbeat for replication (backport)
• IPV6(backport)
• some new field flags
• and a couple of bug fixes

But at least my original diff has shrunk significantly and diffstat tells me "31 files changed, 1164 insertions(+), 384 deletions(-)"

Hope too sort the remaining parts out soon.

Introducing the MySQL Cluster patch(es)

The current release of MySQL Cluster 7.0 is based on MySQL Server 5.1.34, normally we update the MySQL Server version as soon as a new one has been released. That is an almost automated process since it's just another branch in bazaar - ie "bzr pull", resolve any conflicts and commit.

The cluster team mainly work with the files in storage/ndb/ where the source for ndbd, ndb_mgmd and all the ndb_* tools for working with MySQL Cluster is kept. We have also produced improvements to the MySQL Server itself. While most of them have been merged back up - either to 5.1 or 6.0 - some hasn't. This means that when someone touch an are that has been improved we get a few conflicts when merging down the latest MySQL Server version. Fortunately that is quite rare now when 5.1 is GA.

Some of the improvements are generic and simply improves MySQL Server's portability, while some are specific and useful only for ha_ndbcluster(the MySQL Cluster handler). The intention has always been to merge the improvements back into an upcoming release of MySQL Server, unfortunately that has been rare lately.

That's why I started to examine the differences between 5.1.34 and latest 7.0 and produced a patch. Currently the unified diff is 11207 lines(not that much) and diffstat shows 77 files changed with 4082 insertions and 1868 deletions. Note that this does not include anything from storage/ndb. My plan is to split that big patch into a series of smaller patches so it get manageable, can be described what they do and we can determine which parts have been merged back and which hasn't. I have also spotted a few changes that we're just going to revert since they simply change formatting or "my_bool->bool".

Will get back with more detailed information what the improvements are when after the patch has been split up.

In the long run, this will make it possible to take any version of MySQL Server, copy in the latest from storage/ndb/, compile and have a MySQL Server that works with MySQL Cluster. There are of course more changes required to make sure that everything that is part of MySQL Cluster is kept in storage/ndb and some of them are already in the works for the next version. For example moving all the ndb_ system variables away from mysqld.cc and into ha_ndbcluster.cc.


You can view the full patch with diffstat here