Debugging MySQL Cluster data nodes(ndbd) using ndbout

One basic way of debugging the MySQL Cluster data nodes(ndbd) is with good old printf-style debugging. Ie. adding printouts of the variable you want to see, recompile, start the node(s) and run your tests. This is a bit tedious but very universal.

Support for printing is available using "ndbout", which has both C++ style and printf style. This functionality is implemented in storage/ndb/include/util/NdbOut.hpp

Here is an example from debugging BUG#37592, where I wanted to add printouts to "Dbtup::rebuild_page_free_list" to see how the pageId increases while the rebuild is running:

void
Dbtup::rebuild_page_free_list(Signal* signal)
{
  Ptr fragOpPtr;
  fragOpPtr.i = signal->theData[1];
  Uint32 pageId = signal->theData[2];
  Uint32 tail = signal->theData[3];
  ptrCheckGuard(fragOpPtr, cnoOfFragoprec, fragoperrec);

Using "ndbout" with printf style:

 ndbout_c("Dbtup::rebuild_page_free_list, pageId: %u", pageId);

In this case you have to think about what datatype "pageId" is to print it out correctly.
Newline is always added to the end of the printout.

Using C++ style:

 ndbout << "Dbtup::rebuild_page_free_list, pageId: " << pageId << endl;

In this case you need not worry about the datatype of pageId it will automatically select the correct conversion.

The stdout from ndbd will when I run it look like this:

Dbtup::rebuild_page_free_list, pageId: 65
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 67
Dbtup::rebuild_page_free_list, pageId: 67

Using the C++ style it is also possible to add a custom printer to print a whole class or struct easily. An example of that is the one for Restore::Column which in ndb/src/kernel/blocks/restore.cpp looks likes this:

NdbOut& 
operator << (NdbOut& ndbout, const Restore::Column& col)
{
  ndbout << "[ Col: id: " << col.m_id 
  << " size: " << col.m_size 
  << " key: " << (Uint32)(col.m_flags & Restore::Column::COL_KEY)
  << " variable: " << (Uint32)(col.m_flags & Restore::Column::COL_VAR)
  << " null: " << (Uint32)(col.m_flags & Restore::Column::COL_NULL)
  << " disk: " << (Uint32)(col.m_flags & Restore::Column::COL_DISK) 
  << "]";

  return ndbout;
}

Hopefully this give some idea about how to start working with a new feature or how to see what is going on in the black box. Will write more about other ways in the future.