Tuesday, June 24, 2008

Debugging MySQL Cluster data nodes(ndbd) using ndbout

One basic way of debugging the MySQL Cluster data nodes(ndbd) is with good old printf-style debugging. Ie. adding printouts of the variable you want to see, recompile, start the node(s) and run your tests. This is a bit tedious but very universal.

Support for printing is available using "ndbout", which has both C++ style and printf style. This functionality is implemented in storage/ndb/include/util/NdbOut.hpp

Here is an example from debugging BUG#37592, where I wanted to add printouts to "Dbtup::rebuild_page_free_list" to see how the pageId increases while the rebuild is running:

Dbtup::rebuild_page_free_list(Signal* signal)
Ptr fragOpPtr;
fragOpPtr.i = signal->theData[1];
Uint32 pageId = signal->theData[2];
Uint32 tail = signal->theData[3];
ptrCheckGuard(fragOpPtr, cnoOfFragoprec, fragoperrec);

Using "ndbout" with printf style:
 ndbout_c("Dbtup::rebuild_page_free_list, pageId: %u", pageId);

In this case you have to think about what datatype "pageId" is to print it out correctly.
Newline is always added to the end of the printout.

Using C++ style:
 ndbout << "Dbtup::rebuild_page_free_list, pageId: " << pageId << endl;

In this case you need not worry about the datatype of pageId it will automatically select the correct conversion.

The stdout from ndbd will when I run it look like this:

Dbtup::rebuild_page_free_list, pageId: 65
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 66
Dbtup::rebuild_page_free_list, pageId: 67
Dbtup::rebuild_page_free_list, pageId: 67

Using the C++ style it is also possible to add a custom printer to print a whole class or struct easily. An example of that is the one for Restore::Column which in ndb/src/kernel/blocks/restore.cpp looks likes this:

operator << (NdbOut& ndbout, const Restore::Column& col)
ndbout << "[ Col: id: " << col.m_id
<< " size: " << col.m_size
<< " key: " << (Uint32)(col.m_flags & Restore::Column::COL_KEY)
<< " variable: " << (Uint32)(col.m_flags & Restore::Column::COL_VAR)
<< " null: " << (Uint32)(col.m_flags & Restore::Column::COL_NULL)
<< " disk: " << (Uint32)(col.m_flags & Restore::Column::COL_DISK)
<< "]";

return ndbout;

Hopefully this give some idea about how to start working with a new feature or how to see what is going on in the black box. Will write more about other ways in the future.

No comments: