MDS database performance improvements

Recently, work has been done to the MDS update scheduling engine. Under the hood, an SQLite database is used to stage all that the MDS has to do. This work involves scheduling replication activity, resolving partial truncate(2) blocks, and garbage reclamation.

Until now, an elementary method was used to store this work and retrieve it when convenient. All execution threads inside the MDS wishing to do queries performed them in a shared mode of access no caching, placing limits on database performance.

The code has been restructured with the following improvements:

• pass as many database queries as possible to a single thread to prevent cache flushing.
• give each thread its own database handle to provide concurrent read operations.
• move the database into RAM via tmpfs and /dev/shm.
• backup the database occasionally in the event of a crash.

Obviously, between backups, there is a window for lost operations. This issue will need to be addressed.