Techniques for Warming Up a MongoDB Secondary

At Parse we make extensive use of MongoDB replica sets. This is a configuration where you have one primary node and two or more secondary nodes, which can optionally accept read queries, run snapshots and backups, compaction scripts, and so forth.
 
If you are running replica sets, you will occasionally need to elect a new primary. Maybe you need to elect a freshly compacted node as the primary, or you want to move to a different class of hardware or a node with larger volume space. However, your primary is fast because the working set has already been read into RAM. Reading the working set into RAM on a freshly elected secondary is going to be slow and spiky, and you are likely to suffer extremely degraded performance while every query is page faulting and you are simultaneously trying to serve live traffic. (And if you are electing a secondary that has been recently restored from EBS snapshot, performance will be exponentially worse still — EBS snapshot blocks are lazily-loaded off S3.)
 
The answer, obviously, is to warm up your secondary before electing it master. There are several ways to approach this. If your working set is smaller than your available RAM, the simplest solution is just to use dd(1) or vmtouch(8) to load all the MongoDB data files into memory. For example,
 
for file in /var/lib/mongodb/* ; do
  vmtouch -t -m 10G $file
done
for file in /var/lib/mongodb/* ; do
  time dd if=$file of=/dev/null bs=16m
done
If you are restoring from EBS snapshot, the dd and vmtouch utilities are also a great way to touch all of your data and make sure that it’s been downloaded from S3.
 
Touching your data will only read the data into memory, however, and ideally you should warm up both your data files and your indexes. For that you can do a natural sort on all of your collections, or a full table scan, or search for something guaranteed not to be there — any of these will load both your indexes and your data into RAM.

 

This problem gets trickier if you have lots of data. If you have a terabyte of data and only 64 gigs of RAM, how do you choose what to load in to memory?

Our answer to this was to write a pair of utilities: mongo_gatherops.rb and mongo_preheat.rb. The first script runs on the old primary prior to the switch and samples the current ops every quarter second for a configurable number of samples, then sorts and outputs the list of most-active collections to a file. So this command gathers all the collections accessed for 30 minutes (7200 quarter-second samples), and outputs a sorted list of collections:
$ ruby mongo_gatherops.rb 7200 > top_collections_20130306

You can then copy the list to the secondary and use it as an input to the preheat script, which runs a full table scan on each of the collections and reads its indexes into memory.

$ ruby mongo_preheat.rb top_collections_20130306

This takes quite a while to run, but it’s worth it. Once your working set is loaded into RAM, you can elect a new primary with no site outages or degradation of performance.

We’ve made these tools available on our public github repo, at https://github.com/ParsePlatform/Ops. Enjoy!
 
Charity Majors
March 7, 2013
blog comments powered by Disqus

Comments are closed.

Archives

Categories

RSS Feed Follow us Like us