Virtual machine migration can be thought of as a special form of suspend/resume, in which the state file is immediately transferred and resumed on a different target machine. Migration is particularly attractive in the data center, where it allows the current workload to be balanced dynamically across available rack space.
However, although Xen’s suspend/resume mechanism is very efficient, it may not be suitable for migrating latency-sensitive or high-availability applications. This is because the virtual machine cannot resume execution until its state file has been transferred to the target system, and this delay is largely determined by its memory size: a complex VM with a large memory allocation takes a correspondingly long time to transfer.
To avoid prolonged downtimes, Xen provides a migration engine that transfers a VM’s configuration information and memory image while the VM is still executing. The goal of the migration engine is to stop execution of the VM only while its (relatively tiny) register state is transferred. The “fly in the ointment” is that this can lead to an inconsistent memory image at the target machine if the VM modifies a memory location after it’s been copied. Xen avoids these inconsistencies by detecting when a memory page is updated after it is copied, and retransferring that page.
To do this without requiring OS modifications, Xen installs a shadow page table beneath the VM. In this mode of operation, the guest’s page table is no longer registered with the MMU. Instead, regions of the guest page table are translated and copied into the shadow table on demand.
Shadow page tables are not new: they are used in fully-virtualizing machine monitors such as VMware’s products to translate between a guest’s view of a contiguous physical memory space and the reality that its memory pages are scattered across the real physical memory space.
Shadow page tables are not used by the migration engine for full translation, but for dirty logging. The page mappings in the shadow table are therefore identical to those in the guest table, except for pages that the migration engine has transferred to the target system. Transferred pages are always converted to read-only access when their mappings are copied into the shadow table, and any attempt to update such a page causes a page fault. When Xen observes a write-access fault on a transferred page, it marks the page as “dirtied,” which informs the migration engine to schedule another transfer. Writable mappings of the page are again permitted until the page is retransferred (and again marked read-only).