Migrating memory state is conceptually quite straightforward -- all needed memory is collected and shipped to the new processor. This is complicated in practice by various low-level difficulties.
For heap data, we must collect all the allocated data for a particular piece of work. For object-oriented applications, we have built a convenient and general-purpose framework for describing and shipping complicated user-defined objects called the PUP (Pack/UnPack) Framework [19]. For more general applications, it is possible to provide hooks to the memory allocation routines (malloc) to capture all the memory allocated by each thread.
For global variables, process migration is straightforward because each process image contains a separate copy of the global variables. For thread migration, however, if several threads in a process share global variables, it may be impossible to extract the value needed by each thread. Our current solution is to privatize global variables so that each user-level thread has its own copy of the global variables. Thus migration of global variables is conceptually simplified. In order to support the transparent privatization of global variables for existing codebases, we implemented a swap-global scheme by analyzing the Executable and Linking Format (ELF) object files in a way similar to the Weaves runtime framework [31]. A dynamically linked ELF file format executable always accesses global variables via the Global Offset Table (GOT), which contains one pointer to each global variable. To make separate copies of the global variables, we then simply make separate copies of the GOT -- one for each user-level thread. The thread scheduler then swaps the GOT when switching threads.