Thread migration is very challenging due to the fact that the system and user state of a thread is only one part of an enclosing process. It is often very difficult to extract the state of one thread from the others in a process. For example, heap data allocated by a thread may be shared by multiple threads. In our work, we assume that the user data allocated by one thread is used only by that thread. Data sharing among threads can still be achieved via read-only global varaibles or fast local message passing via the thread scheduler.
Another difficult aspect of thread migration is the fact that a thread stack contains a large number of pointers (including function return addresses, frame pointers, and pointer variables), and many of these point into the stack itself. Thus, if we copy the thread stack to another processor, all these pointers must be updated to point to the new copy of the stack instead of the old copy. However, because the stack layout is determined by the vagaries of the machine architecture and compiler, there is no simple and portable method by which all these pointers can even be identified, much less changed.
A feasible approach for thread migration, then, is to guarantee that the stack will have exactly the same address on the new processor as it did on the old processor. Because the stack addresses haven't changed, no pointers need to be updated because all references to the original stack's data remain valid on the new processor. This idea was originally developed for thread migration in the PM2 runtime system [4]. In the following subsections, we present three mechanisms to ensure that the stack's address remains the same after migration. These mechanisms for migration apply to both kernel and user-level threads. However, in this paper, we will mainly focus on the migration of the user-level threads supported by the Charm++ and AMPI runtime systems.