To migrate a piece of work to a new processor, we must ensure that all needed state information will be available on the new processor. In a fully shared-memory parallel system, this is trivial--because all processors implicitly share all state, migration simply means transferring control to the new processor, which will be able to access the work's state directly.
In a distributed-memory parallel system, we must either proactively ship all needed state along with the piece of work; or else lazily ship state as it is needed. Lazy memory state shipping is commonly done by software distributed shared memory systems. State that is shared between several threads clearly can not simply be shipped to a new location, and so must be either shuffled back and forth at runtime (costing performance) or tightly regulated or banned (costing flexibility). We currently allow shared state only via well-defined interfaces in our own work related to Charm++ and AMPI runtime.
The state required by a migrating control-of-flow invariably falls into one of three categories:
Most migration systems only support shipping a subset of the possible state, forcing users to manually reconstruct other state. In the case of migratable threads, when several threads in a process can share memory, kernel, and communication state, it can be very difficult to extract the set of resources needed by a single thread. In our Charm++/AMPI runtime system, we only allow different threads to share state in a small number of well-defined ways, which makes it possible to efficiently migrate a single thread to a different address space.