FEM Meshes for Large Machines

Consider a high-resolution fracture dynamics simulation of a block of metal. We first decompose the block of metal into ``elements'', which are small pieces of the domain with a simple shape, often tetrahedra or cubes. In theory, the number of elements is determined only by the physical fidelity we wish to achieve, but in practice the number of elements, and hence the accuracy of the simulation, is often limited by the memory and speed of the machine, as summarized in Table 1.

For example, a 1-meter cube of metal discretized into a 1cm scale mesh will require one million elements. But 1cm is quite coarse; if we need 1mm resolution the mesh will have one billion elements. If elements require 40 bytes each, such a mesh would require 40 GB of storage. This is larger than current serial machines can handle, but is plausible even on today's parallel machines.

Table 1: Meshes for a 1$m^3$ domain, storage requirements and time per timestep on various machines.

Resolution	Elements	Storage	1 Processor	1K Processors	100K Processors	1M Processors
1cm	$10^6$	40MB	1s	1ms	10us	1us
1mm	$10^9$	40GB	1000s	1s	10ms	1ms

One difficulty is that real problems are defined on complicated domains, like machine parts and fracture surfaces, so generating a mesh for the domain is a nontrivial task. Meshes are usually generated by special meshing software in an offline, serial process, so no publicly available meshing software can generate billion-element meshes. A typical solution to this is to first generate a relatively coarse mesh in serial to capture the basic geometry of the domain, then use parallel mesh refinement (or mesh multiplication) to generate more elements where needed. The FEM framework does not handle mesh generation, but it includes rudimentary capabilities for parallel mesh refinement.

Once a mesh is generated, it must be partitioned, and the pieces sent to different processors for parallel execution. The FEM framework currently uses the serial Metis partitioning library, so the partitioning is performed completely on one processor, which becomes a bottleneck for large meshes. We are working on integrating the parallel ParMetis partitioning package to avoid the serial mesh partitioning bottleneck, which should allow us to use larger meshes and scalably partition the mesh. An alternative approach is to use a simpler but inaccurate mesh partitioner such as geometric recursive coordinate bisection, then fix the resulting load imbalance using our load balancing framework.