Parallel Dynamic Unstructured Mesh Methods with Application to Lagrangian Simulation of Flows with Deformable Boundaries Omar Ghattas and Ivan Malcevic* Mechanics, Algorithms and Computing Laboratory Department of Civil and Environmental Engineering Carnegie Mellon University Pittsburgh, PA 15213 oghattas@cs.cmu.edu malcevic@andrew.cmu.edu Abstract We present a new methodology for parallel mesh regeneration for large-scale simulations of problems with dynamic boundaries and deforming domanis In contrast to the widespread practice of making only incremental changes to the existing mesh, we take a radical approach to dynamic mesh motion by aggressively remeshing in parallel at every time step The key to our approach is to retain as many of the nodes as possible but replace, in parallel, the element structure with a completely new Delaunay triangulation Refinement and coarsening of the nodal set are also employed to ensure that nodes are well-spaced according to solution and dynamic geometry-driven criteria Our current twodimensional implementation has been applied within the framework of a parallel Lagrangian finite element mesh-based incompressible flow solver We illustrate the capability of the Lagrangian simulations through a series of model problems involving moving and deformable boundaries, and provide parallel performance results showing high scalability of our code Introduction The goal of this work is to develop parallel scalable dynamic mesh methods for problems that require widespread and frequent mesh changes The most prominent such problems are those that involve dynamic boundaries Examples include fluidstructure interaction, fluid-fluid interaction, phase changes, free surfaces, and shape optimization This work is a part of the Terascale Algorithms for Optimization of Simulations (TAOS) project at CMU with support from NASA grant NAG-1-2090 and NSF grant ECS 9732301 (under the NSF/Sandia Life Cycle Engineering Program) Computing services on the Pittsburgh Supercomputing Center’s T3E-900 were provided under PSC grant ASC-990003P * Contact author The techniques we develop are also appropriate for problems with fixed boundaries, but for which the mesh must adapt to strongly time-dependent phenomena These include wave, shock, and crack propagation, vortex motion, and shear localization Problems with dynamic boundaries are typically addressed through Lagrangian or Arbitrary Lagrangian Eulerian (ALE) formulations Lagrangian methods are attractive because boundary motion is automatically enforced in the formulation motion is expressed at material, rather than spatial, points However, the motion of material points implies dynamic mesh movement For problems in which material points undergo large relative motion, the mesh becomes hopelessly entangled after a few time steps, and some kind of remeshing is needed (for example, recognized in [1] and [2]) Mesh-based Lagrangian methods are particularly troublesome on highly parallel computers, since parallel dynamic meshing is regarded as a difficult problem As a result, the majority of large-motion Lagrangian simulations are particle-based, typically using some kind of integral kernel representation ALE methods attempt to minimize mesh deformations by mediating between Eulerian and Lagrangian descriptions of material motion Typically, a Lagrangian description is used on moving boundaries (so that the mesh is moving with the boundary), an Eulerian description on stationary boundaries (the mesh is fixed), and some kind of smoothing procedure is used to interpolate mesh motion in between Most ALE techniques retain the mesh topology as the nodes are relocated at each time step, and this works reasonably well for small mesh motion This is advantageous on parallel machines, since most procedures for interpolating the interior mesh motion are based on solution of spring, elasticity, or Laplacian systems, for which good parallel algorithms are known However, this approach breaks down with moderate mesh motion, since keeping a fixed-topology mesh leads to distorted and inverted elements A completely new mesh is thus needed While there has been significant work on parallel adaptive meshing, these techniques are most efficient only for small, incremental changes to existing meshes at each time step Complete remeshing at frequent intervals is strongly avoided, particularly on parallel computers One successful ALE moving domain simulation alternates between parallel and sequential components [3] In the parallel phase, the field variables are updated through a CFD time step computation and the mesh is deformed according to the motion of the boundaries But after a certain time, the mesh becomes severely distorted, so it is transferred to a shared memory computer and regenerated sequentially The remeshing phase is followed by a projection (done in parallel) of the new mesh onto the old one, to interpolate new values of field variables from the old ones The sequential remeshing is clearly a bottleneck, but fast scalable parallel mesh generation remains a challenge In this work we advocate a radical approach to dynamic mesh motion by aggressively remeshing in parallel at every time step We make use of recent advances in parallel computational geometry, in particular of a recently developed parallel Delaunay triangulation algorithm, [4] Together with existing techniques for parallel mesh adaptation and with the use of mesh motion provided by the numerical portion of the simulation algorithm, these methods form a solid basis for efficient frequent parallel mesh regeneration Parallel Dynamic Finite Element Meshes Experience from state-of-the-art mesh-based simulations of multiple rigid balls suspended in a viscous incompressible fluid [3], [5] shows that the mesh undergoes unrecoverable distortions after a certain number of time steps, and thus complete remeshing is unavoidable This number gets progressively smaller with increasing number of balls, and thus relative motion between balls Inclusion of deformable boundaries would add to the level of complexity and would make the task of maintaining a quality mesh even more difficult Using local mesh refinement to maintain a quality mesh is not sufficient for this class of problems, and would postpone the need for complete remeshing for just a few time steps, while producing significant load imbalances on the processors Large mesh distortions occur in zones with high relative motion, spread quickly over time and, if untreated, cover significant portions of the domain A global approach addressing both nodal distribution and elemental structure is required and should be applied frequently, preferably at every time step However, generating a completely new mesh in parallel at every time step is neither practical nor desirable Remeshing procedures typically consist of finding new nodal locations and their triangulation The most time consuming component is obtaining the new nodal set In contrast, retaining the existing nodal set for dynamic mesh problems offers significant advantages First, one avoids the costly step of determining appropriate nodal locations Second, existing nodes carry the computed solution field, so keeping them would eliminate the need for projecting the old solution onto the new nodal set Third, in a Lagrangian nodal framework, nodal positions reflect the underlying physics Nodes have a tendency to concentrate in regions where high resolution is required, and therefore retaining the existing nodes provides a measure of adaptivity On the other hand, the problem with retaining the existing nodal set for the next time step is that its distribution distorts over time and may not satisfy all geometrical and physical spacing criteria Particular problems arise with inadequate resolution of boundary layers and appearance of overcrowded regions However, if nodal set is kept well spaced frequently by local refinement and coarsening, these problems not have time to develop Once the nodal set is adapted, the mesh regeneration problem reduces to one of the triangulation, which is conceptually simpler and much faster Recently, parallel triangulation and repartitioning have been the subject of intensive research The first general-purpose scalable two-dimensional parallel algorithms have begun to appear In this work we use the recently developed and implemented parallel Delaunay triangulation algorithm by Blelloch, Miller, Talmor and Hardwick [4] It uses a divide-and-conquer technique in which all the work of partitioning is done on the divide step and is based on convex hull computation Given an arbitrary set of vertices, distributed across the processors, the algorithm returns a Delaunay triangulation of the points and at the same time partitions the vertices such that they are load-balanced and communication across the processor boundaries is small The algorithm has been shown to perform well for highly nonuniform meshes, which is of great importance to us High scalability has been achieved for large numbers of processors and for meshes with the number of nodes on the order of millions Reported times to triangulate meshes of about one million nodes on 64 processors on Cray T3D are on the order of seconds This is small compared to the time usually required to solve a system of discretized PDEs of the corresponding size, and makes retriangulation of the nodes at every time step feasible To summarize, as shown in Figure 1, the mesh regeneration procedure starts with the transport of the nodes to their new locations according to computed motion (velocity field in Lagrangian and solution of Laplacian, elasticity or similar equations in ALE methods) Then, in local coarsening and refinement phases, Figure 1c and 1d, existing nodes are removed and new nodes are inserted to satisfy the local feature size function, which is based on geometric and solution related indicators Refinement and coarsening phases are purely local and highly efficient Since adaptation of the nodal set is done at every time step, only slight changes to the existing nodal structure are required Our results show that fewer than 1% of the total amount of nodes is added/removed per one mesh regeneration cycle Since most of the nodes acquire velocities from the previous time step computation, projection of the solution to the new nodal set is avoided, and instead we obtain the field values for newly added nodes by interpolation from the parent element In addition, interpolation errors have local character and not diffuse over the domain as they would if a new nodal distribution is used After the removal and insertion phases are completed, the adapted nodal set is guaranteed to be well spaced with respect to the current geometry and flow solution Finally, a completely new parallel Delaunay triangulation is obtained (Figure 1e) with repartitioning done at the same time (Figure 1f) Since the Delaunay property of the mesh is maintained, most of the elements existing in the old configuration reappear in the new structure as well Edge swapping occurs and new edges appear only in places of large relative mesh motion and where nodes have been removed or inserted a) FE mesh at the beginning of a time step b) FE mesh at the end of a time Step (nodes are moved according to the velocity field) c) Nodal structure at the end of a time step Coarsening stage: nodes marked with circles are removed d) Refinement stage Nodes are added to maintain mesh quality or resolution features e) New FE mesh after Delaunay triangulation at the beginning of new time step f) Partitions of the new FE mesh Figure 1: Typical time step parallel dynamic remeshing procedure Simulation of viscous flow around the stationary cylinder (Re=5000) Only portion of the computational domain is shown Application to Parallel Lagrangian Flow Simulation The dynamic mesh strategy described in the previous section has been implemented within the framework of a parallel Lagrangian finite element-based Navier-Stokes solver Lagrangian mesh/grid-based methods date back to the early work of Hirt, Cook and Butler [6] Since then, numerical aspects of these methods have been studied in a number of articles including [7], [8], [9], [1], [2] The consensus has been that Lagrangian methods offer substantial advantages in modeling flows involving moving boundaries, since boundary representation is embedded in the material description of the flow The absence of the nonlinear convective term has been exploited to develop simple numerical schemes often expressed as variations of fractional step methods Yet, despite the introduction of the Lagrangian approach more than three decades ago, application has been limited to the simulation of the simplest of flows The reason for this is that most methods keep the same mesh topology over time leading to mesh deterioration These obstacles led to the development of ALE methods with the primary goal of reducing mesh distortion difficulties The importance of addressing the distorted meshes in Langrangian methods was recognized in [1] and recently in [2] However, mesh distortions were cured with complete remeshing from scratch, which limited the application to small size problems By using parallel retriangulation techniques, we show in this paper that remeshing can be done at every time step and that very large problems can be solved The numerical formulation used in this work has been derived from the Lagrangian form of the Navier-Stokes equations The time step computation consists of a sequence of non-linear iterations in which the initial guessed velocity field is updated until convergence is achieved More details on derivation and the numerical scheme can be found in [10] After the solution for the next time instant is obtained, the mesh-related portion of the code takes control of the simulation, moves the nodal set according to the computed velocity field and proceeds with mesh regeneration as described in the previous section At the end of the time step, a new mesh is obtained and the data structure is in place for starting the next time step computation We have implemented the above procedure in a parallel code built on top of the PETSc parallel numerical library [11], and have applied it to a series of model problems To enable the use of equal order finite-elements (P1P1 element), pressure stabilization from [12] has been used in modified form to allow for symmetric linear systems A conjugate residual symmetric linear solver with additive Schwarz domain decomposition preconditioner was used to compute the velocity and pressure fields Due to the limited space, we report results from just a small subset of model problems with the focus on dynamic mesh regenertion A more detailed list of examples, associated animations and parallel performance results can be found in [10] Our first set of model problems is the simulation of a viscous flow around the stationary cylinder with Reynolds number ranging from the to 10000 Flow parameters - lift and drag coefficients and Strouhol number - match those reported elsewhere in the literature In Figure 2, four snapshots taken at different instants in time show mesh and particle evolution As the flow passes the cylinder from left to right, a characteristic wake with associated von-Karman vortex street develops Note how the mesh nodes/fluid particles resemble the vortex structure Local refinement is most intense near the cylinder boundary where the velocity gradients are highest and where large incoming triangles need to be refined to resolve the boundary layer Results from model problems involving moving and deformable boundaries are shown in Figures and Figure consists of four snapshots from a simulation of two rigid cylinders, one stationary (on the right) and one moving due to the flow The moving cylinder approaches the stationary one until it finally impacts it and then slides to one side to continue its flow out of the domain In Figure 4, four snapshots from the simulation of four heavy viscous liquid drops in a surrounding uniform flow are shown Figure 2: Finite element mesh and particle evolvement over time for simulation of viscous flow around stationary cylinder (Re=5000) Figure 3: Moving boundary problem: Two cylinders example Figure 4: Interaction of liquid cells in surrounding flow While the two moving boundary problems presented are very different from each other and much more complex than the flow around a stationary object, our Lagrangian code was the same for all three examples The only addition to the numerical part of the code was the rigid body dynamics equations for modeling the motion of the rigid cylinders The only change in the mesh-related part of the code was in the local feature size function, which has been modified to sense the approaching contact between two bodies so that appropriate refinement can take place In figure we report the parallel performance results for the simulation of flow around stationary cylinder The conclusion is that the proposed methodology is highly scalable (parallel efficiency of 87% when going from to 64 processors for the overall simulation) In addition, the time breakdown of a typical time step reveals that the cost of mesh regeneration is at least an order of magnitude smaller than the numerical component Size(DOF) No of PEs 8000 32000 16 128000 64 Re 10 100 1000 10 100 1000 10 100 1000 Solver 7.5 16.7 28.6 16.5 37.7 73.1 25.5 42.5 115.4 Mesh 0.26 0.23 0.24 0.63 0.67 0.63 1.97 1.85 1.93 100 Parallel Scalability of Lagrangian Mesh-Based Solver Case: 128115 DOF; Flow around cylinder, Re=100 309.0sec 159.3sec 83.8sec 44.4sec 80 (%) 60 40 20 16 32 Number of processors 64 Figure 5: Parallel performance of Lagrangian flow simulation Concluding Remarks and Acknowledgements A new methodology for parallel mesh regeneration for the simulation of dynamic problems involving moving boundaries has been presented The method has been successfully implemented for parallel Lagrangian flow simulations We have shown that if properly addressed, the cost associated with dynamic remeshing at each time step can be kept small relative to the numerical components of the simulation Our current implementation is two-dimensional but algorithms naturally extend to three-dimensions as well The authors would like to acknowledge the following people for their advice during the course of this research George Biros (Carnegie Mellon University) provided advice on the PETSc numerical library and numerical methods for largescale systems Noel Walkington (CMU) helped in the analysis of Lagrangian methods Guy Blelloch (CMU) and Jonathan Hardwick (Microsoft) provided us with an MPI implementation of their parallel Delaunay triangulation algorithm The help of the PETSc group – Satish Balay, William Gropp, Lois CurfmannMcInnes and Barry Smith (Argonne National Laboratory), is also appreciated References [1] Muttin F., Coupez T., Bellet M., Chenot J., “Lagrangian Finite-Element Analysis of Time-Dependent Viscous Free-Surface Flow Using an Automatic Remeshing Technique: Application to Metal Casting Flow”, International Journal for Numerical Methods in Engineering, Vol 36, pp:2001-2015, 1993 [2] Radovitzky R., Ortiz M., “Lagrangian Finite Element Analysis of Newtonian Fluid Flows”, International Journal for Numerical Methods in Engineering, Vol 43 pp:607-619, 1998 [3] Johnson A., Tezduyar T., “3D Simulation of Fluid-Particle Interactions with the Number of Particles Reaching 100”, Computer Methods in Applied Mechanics and Engineering, Vol 145, pp:301-321, 1997 [4] Blelloch G E., Miller G L., Talmor D., Hardwick J C., “Design and Implementation of a Practical Projection-Based Parallel Delaunay Algorithm”, Algorithmica, Vol 24, pp:243-269, 1999 [5] Joseph D D., “Direct Simulation of the Motion of Particles in Flowing Liquids”, NSF KDI/New Computational Challenge, http://www.aem.umn.edu/Solid-Liquid_Flows, 2000 [6] Hirt C W., Cook J L., Butler T D., “A Lagrangian Method for Calculation of the Dynamics of an Incompressible Fluid with a Free Surface”, Journal of Computational Physics, Vol 5, pp:103-124, 1970 [7] Bach P., Hassager O., “A Lagrangian Finite Element Method for the Simulation of Flow of Newtonian Liquids”, AiCHe Journal, Vol 30, pp:507509, 1984 [8] Hayashi M., Hatanaka K., Kawahara M., “Lagrangian Finite Element Method for Free Surface Navier-Stokes Flow Using Fractional Step Methods”, International Journal for Numerical Methods in Fluids, Vol 13, pp 805-840, 1991 [9] Okamoto T., Kawahara M., “Two-Dimensional Sloshing Analysis by Lagrangian Finite Element Method”, International Journal for Numerical Methods in Fluids, Vol 11, pp:453-477, 1990 [10] Ghattas O., Malcevic I., TAOS Project – Lagrangian Finite Element MeshBased Methods for Incompressible Fluid Flows, http://www.cs.cmu.edu/~malcevic, 2000 [11] Balay S., Gropp W D., Curfmann-McInnes L C., Smith B F., PETSc home page, http://mcs.anl.gov/petsc.html, 2000 [12] Tezduyar T E., “Stabilized Finite Element Formulations for Incompressible Flow Computations”, Advances in Applied Mechanics, Vol 28, pp:1-44, 1991 ... simplest of flows The reason for this is that most methods keep the same mesh topology over time leading to mesh deterioration These obstacles led to the development of ALE methods with the primary... framework of a parallel Lagrangian finite element-based Navier-Stokes solver Lagrangian mesh/ grid-based methods date back to the early work of Hirt, Cook and Butler [6] Since then, numerical aspects of. .. meshes, which is of great importance to us High scalability has been achieved for large numbers of processors and for meshes with the number of nodes on the order of millions Reported times to