An Overview of ParCFD activities at Delft University of Technology

P. Wildersa,*p.wilders@its.tudelft.nl; B.J. Boersmaa,†; J.J. Derksena,‡; A.W. Heeminka,§; B. Nicenoa,¶; M. Pourquiea,||; C. Vuika,**    a Delft University of Technology, J.M. Burgers Centre, Leeghwaterstraat 21, 2628 CJ Delft, The Netherlands,
* Dept. Applied Physics, Kramers Laboratorium
† Dept. Mechanical Engineering, Section Fluid Mechanics
‡ Dept. Applied Physics, Kramers Laboratorium
§ Dept. Applied Mathematical Analysis, Section Large Scale Systems
¶ Dept. Applied Physics, Section Thermofluids
|| Dept. Mechanical Engineering, Section Fluid Mechanics
** Dept. Applied Mathematical Analysis, Section Numerical Mathematics

At Delft University of Technology much research is done in the area of computational fluid dynamics with underlying models ranging from simple desktop-engineering models to advanced research-oriented models. The advanced models have the tendency to grow beyond the limit of single-processor computing. In the last few years research groups, studying such models, have extended their activities towards parallel computational fluid dynamics on distributed memory machines. We present several examples of this, including fundamental studies in the field of turbulence, LES modelling with industrial background and environmental studies for civil engineering purposes. Of course, a profound mathematical back-up helps to support the more engineering oriented studies and we will also treat some aspects regarding this point.

1 Introduction

We present an overview of research, carried out at Delft University of Technology and involving parallel computational fluid dynamics. The overview will not present all activities carried out in this field at our University. We have chosen to present work of those groups, that are or have been active at the yearly ParCFD conferences, which indicates that these groups focus to some extent on purely parallel issues as well. This strategy for selecting contributing groups enabled the main author to work quite directly without extensive communication overhead and results in an overview presenting approximately 70% of the activities at our University in this field. We apologize on forehand if we have overseen major contributions from other groups.

At Delft University of Technology parallel computational fluid dynamics is an ongoing research activity within several research groups. Typically, this research is set up and hosted within departments. For this purpose they use centrally supported facilities, most often only operational facilities. In rare cases, central support is given as well for developing purposes. Central support is provided by HPαC, http://www.hpac.tudelft.nl/, an institution for high performance computing splitted off from the general computing center in 1996. Their main platform is a Cray T3E with 128 DEC-Alpha processors, installed in 1997 and upgraded in 1999.

From the paralllel point of view most of the work is based upon explicit parallel programming using message passing interfaces. The usage of high-level parallel supporting tools is not very common at our university. Only time accurate codes are studied with time stepping procedures ranging from fully explicit to fully implicit. Typically, the explicit codes show a good parallel performance, are favorite in engineering applications and have been correlated with measurements using fine-grid 3D computations with millions of grid points. The more implicit oriented codes are still in the stage of development, can be classified as research-oriented codes using specialized computational linear algebra for medium size grids and show a reasonable parallel performance.

The physical background of the parallel CFD codes is related to the individual research themes. Traditionally, Delft University of Technology is most active in the incompressible or low-speed compressible flow regions. Typically, Delft University is also active in the field of civil engineering, including environmental questions. The present overview reflects both specialisms.

Of course, studying turbulence is an important issue. Direct numerical simulation (DNS) and large eddy simulation (LES) are used, based upon higher order difference methods or Lattice- Boltzmann methods, both to study fundamental questions as well as applied questions, such as mixing properties or sound generation. Parallel distributed computing enables to resolve the smallest turbulent scales with moderate turn-around times. In particular, the DNS codes are real number crunchers with excessive requirements.

A major task in many CFD codes is to solve large linear systems efficiently on parallel platforms. As an example, we mention the pressure correction equation in a non-Cartesian incompressible code. In Delft, Krylov subspace methods combined with domain decomposition are among the most popular methods for solving large linear systems. Besides applying these methods in our implicit codes, separate mathematical model studies are undertaken as well with the objective to improve robustness, convergence speed and parallel performance.

At the level of civil engineering, contaminant transport forms a source of inspiration. Both atmospheric transport as well as transport in surface and subsurface regions is studied. In the latter case the number of contaminants is in general low and there is a need to increase the geometrical flexibility and spatial resolution of the models. For this purpose parallel transport solvers based upon domain decomposition are studied. In the atmospheric transport models the number of contaminants is high and the grids are regular and of medium size. However, in this case a striking feature is the large uncertainty. One way to deal with this latter aspect is to explore the numerous measurements for improvement of the predictions. For this purpose parallel Kalman filtering techniques are used in combination with parallel transport solvers.

We will present various details encountered in the separate studies and discuss the role of parallel computing, quoting some typical parallel aspects and results. The emphasis will be more on showing where parallel CFD is used for and how this is done than on discussing parallel CFD as a research object on its own.

2 Turbulence

Turbulence research forms a major source of inspiration for parallel computing. Of all activities taking place at Delft University we want to mention two, both in the field of incompressible flow.

A research oriented code has been developed in [12], [13]. Both DNS and LES methods are investigated and compared. The code explores staggered second-order finite differencing on Cartesian grids and the pressure correction method with an explicit Adams-Bathford or Runge- Kutta method for time stepping. The pressure Poisson equation is solved directly using the Fast Fourier transform in two spatial directions, leaving a tridiagonal system in the third spatial direction. The parallel MPI-based implementation relies upon the usual ghost-cell type communication, enabling the computation of fluxes, etc., as well as upon a more global communication operation, supporting the Poisson solver. For a parallel implementation of the Fast Fourier transform it suffices to distribute the frequencies over the processors. However, when doing a transform along a grid line all data associated with this line must be present on the processor. This means that switching to the second spatial direction introduces the necessity of a global exchange of data. Of course, the final tridiagonal system is parallelized by distributing the lines in the associated spatial direction over the processors. Despite the need of global communication, the communication overhead remains in general below 10%. Figure 1 gives an example of the measured wall clock time. The speed-up is nearly linear.

In figure 2 a grid type configuration at inflow generates a number of turbulent jet flows in a channel (modelling wind tunnel turbulence). Due to the intensive interaction and mixing, the distribution of turbulence becomes very quickly homogeneous in the lateral direction. A way to access the numerical results, see figure 3, is to compute the Kolmogorov length scales (involving sensitive derivatives of flow quantities). The grid size is 600 × 48 × 48 (1.5 million points), which is reported to be sufficient to resolve all scales in the mixing region with DNS for R = 1000. For R = 4000 subgrid LES modelling is needed: measured subgrid contributions are of the order of 10 %.

A second example of turbulence modelling can be found in [11]. In this study the goals are directed towards industrial applications with complex geometries using LES. Unstructured co-located second-order finite volumes, slightly stabilized, are used in combination with the pressure correction method and implicit time stepping. Solving the linear systems is done with diagonally preconditioned Krylov methods, i.e. CGS for...