MP ZEUS: testing and performance analysis of the message passing ZEUS fluid simulation package

CISC 879 and PHYS 838
Parallelization for Scientific Applications
20 Sep 2000

Group 3
Advisor	Hans-Reinhard Müller
Group Members	Asif ud-Doula Samuel Kalet Joe Pellegrino

ZEUS has become a standard tool in the astrophysical community for a wide range of fluid dynamics simulations. Here 'fluid' might be a misnomer since actually what is meant is ionized gas or plasma that resides in the interstellar medium. But because they behave just like idealized fluids the name remains. It has been developed over many years by the Laboratory for Computational Astrophysics (LCA) of National Center for Supercomputing Applications. ZEUS has been used to simulate supernova explosions, study solar wind, or even star formation, and for many other astrophysical problems.

Many of these problems require fine grids which requires significant computing time. But as the problems become more complex, for example if magnetic field or self-gravity is included, the simulation times increase many folds. Also, astrophysicists very often wish to follow the evolution of their problem over long period of physical time which usually corresponds to a large number of iterations. To ease this problem LCA released parallel version of the sequential ZEUS code called ZEUS-MP: Multi-Physics, Massively-Parallel, Message-Passing code.

We wish to examine ZEUS-MP for its performance and correctness. In order to develop confidence in any numerical code, one has to do some tests the result of which we know analytically, and make sure that the code gives reasonable results. In addition, we want to be certain that it is worthwhile to use ZEUS-MP as opposed to using sequential ZEUS. So, it is important for us to analyze ZEUS-MP for its performance across different platforms.

We are looking to thoroughly evaluate the ZEUS-MP package that is publically available. This evaluation would include installation through portability and scalability. The following list identifies each component that we include in our evaluation:

• Correctness: By running a short test problem and comparing the results to those of the serial ZEUS we can determine that ZEUS-MP produces correct solutions. There are sample programs bundled with ZEUS-MP that may serve as appropriate test problems, in addition to the easiest problem in our test suite detailed in the proceeding section.
• Analysis of program structure: We will analyze the structure of the ZEUS-MP program and the techniques that it uses to achieve parallelization. We will clarify some details of the domain decompostition that the application uses to distribute the problem's data and also the ways that it uses MPI.
• Performance comparison to serial ZEUS: By using our test suite of problems we can identify what type of performance gains ZEUS-MP offers over the known computation times of serial ZEUS for these same problems.
• Scalability: By varying the number of processors used for parallel computation we can gain an idea of how effectively ZEUS-MP scales.
• Portability (across platform): We would like to attempt to run ZEUS-MP under Linux or IRIX to see how easily it can be ported.
• Portability (across MPI implementation): Following with running ZEUS-MP under Linux or IRIX, we might be using a different MPI implementation which could lead to difficulty.
• Evaluate Linux Beowulf clustering for ZEUS-MP: These clusters would allow us to run one problem concurrently on more than 8 physical processors to test scalability. Information we may gather regarding configuration and performance of ZEUS-MP on a Linux Beowulf cluster could be an important contribution due to the rapid growth in popularity of these clusters for these types of massively-parallel problems.

Our plan of investigation is to analyze the performance of the ZEUS-MP package. This will involve running each of our test suite of problems through ZEUS-MP individually and interpreting the performance with standard methods as well as any other methods that we feel will provide meaningful results relating the the performance of ZEUS-MP. We plan to use the following three problems to evaluate ZEUS-MP:

1. Isothermal solar wind (easy): The CORONA of the sun is very hot in relation to its surroundings, in fact it can reach several millions degrees Kelvin. Due to this large temperature and density stratification, there is a significant pressure gradient present near the surface of the sun and this pressure gradient causes an outflow of matter from the sun into its surroundings known as the solar wind, predicted by Eugene Parker in 1958, and first discovered in 1960. This is a purely hydrodynamics (HD) simulation represented in spherical coordinates in one dimension and runs should take no more than a few minutes assuming about 400 grid points equally spaced. We expect that this problem will take longer running in ZEUS-MP because its serial runtimes are very small.
   
   
2. Regenerating spiral structure of solar magnetic field at the equatorial plane following Weber and Davis (more difficult): The sun rotates continually and the magnetic fields that extend from the sun into its surroundings "bend" as the sun rotates. This problem studies how these magnetic fields are altered by the sun's rotation. This problem is more difficult because it is a magnetohydrodynamics (MHD) problem. The introduction of magnetic fields to a hydrodynamics (HD) problem makes the problem harder to solve and requires significantly more computation. This simulation is expected to take an order of magnitude longer than the first one - approximately 10 minutes. This problem should see some performance gains with ZEUS-MP over the serial version.
   
   
3. Structure of the global heliosphere (realistically difficult): This simulation involves a parallel stream of matter (the interstellar medium, a "fluid" consisting mostly of neutral hydrogen, protons, and electrons) flowing unobstructed until encountering the Sun and its immediate neighborhood (the heliosphere). Numerically, the Sun is a point in the simulation grid that produces matter flowing out radially in all directions (the solar wind, a "fluid" consisting mostly of protons and electrons). The simulation determines the effect that this source has on the flow of the interstellar stream, and the structure of the resulting stationary state of the global heliosphere. This 2-D problem uses polar coordinates, and its numerical scale (435 X 41 grid points, with typically millions of iterations) is very representative of the magnitude of problems that ZEUS is typically being used to solve. It is expected to clearly identify whether there are gains to using ZEUS-MP over the serial version which takes about 19 hours CPU on a 600 MHz alpha.
   
   
   Sample output of a previous, serial Zeus simulation of the global heliosphere. The Sun is at the origin (0,0), the interstellar material enters at the right hand side. The structure of the global heliosphere is identifiable through sharp rises and drops in the plasma temperature and the neutral hydrogen density. The flow field of the plasma is indicated with red streamlines. Distances are given in Astronomical Units (1 AU = Earth-Sun distance).