MADRE: The Memory-Aware Data Redistribution Engine
On this page... (hide)
- 1. Overview
- 2. Background
- 3. Download and installation
- 4. Documents
- 5. License
1. Overview
MADRE is an MPI-based library for carrying out data redistribution in a distributed, limited-memory environment. It is written in C (with MPI) and provides a simple, documented, easy-to-use interface for accessing a number of different redistribution algorithms. The library is free and open-source, and is distributed under the GNU Lesser General Public License. The authors of MADRE are Stephen F. Siegel and Andrew R. Siegel.
2. Background
A large class of massively parallel scientific applications are memory-bound. To achieve their scientific aims, considerable care must be taken to reduce their memory footprint via careful programming techniques, e.g. avoiding unnecessary data copies of main data structures, not storing global meta-data locally, and so forth. The need for efficient many-to-many parallel data movement arises in a wide variety of such scientific applications. Adaptive algorithms, for example, require extensive load balancing to ensure an optimal distribution of mesh elements across processors (both in terms of equal balance and spatial locality). Time-dependent particle-tracking codes are another good example, with frequent rebalancing of the main data structures as particles cross processor boundaries.
Typically, such load balancing operations consist of two parts: 1) computing the map that specifies the new destination for each block of data and 2) efficiently carrying out the movement of data blocks to their new location without exceeding the memory available to the application on any process. There has been much research into the first problem, resulting, for example, in numerous efficient algorithms for computing space-filling curves. The second problem, though, has received little attention, especially in the critical case where memory resources are severely limited and memory use must be completely transparent to the application developer.
The purpose of the MADRE library is to provide a number of solutions to the second problem through an easy-to-use, clearly-documented common interface. Once the application has computed an appropriate map, it calls a function in the MADRE library to carry out the redistribution. The MADRE library is also designed to be transparent in its use of memory (and other resources).
3. Download and installation
The current version of MADRE is 0.3. Installation instructions can be found in the MADRE manual, which is also included in the distribution.
- The MADRE Manual only
- madre-0.3.tgz: tar, gzipped archive of complete MADRE source and manual
4. Documents
- MADRE: The Memory-Aware Data Redistribution Engine, a paper presented at EuroPVM/MPI 2008 describing the ideas behind MADRE and two of the redistribution algorithms
5. License
MADRE is copyright 2008, Stephen F. Siegel and Andrew R. Siegel.
MADRE is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
MADRE is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU Lesser General Public License for more details. VSL
