The TeraGyroid Project Uses Transatlantic Supercomputing

By Roland Piquepaille

Before going further, what is the TeraGyroid project? It couples "cutting-edge grid technologies, high-performance computing, visualisation and computational steering capabiltities to produce a major leap forward in soft condensed matter simulation."

Let's look at a technical description of the project -- skip the paragraph if you like. Its name is "Grid-based Lattice-Boltzmann simulations of Defect Dynamics in Amphiphilic Liquid Crystals."

An amphiphile is a chemical species whose molecules consist of a hydrophobic tail attached to a hydrophilic head. This dichotomy causes the molecules to self-assemble into complex morphologies when dispersed in solvents, binary immiscible fluid mixtures or melts. Some mesophases are liquid crystalline, with features intermediate between a liquid and a solid. The gyroid is one of these: ubiquitous in biological systems, it has important applications in membrane protein crystallisation, controlled drug release and biosensors. Being a mesophase, the gyroid exhibits weak crystallinity and the presence of defects, which play an important role in determining its mechanical properties. The objective of TeraGyroid is to study defect pathways and dynamics in gyroid self-assembly via the largest set of lattice-Boltzmann(LB) simulations ever performed, involving lattices of over one billion sites.

Here is an illustration showing the "pyroid phase."

The TeraGyroid experiment during SC2003 was jointly funded by the UK's Engineering and Physical Sciences Research Council (EPSRC) and the National Science Foundation, USA (NSF). TeraGyroid is based on the UK e-Science pilot project RealityGrid.

This news release from EPSRC, "EPSRC achieves a world first in high performance computing" will give you more information about the experiment.

The TeraGyroid experiment has access to a substantial fraction of the world's largest supercomputing resources, including the whole of the UK's supercomputing facilities and the USA's supercomputers based in Illinois, Pittsburgh and San Diego. Trans-Atlantic optical bandwidth is supported by British Telecommunications. The largest simulations are in excess of one billion lattice sites. These larger simulations can only be accommodated on the HPCx, NCSA (National Center for Supercomputing Applications, Illinois) and PSC (Pittsburgh Supercomputing Center) platforms.

This interactive grid computing and visualisation received an HPC Challenge Award during SC 2003. Here is the abstract of the presentation, called "Transcontinental RealityGrids for Interactive Collaborative Exploration of Parameter Space (TRICEPS)."

The slides for TeraGyroid's presentation at SC2003 are in Powerpoint format ( 15 slides, 4,098 KB). The slide 11 gives the details of the supercomputing facilities used, which totalled more than 6,000 processors, including the following resources.

• HPCx (Daresbury), 1280 procs IBM Power4 Regatta, 6.6 Tflops peak, 1.024 TB
• Lemieux (PSC), 3000 procs HP/Compaq, 3TB memory, 6 Tflops peak
• TeraGrid Itanium2 cluster (NCSA), 256 procs, 1.3 Tflops peak
• TeraGrid Itanium2 cluster (SDSC), 256 procs, 1.3 Tflops peak
• Green (CSAR), SGI Origin 3800, 512 procs, 0.512 TB memory (shared)
• Newton (CSAR), SGI Altix 3700, 256 Itanium 2 procs, 384GB memory (shared)

The experiment also needed huge visualization systems, such as an SGI Onyx 300 with 6xIR3 graphical pipes and 32 processors in Manchester.

The transatlantic bandwidth was provided by British Telecommunications, using a 10 gigabits/second link between the Netherlands and Chicago.

What can we learn from this experiment, which was certainly hard to set up?

The success of this experiment demonstrates that this linking of multiple high performance computers can allow scientists to expand their knowledge of small systems containing a few molecules to larger, macroscopic, real-world situations containing hundreds of thousands of billions of molecules.

Source: Roland Piquepaille, with various websites