01_introduction.tex

\chapter{Introduction}

The software package SPECFEM3D\_GLOBE simulates three-dimensional global and regional seismic wave propagation
and performs full waveform imaging (FWI) or adjoint tomography based upon the spectral-element method (SEM).
The SEM is a continuous Galerkin technique \citep{TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11},
which can easily be made discontinuous \citep{BeMaPa94,Ch00,KoWoHu02,ChCaVi03,LaWaBe05,Kop06,WiStBuGh10,AcKo11};
it is then close to a particular case of the discontinuous Galerkin technique \citep{ReHi73,LeRa74,Arn82,JoPi86,BoMaHe91,FaRi99,HuHuRa99,CoKaSh00,GiHeWa02,RiWh03,MoRi05,GrScSc06,AiMoMu06,BeLaPi06,DuKa06,DeSeWh08,PuAmKa09,WiStBuGh10,DeSe10,EtChViGl10}, with optimized efficiency because of its tensorized basis functions \citep{WiStBuGh10,AcKo11}.
In particular, it can accurately handle very distorted mesh elements \citep{OlSe11}.\\

It has very good accuracy and convergence properties \citep{MaPa89,SePr94,DeFiMu02,Coh02,DeSe07,SeOl08,AiWa09,AiWa10,MeStTh12}.
The spectral element approach admits spectral rates of convergence and allows exploiting $hp$-convergence schemes.
It is also very well suited to parallel implementation on very large supercomputers \citep{KoTsChTr03,TsKoChTr03,KoLaMi08a,CaKoLaTiMiLeSnTr08,KoViCh10} as well as on clusters of GPU accelerating graphics cards \citep{Kom11,MiKo10,KoMiEr09,KoErGoMi10}. Tensor products inside each element can be optimized to reach very high efficiency \citep{DeFiMu02}, and mesh point and element numbering can be optimized to reduce processor cache misses and improve cache reuse \citep{KoLaMi08a}. The SEM can also handle triangular (in 2D) or tetrahedral (in 3D) elements \citep{WinBoyd96,TaWi00,KoMaTrTaWi01,Coh02,MeViSa06} as well as mixed meshes, although with increased cost and reduced accuracy in these elements, as in the discontinuous Galerkin method.\\

Note that in many geological models in the context of seismic wave propagation studies
(except for instance for fault dynamic rupture studies, in which very high frequencies or supershear rupture need to be modeled near the fault, see e.g. \cite{BeGlCrViPi07,BeGlCrVi09,PuAmKa09,TaCrEtViBeSa10})
a continuous formulation is sufficient because material property contrasts are not drastic and thus
conforming mesh doubling bricks can efficiently handle mesh size variations \citep{KoTr02a,KoLiTrSuStSh04,LeChLiKoHuTr08,LeChKoHuTr09,LeKoHuTr09}.
This is particularly true at the scale of the full Earth.\\

For a detailed introduction to the SEM as applied to
global and regional seismic wave propagation, please consult \citet{TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11,KoVi98,KoTr99,Ch00,KoTr02a,KoTr02b,KoRiTr02,ChCaVi03,CaChViMo03,ChVa04,ChKoViCaVaFe07}.
A detailed theoretical analysis of the dispersion
and stability properties of the SEM is available in \citet{Coh02}, \citet{DeSe07}, \citet{SeOl07}, \citet{SeOl08} and \citet{MeStTh12}.\\

Effects due to lateral variations in compressional-wave
speed, shear-wave speed, density, a 3D crustal model, ellipticity,
topography and bathymetry, the oceans, rotation, and self-gravitation are included.
The package can accommodate full 21-parameter anisotropy \citep{ChTr07}
as well as lateral variations in attenuation \citep{SaKoTr10}. Adjoint
capabilities and finite-frequency kernel simulations are also included
\citep{TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11,LiTr06,LiTr08,FiIgBuKe09,ViOp09}.\\

The SEM was originally developed in computational fluid dynamics \citep{Pat84,MaPa89}
and has been successfully adapted to address problems in seismic wave propagation.
Early seismic wave propagation applications of the SEM, utilizing Legendre basis functions and a
perfectly diagonal mass matrix, include \cite{CoJoTo93}, \cite{Kom97},
\cite{FaMaPaQu97}, \cite{CaGa97}, \cite{KoVi98} and \cite{KoTr99},
whereas applications involving Chebyshev basis functions and a nondiagonal mass matrix
include \cite{SePr94}, \cite{PrCaSe94} and \cite{SePrPr95}.\\

All SPECFEM3D\_GLOBE software is written in Fortran2003 with full portability
in mind, and conforms strictly to the Fortran2003 standard. It uses
no obsolete or obsolescent features of Fortran. The package uses
parallel programming based upon the Message Passing Interface (MPI)
\citep{GrLuSk94,Pac97}.\\

SPECFEM3D\_GLOBE won the Gordon Bell award for best performance at the SuperComputing~2003
conference in Phoenix, Arizona (USA) (see \cite{KoTsChTr03}).
It was a finalist again in 2008 for a run at 0.16 petaflops (sustained) on 149,784 processors of the `Jaguar' Cray XT5 system at Oak Ridge National Laboratories (USA) \citep{CaKoLaTiMiLeSnTr08}.
It also won the BULL Joseph Fourier supercomputing award in 2010.\\

It reached the sustained one petaflop performance level for the first time in February 2013
on the Blue Waters Cray supercomputer at the National Center for Supercomputing Applications (NCSA), located at the University of Illinois at Urbana-Champaign (USA).\\

The package includes support for GPU graphics card acceleration \citep{Kom11,MiKo10,KoMiEr09,KoErGoMi10} and also supports OpenCL.\\

The next release of the code will include support for
Convolutional or Auxiliary Differential Equation Perfectly Matched absorbing Layers (C-PML or ADE-PML)
\citep{MaKoEz08,MaKoGe08,MaKo09,MaKoGeBr10,KoMa07} for the case of single-chunk simulations in regional models.

\section{Citation}

If you use SPECFEM3D\_GLOBE for your own research, please cite at least one
of the following articles: \cite{TrKoLi08,PeKoLuMaLeCaLeMaLiBlNiBaTr11,VaCaSaKoVi99,LeChLiKoHuTr08,LeChKoHuTr09,LeKoHuTr09,KoMiEr09,KoErGoMi10,WiKoScTr04,KoLiTrSuStSh04,ChKoViCaVaFe07,MaKoDi09,KoViCh10,CaKoLaTiMiLeSnTr08,TrKoHjLiZhPeBoMcFrTrHu10,KoRiTr02,KoTr02a,KoTr02b,KoTr99} or \cite{KoVi98}.\\

If you use this new version, which has non blocking MPI for much better performance for medium or large runs,
please cite at least one of these five articles,
in which results of 3D non blocking MPI runs are presented: \cite{KoErGoMi10,KoViCh10,Kom11,PeKoLuMaLeCaLeMaLiBlNiBaTr11,CaKoLaTiMiLeSnTr08}.\\

If you use GPU graphics card acceleration please cite e.g. \cite{Kom11}, \cite{MiKo10}, \cite{KoMiEr09}, and/or \cite{KoErGoMi10}.\\

If you work on geophysical applications, you may be interested in citing some of these application articles as well, among others:
\cite{WiKoScTr04,JiTsKoTr05,KrJiKoTr06a,KrJiKoTr06b,LeChLiKoHuTr08,LeChKoHuTr09,LeKoHuTr09,ChFaKo04,FaChKo04,RiRiKoTrHe02,GoAmTaCaSmSaMaKo09,TrKo00,SaKoTr10}.
If you use 3D mantle model S20RTS, please cite \citet{RiVaWo99}.\\

Domain decomposition is explained in detail in \cite{MaKoBlLe08}, and excellent scaling up to 150,000 processor cores in shown for instance in
\cite{CaKoLaTiMiLeSnTr08,KoLaMi08a,MaKoBlLe08,KoErGoMi10,Kom11}.\\

The corresponding Bib\TeX{} entries may be found
in file \texttt{doc/USER\_MANUAL/bibliography.bib}.

\section{Support}

This material is based upon work supported by the U.S. National Science
Foundation under Grants No. EAR-0406751 and EAR-0711177, by the French
CNRS, French INRIA Sud-Ouest MAGIQUE-3D, French ANR NUMASIS under
Grant No. ANR-05-CIGC-002, and European FP6 Marie Curie International
Reintegration Grant No. MIRG-CT-2005-017461.
Any opinions, findings, and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reflect the views of the
U.S. National Science Foundation, CNRS, INRIA, ANR or the European
Marie Curie program.