s_phys_pkgs/text/exch2.tex

% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.10 2004/03/15 20:11:56 afe Exp $
% $Name:  $

%%  * Introduction
%%    o what it does, citations (refs go into mitgcm_manual.bib, 
%%      preferably in alphabetic order)
%%    o Equations 
%%  * Key subroutines and parameters
%%  * Reference material (auto generated from Protex and structured comments)
%%    o automatically inserted at \section{Reference} 


\section{exch2: Extended Cubed Sphere \mbox{Topology}}
\label{sec:exch2}


\subsection{Introduction}

The \texttt{exch2} package is an extension to the original cubed
sphere topological configuration that allows more flexible domain
decomposition and parallelization.  Cube faces (also called
subdomains) may be divided into any number of tiles that divide evenly
into the grid point dimensions of the subdomain.  Furthermore, the
individual tiles may be run on separate processors in different
combinations, and whether exchanges between particular tiles occur
between different processors is determined at runtime.  This
flexibility provides for manual compile-time load balancing across a
relatively arbitrary number of processors. \\

The exchange parameters are declared in
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
and assigned in
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
validity of the cube topology depends on the \file{SIZE.h} file as
detailed below.  Both files are generated by Matlab scripts in
\file{utils/exch2/matlab-topology-generator}; see Section
\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
for details on creating alternate topologies.  The default files
provided in the release configure a cubed sphere topology of six
tiles, one per subdomain, each with 32$\times$32 grid points, all
running on a single processor.  Pregenerated examples of these files
with alternate topologies are provided under
\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
file for single-processor execution.

\subsection{Invoking exch2}

To use exch2 with the cubed sphere, the following conditions must be
met: \\

$\bullet$ The exch2 package is included when \file{genmake2} is run.
  The easiest way to do this is to add the line \code{exch2} to the
  \file{profile.conf} file -- see Section
  \ref{sect:buildingCode}\sectiontitle{Building the code} for general
  details. \\

$\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
  \file{w2\_e2setup.F} must reside in a directory containing code
  linked when \file{genmake2} runs.  The safest place to put these
  is the directory indicated in the \code{-mods=DIR} command line
  modifier (typically \file{../code}), or the build directory.  The
  default versions of these files reside in \file{pkg/exch2} and are
  linked automatically if no other versions exist elsewhere in the
  link path, but they should be left untouched to avoid breaking
  configurations other than the one you intend to modify.\\

$\bullet$ Files containing grid parameters, named
  \file{tile???.mitgrid} where \file{???} is \file{001} through
  \file{006} (one per subdomain), must be in the working directory
  when the MITgcm executable is run.  These files are provided in the
  example experiments for cubed sphere configurations with
  32$\times$32 cube sides and are non-trivial to generate -- please
  contact MITgcm support if you want to generate files for other
  configurations. \\

$\bullet$ As always when compiling MITgcm, the file \file{SIZE.h}
  must be placed where \file{genmake2} will find it.  In particular
  for the exch2, the domain decomposition specified in \file{SIZE.h}
  must correspond with the particular configuration's topology
  specified in \file{W2\_EXCH2\_TOPOLOGY.h} and
  \file{w2\_e2setup.F}.  Domain decomposition issues particular to
  exch2 are addressed in Section \ref{sec:topogen} \sectiontitle{Generating
  Topology Files for exch2}; a more general background on the subject
  relvant to MITgcm is presented in Section
  \ref{sect:specifying_a_decomposition}\sectiontitle{Specifying a
  decomposition}.\\

As of the time of writing the following examples use exch2 and may be
used for guidance:

\begin{verbatim}
verification/adjust_nlfs.cs-32x32x1
verification/adjustment.cs-32x32x1 
verification/aim.5l_cs
verification/global_ocean.cs32x15
verification/hs94.cs-32x32x5
\end{verbatim}


\subsection{Generating Topology Files for exch2}
\label{sec:topogen}

Alternate cubed sphere topologies may be created using the Matlab
scripts in \file{utils/exch2/matlab-topology-generator}. Running the
m-file \file{driver.m} from the Matlab prompt (there are no parameters
to pass) generates exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h}
and \file{w2\_e2setup.F} in the working directory and displays a
figure of the topology via Matlab.  The other m-files in the directory
are subroutines of \file{driver.m} and should not be run except for
development purposes. \\

The parameters that determine the dimensions and topology of the
generated configuration are \code{nr}, \code{nb}, \code{ng},
\code{tnx} and \code{tny}, and all are assigned early in the script.

The first three determine the size of the subdomains (cube faces) and
hence the size of the overall domain.  Each one determines the number
of grid points, and therefore the resolution, along the subdomain
sides in a ``great circle'' around each axis of the cube.  At the time
of this writing MITgcm requires these three parameters to be equal,
but they provide for future releases of MITgcm to accomodate different
resolutions around the axes to allow (for example) greater resolution
around the equator.\\

The parameters \code{tnx} and \code{tny} determine the dimensions of
the tiles into which the subdomains are decomposed, and must evenly
divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
The result is a rectangular tiling of the subdomain.  Figure
\ref{fig:24tile} shows one possible topology for a twenty-four tile
cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\

\begin{figure}
\begin{center}
 \resizebox{4in}{!}{
  \includegraphics{part6/s24t_16x16.ps}
 }
\end{center} 
\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
twenty-four tiles (\code{tnx=16, tny=16})
} \label{fig:24tile}
\end{figure}

\begin{figure}
\begin{center}
 \resizebox{4in}{!}{
  \includegraphics{part6/s12t_16x32.ps}
 }
\end{center} 
\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
twelve tiles (\code{tnx=16, tny=32})
} \label{fig:12tile}
\end{figure}

Tiles can be selected from the topology to be omitted from being
allocated memory and processors.  This kind otuning is useful in
ocean modeling for omitting tiles that fall entirely on land.  The
tiles omitted are specified in the file \file{blanklist.txt} by
their tile number in the topology, separated by a newline. \\


\subsection{Key Variables}

The descriptions of the variables are divided up into scalars,
one-dimensional arrays indexed to the tile number, and two and three
dimensional arrays indexed to tile number and neighboring tile.  This
division actually reflects the functionality of these variables: the
scalars are common to every part of the topology, the tile-indexed
arrays to individual tiles, and the arrays indexed to tile and
neighbor to relationships between tiles and their neighbors.

\subsubsection{Scalars}

The number of tiles in a particular topology is set with the parameter
\texttt{NTILES}, and the maximum number of neighbors of any tiles by
\texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
size of the various one and two dimensional arrays that store tile
parameters indexed to the tile number.\\

The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
of tiles in the x and y global indices.  For example, the default
setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
\texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
gridpoints) tiles, four (2x2) per subdomain, will have
\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
Note that these parameters express the tile layout to allow global
data files that are tile-layout-neutral and have no bearing on the
internal storage of the arrays.  The tiles are internally stored in a
range from \texttt{1,bi} (in the x axis) and y-axis variable
\texttt{bj} is generally ignored within the package.

\subsubsection{Arrays Indexed to Tile Number}

The following arrays are of size \texttt{NTILES}, are indexed to the
tile number, and the indices are omitted in their descriptions.

The arrays \varlink{exch2\_tnx}{exch2_tnx} and
\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
releases of MITgcm are to allow varying tile sizes.

The location of the tiles' Cartesian origin within a subdomain are
determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
relate the location of the edges of the tiles to each other.  As an
example, in the default six-tile topology (the degenerate case) each
index in these arrays are set to 0.  The twenty-four, 32x32 cube face
case discussed above will have values of 0 or 16, depending on the
quadrant the tile falls within the subdomain.  The array 
\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
cubeface/subdomain of each tile, numbered 1-6 in the case of the
standard cube topology.

The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
\varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
global address space, similar to that used by global files.

The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
\varlink{exch2\_isEedge}{exch2_isEedge},
\varlink{exch2\_isSedge}{exch2_isSedge}, and
\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
tile lies on the edge of a subdomain, 0 if not.  The values are used
within the topology generator to determine the orientation of
neighboring tiles and to indicate whether a tile lies on the corner of
a subdomain.  The latter case indicates special exchange and numerical
handling for the singularities at the eight corners of the cube.
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
how many neighboring tiles each tile has, and is used for setting
bounds for looping over neighboring tiles.
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
tile, and is used in interprocess communication.

\subsubsection{Arrays Indexed to Tile Number and Neighbor}

The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
\texttt{NTILES} and describe the orientations between the the tiles.

The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
each of the $n$ neighboring tiles.  The neighbor tiles are indexed
\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
then south edges, and then top to bottom on the east and west edges.
Maybe throw in a fig here, eh?

The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
In other words, 
\begin{verbatim}
   exch2_neighbourId( exch2_opposingSend_record(a,T),
                      exch2_neighbourId(a,T) ) = T
\end{verbatim}
and this provides a back-reference from the neighbor tiles.

The arrays \varlink{exch2\_pi}{exch2_pi},
\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
exchanges between the neighboring tiles.  The dimensions of
\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
neighbor ID \textit{N} and the tile number \textit{T} as explained
above, plus the transformation vector {\em t }, of length two.  The
first element of the transformation vector indicates the factor by
which variables representing the same vector component of a tile will
be multiplied, and the second element indicates the transform to the
variable in the other direction.  As an example,
\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
vector variable in tile \texttt{T} to the i-component of tile
\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
the component of neighbor \texttt{N}'s j-component.

Under the current cube topology, one of the two elements of
\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
and neighbor \texttt{N} will be 0, reflecting the fact that the vector
components are orthogonal.  The other element will be 1 or -1,
depending on whether the components are indexed in the same or
opposite directions.  For example, the transform dimension of the
arrays for all tile neighbors on the same subdomain will be [1,0],
since all tiles on the same subdomain are oriented identically.
Vectors that correspond to the orthogonal dimension with the same
index direction will have [0,1], whereas those in the opposite index
direction will have [0,-1].


{\footnotesize
\begin{verbatim}
C      exch2_pi          :: X index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_pj          :: Y index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_oi          :: X index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oj          :: Y index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oi_f        :: X index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
C      exch2_oj_f        :: Y index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
\end{verbatim}
}


\subsection{Key Routines}


\subsection{References}
1	afe	1.11	% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.10 2004/03/15 20:11:56 afe Exp $
2	afe	1.1	% $Name: $
3
4			%% * Introduction
5			%% o what it does, citations (refs go into mitgcm_manual.bib,
6			%% preferably in alphabetic order)
7			%% o Equations
8			%% * Key subroutines and parameters
9			%% * Reference material (auto generated from Protex and structured comments)
10			%% o automatically inserted at \section{Reference}
11
12
13	afe	1.10	\section{exch2: Extended Cubed Sphere \mbox{Topology}}
14	afe	1.3	\label{sec:exch2}
15
16	afe	1.1
17			\subsection{Introduction}
18	afe	1.2
19	edhill	1.8	The \texttt{exch2} package is an extension to the original cubed
20	afe	1.9	sphere topological configuration that allows more flexible domain
21			decomposition and parallelization. Cube faces (also called
22			subdomains) may be divided into any number of tiles that divide evenly
23			into the grid point dimensions of the subdomain. Furthermore, the
24			individual tiles may be run on separate processors in different
25			combinations, and whether exchanges between particular tiles occur
26			between different processors is determined at runtime. This
27	afe	1.10	flexibility provides for manual compile-time load balancing across a
28			relatively arbitrary number of processors. \\
29	edhill	1.8
30			The exchange parameters are declared in
31			\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32			and assigned in
33	afe	1.9	\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34	afe	1.11	validity of the cube topology depends on the \file{SIZE.h} file as
35	afe	1.10	detailed below. Both files are generated by Matlab scripts in
36	afe	1.11	\file{utils/exch2/matlab-topology-generator}; see Section
37			\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
38			for details on creating alternate topologies. The default files
39			provided in the release configure a cubed sphere topology of six
40			tiles, one per subdomain, each with 32$\times$32 grid points, all
41			running on a single processor. Pregenerated examples of these files
42			with alternate topologies are provided under
43			\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
44			file for single-processor execution.
45	afe	1.9
46			\subsection{Invoking exch2}
47
48	afe	1.10	To use exch2 with the cubed sphere, the following conditions must be
49			met: \\
50	afe	1.9
51	afe	1.11	$\bullet$ The exch2 package is included when \file{genmake2} is run.
52			The easiest way to do this is to add the line \code{exch2} to the
53			\file{profile.conf} file -- see Section
54			\ref{sect:buildingCode}\sectiontitle{Building the code} for general
55			details. \\
56
57			$\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
58			\file{w2\_e2setup.F} must reside in a directory containing code
59			linked when \file{genmake2} runs. The safest place to put these
60			is the directory indicated in the \code{-mods=DIR} command line
61			modifier (typically \file{../code}), or the build directory. The
62			default versions of these files reside in \file{pkg/exch2} and are
63	afe	1.10	linked automatically if no other versions exist elsewhere in the
64			link path, but they should be left untouched to avoid breaking
65			configurations other than the one you intend to modify.\\
66
67			$\bullet$ Files containing grid parameters, named
68	afe	1.11	\file{tile???.mitgrid} where \file{???} is \file{001} through
69			\file{006} (one per subdomain), must be in the working directory
70	afe	1.10	when the MITgcm executable is run. These files are provided in the
71			example experiments for cubed sphere configurations with
72			32$\times$32 cube sides and are non-trivial to generate -- please
73			contact MITgcm support if you want to generate files for other
74			configurations. \\
75
76	afe	1.11	$\bullet$ As always when compiling MITgcm, the file \file{SIZE.h}
77			must be placed where \file{genmake2} will find it. In particular
78			for the exch2, the domain decomposition specified in \file{SIZE.h}
79	afe	1.10	must correspond with the particular configuration's topology
80	afe	1.11	specified in \file{W2\_EXCH2\_TOPOLOGY.h} and
81			\file{w2\_e2setup.F}. Domain decomposition issues particular to
82			exch2 are addressed in Section \ref{sec:topogen} \sectiontitle{Generating
83			Topology Files for exch2}; a more general background on the subject
84	afe	1.10	relvant to MITgcm is presented in Section
85	afe	1.11	\ref{sect:specifying_a_decomposition}\sectiontitle{Specifying a
86			decomposition}.\\
87	afe	1.9
88			As of the time of writing the following examples use exch2 and may be
89			used for guidance:
90
91			\begin{verbatim}
92			verification/adjust_nlfs.cs-32x32x1
93			verification/adjustment.cs-32x32x1
94			verification/aim.5l_cs
95			verification/global_ocean.cs32x15
96			verification/hs94.cs-32x32x5
97			\end{verbatim}
98
99
100
101
102	afe	1.10	\subsection{Generating Topology Files for exch2}
103			\label{sec:topogen}
104
105			Alternate cubed sphere topologies may be created using the Matlab
106	afe	1.11	scripts in \file{utils/exch2/matlab-topology-generator}. Running the
107			m-file \file{driver.m} from the Matlab prompt (there are no parameters
108			to pass) generates exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h}
109			and \file{w2\_e2setup.F} in the working directory and displays a
110			figure of the topology via Matlab. The other m-files in the directory
111			are subroutines of \file{driver.m} and should not be run except for
112			development purposes. \\
113	afe	1.10
114			The parameters that determine the dimensions and topology of the
115	afe	1.11	generated configuration are \code{nr}, \code{nb}, \code{ng},
116			\code{tnx} and \code{tny}, and all are assigned early in the script.
117	afe	1.10
118			The first three determine the size of the subdomains (cube faces) and
119			hence the size of the overall domain. Each one determines the number
120			of grid points, and therefore the resolution, along the subdomain
121			sides in a ``great circle'' around each axis of the cube. At the time
122			of this writing MITgcm requires these three parameters to be equal,
123			but they provide for future releases of MITgcm to accomodate different
124			resolutions around the axes to allow (for example) greater resolution
125			around the equator.\\
126
127	afe	1.11	The parameters \code{tnx} and \code{tny} determine the dimensions of
128			the tiles into which the subdomains are decomposed, and must evenly
129			divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
130			The result is a rectangular tiling of the subdomain. Figure
131			\ref{fig:24tile} shows one possible topology for a twenty-four tile
132			cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
133	afe	1.10
134			\begin{figure}
135			\begin{center}
136			\resizebox{4in}{!}{
137			\includegraphics{part6/s24t_16x16.ps}
138			}
139			\end{center}
140			\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
141	afe	1.11	twenty-four tiles (\code{tnx=16, tny=16})
142	afe	1.10	} \label{fig:24tile}
143			\end{figure}
144
145			\begin{figure}
146			\begin{center}
147			\resizebox{4in}{!}{
148			\includegraphics{part6/s12t_16x32.ps}
149			}
150			\end{center}
151			\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
152	afe	1.11	twelve tiles (\code{tnx=16, tny=32})
153	afe	1.10	} \label{fig:12tile}
154			\end{figure}
155
156			Tiles can be selected from the topology to be omitted from being
157	afe	1.11	allocated memory and processors. This kind otuning is useful in
158	afe	1.10	ocean modeling for omitting tiles that fall entirely on land. The
159	afe	1.11	tiles omitted are specified in the file \file{blanklist.txt} by
160	afe	1.10	their tile number in the topology, separated by a newline. \\
161
162
163
164
165
166	afe	1.4
167			\subsection{Key Variables}
168
169			The descriptions of the variables are divided up into scalars,
170	edhill	1.8	one-dimensional arrays indexed to the tile number, and two and three
171			dimensional arrays indexed to tile number and neighboring tile. This
172			division actually reflects the functionality of these variables: the
173			scalars are common to every part of the topology, the tile-indexed
174			arrays to individual tiles, and the arrays indexed to tile and
175			neighbor to relationships between tiles and their neighbors.
176	afe	1.4
177			\subsubsection{Scalars}
178
179			The number of tiles in a particular topology is set with the parameter
180	edhill	1.8	\texttt{NTILES}, and the maximum number of neighbors of any tiles by
181			\texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the
182			size of the various one and two dimensional arrays that store tile
183	afe	1.9	parameters indexed to the tile number.\\
184	edhill	1.8
185			The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
186			and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
187			of tiles in the x and y global indices. For example, the default
188			setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
189			\texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in
190			gridpoints) tiles, four (2x2) per subdomain, will have
191			\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
192			Note that these parameters express the tile layout to allow global
193			data files that are tile-layout-neutral and have no bearing on the
194			internal storage of the arrays. The tiles are internally stored in a
195			range from \texttt{1,bi} (in the x axis) and y-axis variable
196			\texttt{bj} is generally ignored within the package.
197	afe	1.4
198	afe	1.6	\subsubsection{Arrays Indexed to Tile Number}
199	afe	1.4
200	edhill	1.8	The following arrays are of size \texttt{NTILES}, are indexed to the
201			tile number, and the indices are omitted in their descriptions.
202	afe	1.4
203	edhill	1.8	The arrays \varlink{exch2\_tnx}{exch2_tnx} and
204			\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
205			tile. At present for each tile \texttt{exch2\_tnx=sNx} and
206			\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future
207			releases of MITgcm are to allow varying tile sizes.
208
209			The location of the tiles' Cartesian origin within a subdomain are
210			determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
211			\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to
212			relate the location of the edges of the tiles to each other. As an
213			example, in the default six-tile topology (the degenerate case) each
214			index in these arrays are set to 0. The twenty-four, 32x32 cube face
215			case discussed above will have values of 0 or 16, depending on the
216			quadrant the tile falls within the subdomain. The array
217			\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
218			cubeface/subdomain of each tile, numbered 1-6 in the case of the
219			standard cube topology.
220
221			The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
222			\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
223			\varlink{exch2\_tbasex}{exch2_tbasex} and
224			\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
225			global address space, similar to that used by global files.
226
227			The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
228			\varlink{exch2\_isEedge}{exch2_isEedge},
229			\varlink{exch2\_isSedge}{exch2_isSedge}, and
230			\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
231			tile lies on the edge of a subdomain, 0 if not. The values are used
232			within the topology generator to determine the orientation of
233			neighboring tiles and to indicate whether a tile lies on the corner of
234			a subdomain. The latter case indicates special exchange and numerical
235			handling for the singularities at the eight corners of the cube.
236			\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
237			how many neighboring tiles each tile has, and is used for setting
238			bounds for looping over neighboring tiles.
239			\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
240			tile, and is used in interprocess communication.
241	afe	1.4
242	afe	1.6	\subsubsection{Arrays Indexed to Tile Number and Neighbor}
243	afe	1.4
244	edhill	1.8	The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
245			\texttt{NTILES} and describe the orientations between the the tiles.
246	afe	1.5
247	edhill	1.8	The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
248			each of the $n$ neighboring tiles. The neighbor tiles are indexed
249			\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
250			then south edges, and then top to bottom on the east and west edges.
251			Maybe throw in a fig here, eh?
252
253			The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
254			in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
255			In other words,
256			\begin{verbatim}
257			exch2_neighbourId( exch2_opposingSend_record(a,T),
258			exch2_neighbourId(a,T) ) = T
259	afe	1.5	\end{verbatim}
260	edhill	1.8	and this provides a back-reference from the neighbor tiles.
261	afe	1.5
262	edhill	1.8	The arrays \varlink{exch2\_pi}{exch2_pi},
263			\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
264			\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
265			\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
266			exchanges between the neighboring tiles. The dimensions of
267			\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
268			neighbor ID \textit{N} and the tile number \textit{T} as explained
269			above, plus the transformation vector {\em t }, of length two. The
270			first element of the transformation vector indicates the factor by
271			which variables representing the same vector component of a tile will
272			be multiplied, and the second element indicates the transform to the
273			variable in the other direction. As an example,
274			\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
275			vector variable in tile \texttt{T} to the i-component of tile
276			\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
277			the component of neighbor \texttt{N}'s j-component.
278
279			Under the current cube topology, one of the two elements of
280			\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
281			and neighbor \texttt{N} will be 0, reflecting the fact that the vector
282			components are orthogonal. The other element will be 1 or -1,
283			depending on whether the components are indexed in the same or
284			opposite directions. For example, the transform dimension of the
285			arrays for all tile neighbors on the same subdomain will be [1,0],
286			since all tiles on the same subdomain are oriented identically.
287			Vectors that correspond to the orthogonal dimension with the same
288			index direction will have [0,1], whereas those in the opposite index
289			direction will have [0,-1].
290	afe	1.5
291	afe	1.4
292	edhill	1.8	{\footnotesize
293	afe	1.4	\begin{verbatim}
294			C exch2_pi :: X index row of target to source permutation
295			C :: matrix for each neighbour entry.
296			C exch2_pj :: Y index row of target to source permutation
297			C :: matrix for each neighbour entry.
298			C exch2_oi :: X index element of target to source
299			C :: offset vector for cell-centered quantities
300			C :: of each neighbor entry.
301			C exch2_oj :: Y index element of target to source
302			C :: offset vector for cell-centered quantities
303			C :: of each neighbor entry.
304			C exch2_oi_f :: X index element of target to source
305			C :: offset vector for face quantities
306			C :: of each neighbor entry.
307			C exch2_oj_f :: Y index element of target to source
308			C :: offset vector for face quantities
309			C :: of each neighbor entry.
310			\end{verbatim}
311	edhill	1.8	}
312	afe	1.1
313
314
315			\subsection{Key Routines}
316
317
318
319			\subsection{References}