s_phys_pkgs/text/exch2.tex

% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.7 2004/02/11 20:48:14 afe Exp $
% $Name:  $

%%  * Introduction
%%    o what it does, citations (refs go into mitgcm_manual.bib, 
%%      preferably in alphabetic order)
%%    o Equations 
%%  * Key subroutines and parameters
%%  * Reference material (auto generated from Protex and structured comments)
%%    o automatically inserted at \section{Reference} 


\section{Extended Cubed Sphere Exchange}
\label{sec:exch2}


\subsection{Introduction}

The \texttt{exch2} package is an extension to the original cubed
sphere exchanges to allow more flexible domain decomposition and
parallelization.  Cube faces (subdomains) may be divided into whatever
number of tiles that divide evenly into the grid point dimensions of
the subdomain.  Furthermore, the individual tiles may be run on
separate processors in different combinations, and whether exchanges
between particular tiles occur between different processors is
determined at runtime.

The exchange parameters are declared in
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
and assigned in
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}, both in
the \texttt{pkg/exch2} directory.  The validity of the cube topology
depends on the \texttt{SIZE.h} file as detailed below.  Both files are
generated by Matlab scripts and should not be edited.  The default
files provided in the release set up a cube sphere arrangement of six
tiles, one per subdomain, each with 32x32 grid points, running on a
single processor.

\subsection{Key Variables}

The descriptions of the variables are divided up into scalars,
one-dimensional arrays indexed to the tile number, and two and three
dimensional arrays indexed to tile number and neighboring tile.  This
division actually reflects the functionality of these variables: the
scalars are common to every part of the topology, the tile-indexed
arrays to individual tiles, and the arrays indexed to tile and
neighbor to relationships between tiles and their neighbors.

\subsubsection{Scalars}

The number of tiles in a particular topology is set with the parameter
\texttt{NTILES}, and the maximum number of neighbors of any tiles by
\texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
size of the various one and two dimensional arrays that store tile
parameters indexed to the tile number.

The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
of tiles in the x and y global indices.  For example, the default
setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
\texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
gridpoints) tiles, four (2x2) per subdomain, will have
\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
Note that these parameters express the tile layout to allow global
data files that are tile-layout-neutral and have no bearing on the
internal storage of the arrays.  The tiles are internally stored in a
range from \texttt{1,bi} (in the x axis) and y-axis variable
\texttt{bj} is generally ignored within the package.

\subsubsection{Arrays Indexed to Tile Number}

The following arrays are of size \texttt{NTILES}, are indexed to the
tile number, and the indices are omitted in their descriptions.

The arrays \varlink{exch2\_tnx}{exch2_tnx} and
\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
releases of MITgcm are to allow varying tile sizes.

The location of the tiles' Cartesian origin within a subdomain are
determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
relate the location of the edges of the tiles to each other.  As an
example, in the default six-tile topology (the degenerate case) each
index in these arrays are set to 0.  The twenty-four, 32x32 cube face
case discussed above will have values of 0 or 16, depending on the
quadrant the tile falls within the subdomain.  The array 
\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
cubeface/subdomain of each tile, numbered 1-6 in the case of the
standard cube topology.

The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
\varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
global address space, similar to that used by global files.

The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
\varlink{exch2\_isEedge}{exch2_isEedge},
\varlink{exch2\_isSedge}{exch2_isSedge}, and
\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
tile lies on the edge of a subdomain, 0 if not.  The values are used
within the topology generator to determine the orientation of
neighboring tiles and to indicate whether a tile lies on the corner of
a subdomain.  The latter case indicates special exchange and numerical
handling for the singularities at the eight corners of the cube.
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
how many neighboring tiles each tile has, and is used for setting
bounds for looping over neighboring tiles.
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
tile, and is used in interprocess communication.

\subsubsection{Arrays Indexed to Tile Number and Neighbor}

The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
\texttt{NTILES} and describe the orientations between the the tiles.

The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
each of the $n$ neighboring tiles.  The neighbor tiles are indexed
\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
then south edges, and then top to bottom on the east and west edges.
Maybe throw in a fig here, eh?

The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
In other words, 
\begin{verbatim}
   exch2_neighbourId( exch2_opposingSend_record(a,T),
                      exch2_neighbourId(a,T) ) = T
\end{verbatim}
and this provides a back-reference from the neighbor tiles.

The arrays \varlink{exch2\_pi}{exch2_pi},
\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
exchanges between the neighboring tiles.  The dimensions of
\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
neighbor ID \textit{N} and the tile number \textit{T} as explained
above, plus the transformation vector {\em t }, of length two.  The
first element of the transformation vector indicates the factor by
which variables representing the same vector component of a tile will
be multiplied, and the second element indicates the transform to the
variable in the other direction.  As an example,
\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
vector variable in tile \texttt{T} to the i-component of tile
\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
the component of neighbor \texttt{N}'s j-component.

Under the current cube topology, one of the two elements of
\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
and neighbor \texttt{N} will be 0, reflecting the fact that the vector
components are orthogonal.  The other element will be 1 or -1,
depending on whether the components are indexed in the same or
opposite directions.  For example, the transform dimension of the
arrays for all tile neighbors on the same subdomain will be [1,0],
since all tiles on the same subdomain are oriented identically.
Vectors that correspond to the orthogonal dimension with the same
index direction will have [0,1], whereas those in the opposite index
direction will have [0,-1].


{\footnotesize
\begin{verbatim}
C      exch2_pi          :: X index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_pj          :: Y index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_oi          :: X index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oj          :: Y index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oi_f        :: X index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
C      exch2_oj_f        :: Y index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
\end{verbatim}
}


\subsection{Key Routines}


\subsection{References}
1	edhill	1.8	% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.7 2004/02/11 20:48:14 afe Exp $
2	afe	1.1	% $Name: $
3
4			%% * Introduction
5			%% o what it does, citations (refs go into mitgcm_manual.bib,
6			%% preferably in alphabetic order)
7			%% o Equations
8			%% * Key subroutines and parameters
9			%% * Reference material (auto generated from Protex and structured comments)
10			%% o automatically inserted at \section{Reference}
11
12
13	edhill	1.8	\section{Extended Cubed Sphere Exchange}
14	afe	1.3	\label{sec:exch2}
15
16	afe	1.1
17			\subsection{Introduction}
18	afe	1.2
19	edhill	1.8	The \texttt{exch2} package is an extension to the original cubed
20			sphere exchanges to allow more flexible domain decomposition and
21			parallelization. Cube faces (subdomains) may be divided into whatever
22			number of tiles that divide evenly into the grid point dimensions of
23			the subdomain. Furthermore, the individual tiles may be run on
24			separate processors in different combinations, and whether exchanges
25			between particular tiles occur between different processors is
26			determined at runtime.
27
28			The exchange parameters are declared in
29			\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
30			and assigned in
31			\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}, both in
32			the \texttt{pkg/exch2} directory. The validity of the cube topology
33			depends on the \texttt{SIZE.h} file as detailed below. Both files are
34			generated by Matlab scripts and should not be edited. The default
35			files provided in the release set up a cube sphere arrangement of six
36			tiles, one per subdomain, each with 32x32 grid points, running on a
37			single processor.
38	afe	1.4
39			\subsection{Key Variables}
40
41			The descriptions of the variables are divided up into scalars,
42	edhill	1.8	one-dimensional arrays indexed to the tile number, and two and three
43			dimensional arrays indexed to tile number and neighboring tile. This
44			division actually reflects the functionality of these variables: the
45			scalars are common to every part of the topology, the tile-indexed
46			arrays to individual tiles, and the arrays indexed to tile and
47			neighbor to relationships between tiles and their neighbors.
48	afe	1.4
49			\subsubsection{Scalars}
50
51			The number of tiles in a particular topology is set with the parameter
52	edhill	1.8	\texttt{NTILES}, and the maximum number of neighbors of any tiles by
53			\texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the
54			size of the various one and two dimensional arrays that store tile
55			parameters indexed to the tile number.
56
57			The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
58			and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
59			of tiles in the x and y global indices. For example, the default
60			setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
61			\texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in
62			gridpoints) tiles, four (2x2) per subdomain, will have
63			\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
64			Note that these parameters express the tile layout to allow global
65			data files that are tile-layout-neutral and have no bearing on the
66			internal storage of the arrays. The tiles are internally stored in a
67			range from \texttt{1,bi} (in the x axis) and y-axis variable
68			\texttt{bj} is generally ignored within the package.
69	afe	1.4
70	afe	1.6	\subsubsection{Arrays Indexed to Tile Number}
71	afe	1.4
72	edhill	1.8	The following arrays are of size \texttt{NTILES}, are indexed to the
73			tile number, and the indices are omitted in their descriptions.
74	afe	1.4
75	edhill	1.8	The arrays \varlink{exch2\_tnx}{exch2_tnx} and
76			\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
77			tile. At present for each tile \texttt{exch2\_tnx=sNx} and
78			\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future
79			releases of MITgcm are to allow varying tile sizes.
80
81			The location of the tiles' Cartesian origin within a subdomain are
82			determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
83			\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to
84			relate the location of the edges of the tiles to each other. As an
85			example, in the default six-tile topology (the degenerate case) each
86			index in these arrays are set to 0. The twenty-four, 32x32 cube face
87			case discussed above will have values of 0 or 16, depending on the
88			quadrant the tile falls within the subdomain. The array
89			\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
90			cubeface/subdomain of each tile, numbered 1-6 in the case of the
91			standard cube topology.
92
93			The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
94			\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
95			\varlink{exch2\_tbasex}{exch2_tbasex} and
96			\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
97			global address space, similar to that used by global files.
98
99			The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
100			\varlink{exch2\_isEedge}{exch2_isEedge},
101			\varlink{exch2\_isSedge}{exch2_isSedge}, and
102			\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
103			tile lies on the edge of a subdomain, 0 if not. The values are used
104			within the topology generator to determine the orientation of
105			neighboring tiles and to indicate whether a tile lies on the corner of
106			a subdomain. The latter case indicates special exchange and numerical
107			handling for the singularities at the eight corners of the cube.
108			\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
109			how many neighboring tiles each tile has, and is used for setting
110			bounds for looping over neighboring tiles.
111			\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
112			tile, and is used in interprocess communication.
113	afe	1.4
114	afe	1.6	\subsubsection{Arrays Indexed to Tile Number and Neighbor}
115	afe	1.4
116	edhill	1.8	The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
117			\texttt{NTILES} and describe the orientations between the the tiles.
118	afe	1.5
119	edhill	1.8	The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
120			each of the $n$ neighboring tiles. The neighbor tiles are indexed
121			\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
122			then south edges, and then top to bottom on the east and west edges.
123			Maybe throw in a fig here, eh?
124
125			The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
126			in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
127			In other words,
128			\begin{verbatim}
129			exch2_neighbourId( exch2_opposingSend_record(a,T),
130			exch2_neighbourId(a,T) ) = T
131	afe	1.5	\end{verbatim}
132	edhill	1.8	and this provides a back-reference from the neighbor tiles.
133	afe	1.5
134	edhill	1.8	The arrays \varlink{exch2\_pi}{exch2_pi},
135			\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
136			\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
137			\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
138			exchanges between the neighboring tiles. The dimensions of
139			\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
140			neighbor ID \textit{N} and the tile number \textit{T} as explained
141			above, plus the transformation vector {\em t }, of length two. The
142			first element of the transformation vector indicates the factor by
143			which variables representing the same vector component of a tile will
144			be multiplied, and the second element indicates the transform to the
145			variable in the other direction. As an example,
146			\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
147			vector variable in tile \texttt{T} to the i-component of tile
148			\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
149			the component of neighbor \texttt{N}'s j-component.
150
151			Under the current cube topology, one of the two elements of
152			\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
153			and neighbor \texttt{N} will be 0, reflecting the fact that the vector
154			components are orthogonal. The other element will be 1 or -1,
155			depending on whether the components are indexed in the same or
156			opposite directions. For example, the transform dimension of the
157			arrays for all tile neighbors on the same subdomain will be [1,0],
158			since all tiles on the same subdomain are oriented identically.
159			Vectors that correspond to the orthogonal dimension with the same
160			index direction will have [0,1], whereas those in the opposite index
161			direction will have [0,-1].
162	afe	1.5
163	afe	1.4
164	edhill	1.8	{\footnotesize
165	afe	1.4	\begin{verbatim}
166			C exch2_pi :: X index row of target to source permutation
167			C :: matrix for each neighbour entry.
168			C exch2_pj :: Y index row of target to source permutation
169			C :: matrix for each neighbour entry.
170			C exch2_oi :: X index element of target to source
171			C :: offset vector for cell-centered quantities
172			C :: of each neighbor entry.
173			C exch2_oj :: Y index element of target to source
174			C :: offset vector for cell-centered quantities
175			C :: of each neighbor entry.
176			C exch2_oi_f :: X index element of target to source
177			C :: offset vector for face quantities
178			C :: of each neighbor entry.
179			C exch2_oj_f :: Y index element of target to source
180			C :: offset vector for face quantities
181			C :: of each neighbor entry.
182			\end{verbatim}
183	edhill	1.8	}
184	afe	1.1
185
186
187			\subsection{Key Routines}
188
189
190
191			\subsection{References}