s_phys_pkgs/text/exch2.tex

% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.3 2004/01/29 15:39:49 afe Exp $
% $Name:  $

%%  * Introduction
%%    o what it does, citations (refs go into mitgcm_manual.bib, 
%%      preferably in alphabetic order)
%%    o Equations 
%%  * Key subroutines and parameters
%%  * Reference material (auto generated from Protex and structured comments)
%%    o automatically inserted at \section{Reference} 


\section{exch2: Extended Cubed Sphere Exchange}
\label{sec:exch2}


\subsection{Introduction}

The exch2 package is an extension to the original cubed sphere exchanges
to allow more flexible domain decomposition and parallelization.  Cube faces
(subdomains) may be divided into whatever number of tiles that divide evenly
into the grid point dimensions of the subdomain.  Furthermore, the individual
tiles may be run on separate processors in different combinations,
and whether exchanges between particular tiles occur between different
processors is determined at runtime.

The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and 
assigned in {\em w2\_e2setup.F}, both in the 
{\em pkg/exch2} directory.  The validity of the cube topology depends
on the {\em SIZE.h} file as detailed below.  Both files are generated by 
Matlab scripts and
should not be edited.  The default files provided in the release set up
a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid
points, running on a single processor.  

\subsection{Key Variables}

The descriptions of the variables are divided up into scalars,
one-dimensional arrays indexed to the tile number, and two-dimensional
arrays indexed to tile number and neighboring tile.  This division
actually reflects  the functionality of these variables, not just the
whim of some FORTRAN enthusiast.

\subsubsection{Scalars}

The number of tiles in a particular topology is set with the parameter
{\em NTILES}, and the maximum number of neighbors of any tiles by 
{\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of
the various one and two dimensional arrays that store tile parameters
indexed to the tile number.

The scalar parameters {\em exch2\_domain\_nxt} and 
{\em exch2\_domain\_nyt} express the number of tiles in the x and y global
indices.  For example, the default setup of six tiles has 
{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of
twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will
have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note 
that these parameters express the tile layout to allow global data files that
are tile-layout-neutral and have no bearing on the internal storage of the
arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the
x axis) and y-axis variable {\em bj} is generally ignored within the package.

\subsubsection{One-Dimensional Arrays}

The following arrays are indexed to the tile number, and the indices are
omitted in their descriptions.

The arrays {\em exch2\_tnx} and {\em exch2\_tny} 
express the x and y dimensions of each tile.  At present for each tile
{\em exch2\_tnx = sNx} 
and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of 
MITgcm are to allow varying tile sizes.

The location of the tiles' Cartesian origin within a subdomain are determined 
by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These

\subsubsection{Two-Dimensional Arrays}


//

\begin{verbatim}
C      NTILES            :: Number of tiles in this topology 
C      MAX_NEIGHBOURS    :: Maximum number of neighbours any tile has.
C      exch2_domain_nxt  :: Total domain length in tiles. 
C      exch2_domain_nyt  :: Maximum domain height in tiles. 
C      exch2_tnx         :: Size in X for each tile.                  
C      exch2_tny         :: Size in Y for each tile.                  
C      exch2_tbasex      :: Tile offset in X within its sub-domain (cube face)
C      exch2_tbasey      :: Tile offset in Y within its sub-domain (cube face)
C      exch2_tglobalxlo  :: Tile base X index within global index space.
C      exch2_tglobalylo  :: Tile base Y index within global index space.
C      exch2_isWedge     :: 0 if West not at domain edge, 1 if it is.   
C      exch2_isNedge     :: 0 if North not at domain edge, 1 if it is.   
C      exch2_isEedge     :: 0 if East not at domain edge, 1 if it is.   
C      exch2_isSedge     :: 0 if South not at domain edge, 1 if it is.   
C      exch2_myFace      :: Cube face number used for I/O.               
C      exch2_nNeighbours :: Tile neighbour entries count.               
C      exch2_tProc       :: Rank of process owning tile                 
C                        :: (filled at run time).                       
C      exch2_neighbourId :: Tile number for each neighbour entry.        
C      exch2_opposingSend_record :: Record for entry in target tile send 
C                                :: list that has this tile and face     
C                                :: as its target.                       
C      exch2_pi          :: X index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_pj          :: Y index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_oi          :: X index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oj          :: Y index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oi_f        :: X index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
C      exch2_oj_f        :: Y index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
\end{verbatim}


\subsection{Key Routines}


\subsection{References}
1	afe	1.4	% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.3 2004/01/29 15:39:49 afe Exp $
2	afe	1.1	% $Name: $
3
4			%% * Introduction
5			%% o what it does, citations (refs go into mitgcm_manual.bib,
6			%% preferably in alphabetic order)
7			%% o Equations
8			%% * Key subroutines and parameters
9			%% * Reference material (auto generated from Protex and structured comments)
10			%% o automatically inserted at \section{Reference}
11
12
13			\section{exch2: Extended Cubed Sphere Exchange}
14	afe	1.3	\label{sec:exch2}
15
16	afe	1.1
17			\subsection{Introduction}
18	afe	1.2
19			The exch2 package is an extension to the original cubed sphere exchanges
20			to allow more flexible domain decomposition and parallelization. Cube faces
21	afe	1.4	(subdomains) may be divided into whatever number of tiles that divide evenly
22	afe	1.2	into the grid point dimensions of the subdomain. Furthermore, the individual
23	afe	1.4	tiles may be run on separate processors in different combinations,
24			and whether exchanges between particular tiles occur between different
25			processors is determined at runtime.
26
27			The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and
28			assigned in {\em w2\_e2setup.F}, both in the
29			{\em pkg/exch2} directory. The validity of the cube topology depends
30			on the {\em SIZE.h} file as detailed below. Both files are generated by
31			Matlab scripts and
32			should not be edited. The default files provided in the release set up
33			a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid
34			points, running on a single processor.
35
36			\subsection{Key Variables}
37
38			The descriptions of the variables are divided up into scalars,
39			one-dimensional arrays indexed to the tile number, and two-dimensional
40			arrays indexed to tile number and neighboring tile. This division
41			actually reflects the functionality of these variables, not just the
42			whim of some FORTRAN enthusiast.
43
44			\subsubsection{Scalars}
45
46			The number of tiles in a particular topology is set with the parameter
47			{\em NTILES}, and the maximum number of neighbors of any tiles by
48			{\em MAX\_NEIGHBOURS}. These parameters are used for defining the size of
49			the various one and two dimensional arrays that store tile parameters
50			indexed to the tile number.
51
52			The scalar parameters {\em exch2\_domain\_nxt} and
53			{\em exch2\_domain\_nyt} express the number of tiles in the x and y global
54			indices. For example, the default setup of six tiles has
55			{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}. A topology of
56			twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will
57			have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}. Note
58			that these parameters express the tile layout to allow global data files that
59			are tile-layout-neutral and have no bearing on the internal storage of the
60			arrays. The tiles are internally stored in a range from {\em 1,bi} (in the
61			x axis) and y-axis variable {\em bj} is generally ignored within the package.
62
63			\subsubsection{One-Dimensional Arrays}
64
65			The following arrays are indexed to the tile number, and the indices are
66			omitted in their descriptions.
67
68			The arrays {\em exch2\_tnx} and {\em exch2\_tny}
69			express the x and y dimensions of each tile. At present for each tile
70			{\em exch2\_tnx = sNx}
71			and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}. Future releases of
72			MITgcm are to allow varying tile sizes.
73
74			The location of the tiles' Cartesian origin within a subdomain are determined
75			by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}. These
76
77			\subsubsection{Two-Dimensional Arrays}
78
79
80			//
81
82			\begin{verbatim}
83			C NTILES :: Number of tiles in this topology
84			C MAX_NEIGHBOURS :: Maximum number of neighbours any tile has.
85			C exch2_domain_nxt :: Total domain length in tiles.
86			C exch2_domain_nyt :: Maximum domain height in tiles.
87			C exch2_tnx :: Size in X for each tile.
88			C exch2_tny :: Size in Y for each tile.
89			C exch2_tbasex :: Tile offset in X within its sub-domain (cube face)
90			C exch2_tbasey :: Tile offset in Y within its sub-domain (cube face)
91			C exch2_tglobalxlo :: Tile base X index within global index space.
92			C exch2_tglobalylo :: Tile base Y index within global index space.
93			C exch2_isWedge :: 0 if West not at domain edge, 1 if it is.
94			C exch2_isNedge :: 0 if North not at domain edge, 1 if it is.
95			C exch2_isEedge :: 0 if East not at domain edge, 1 if it is.
96			C exch2_isSedge :: 0 if South not at domain edge, 1 if it is.
97			C exch2_myFace :: Cube face number used for I/O.
98			C exch2_nNeighbours :: Tile neighbour entries count.
99			C exch2_tProc :: Rank of process owning tile
100			C :: (filled at run time).
101			C exch2_neighbourId :: Tile number for each neighbour entry.
102			C exch2_opposingSend_record :: Record for entry in target tile send
103			C :: list that has this tile and face
104			C :: as its target.
105			C exch2_pi :: X index row of target to source permutation
106			C :: matrix for each neighbour entry.
107			C exch2_pj :: Y index row of target to source permutation
108			C :: matrix for each neighbour entry.
109			C exch2_oi :: X index element of target to source
110			C :: offset vector for cell-centered quantities
111			C :: of each neighbor entry.
112			C exch2_oj :: Y index element of target to source
113			C :: offset vector for cell-centered quantities
114			C :: of each neighbor entry.
115			C exch2_oi_f :: X index element of target to source
116			C :: offset vector for face quantities
117			C :: of each neighbor entry.
118			C exch2_oj_f :: Y index element of target to source
119			C :: offset vector for face quantities
120			C :: of each neighbor entry.
121			\end{verbatim}
122	afe	1.1
123
124
125
126			\subsection{Key Routines}
127
128
129
130			\subsection{References}