| 10 |
%% o automatically inserted at \section{Reference} |
%% o automatically inserted at \section{Reference} |
| 11 |
|
|
| 12 |
|
|
| 13 |
\section{exch2: Extended Cubed Sphere Exchange} |
\section{Extended Cubed Sphere Exchange} |
| 14 |
\label{sec:exch2} |
\label{sec:exch2} |
| 15 |
|
|
| 16 |
|
|
| 17 |
\subsection{Introduction} |
\subsection{Introduction} |
| 18 |
|
|
| 19 |
The exch2 package is an extension to the original cubed sphere exchanges |
The \texttt{exch2} package is an extension to the original cubed |
| 20 |
to allow more flexible domain decomposition and parallelization. Cube faces |
sphere topological configuration that allows more flexible domain |
| 21 |
(subdomains) may be divided into whatever number of tiles that divide evenly |
decomposition and parallelization. Cube faces (also called |
| 22 |
into the grid point dimensions of the subdomain. Furthermore, the individual |
subdomains) may be divided into any number of tiles that divide evenly |
| 23 |
tiles may be run on separate processors in different combinations, |
into the grid point dimensions of the subdomain. Furthermore, the |
| 24 |
and whether exchanges between particular tiles occur between different |
individual tiles may be run on separate processors in different |
| 25 |
processors is determined at runtime. |
combinations, and whether exchanges between particular tiles occur |
| 26 |
|
between different processors is determined at runtime. This |
| 27 |
The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and |
flexibility provides for manual load balancing across a relatively |
| 28 |
assigned in {\em w2\_e2setup.F}, both in the |
arbitrary number of processors. |
| 29 |
{\em pkg/exch2} directory. The validity of the cube topology depends |
|
| 30 |
on the {\em SIZE.h} file as detailed below. Both files are generated by |
The exchange parameters are declared in |
| 31 |
Matlab scripts and |
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} |
| 32 |
should not be edited. The default files provided in the release set up |
and assigned in |
| 33 |
a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid |
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The |
| 34 |
points, running on a single processor. |
validity of the cube topology depends on the \texttt{SIZE.h} file as |
| 35 |
|
detailed below. Both files are generated by Matlab scripts in ?? |
| 36 |
|
check these in already! and should not be edited. The default files |
| 37 |
|
provided in the release configure a cubed sphere arrangement of six |
| 38 |
|
tiles, one per subdomain, each with 32$\times$32 grid points, all |
| 39 |
|
running on a single processor. Pregenerated examples of these files |
| 40 |
|
with alternate topologies are provided in ??. |
| 41 |
|
|
| 42 |
|
\subsection{Invoking exch2} |
| 43 |
|
|
| 44 |
|
To use exch2 with the cubed sphere, the following conditions must be met: |
| 45 |
|
|
| 46 |
|
- the exch2 package is included when \texttt{genmake2} is run. The |
| 47 |
|
easiest way to do this is to add the line \texttt{exch2} to the |
| 48 |
|
\texttt{profile.conf} file -- see Section \ref{sect:buildingCode} |
| 49 |
|
for general details. \\ |
| 50 |
|
|
| 51 |
|
- an example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and |
| 52 |
|
\texttt{w2\_e2setup.F} must reside in a directory containing code |
| 53 |
|
linked when \texttt{genmake2} runs. The safest place to put these |
| 54 |
|
is the directory indicated in the \texttt{-mods=DIR} command line |
| 55 |
|
modifier (typically \texttt{../code}), or the build directory. The |
| 56 |
|
default versions of these files reside in \texttt{pkg/exch2}, but |
| 57 |
|
they should be left untouched to avoid breaking configurations other |
| 58 |
|
than the one you intend to modify.\\ |
| 59 |
|
|
| 60 |
|
- files containing grid parameters, named |
| 61 |
|
\texttt{tile}xxx\texttt{.mitgrid} where xxx is \texttt{001} through |
| 62 |
|
\texttt{006}, must be in the working directory when the MITgcm |
| 63 |
|
executable is run. These files are provided in the example |
| 64 |
|
experiments for cubed sphere configurations with 32$\times$32 cube |
| 65 |
|
sides and are non-trivial to generate -- please contact MITgcm |
| 66 |
|
support if you want to generate files for other configurations. |
| 67 |
|
This is lame. ?? \\ |
| 68 |
|
|
| 69 |
|
As of the time of writing the following examples use exch2 and may be |
| 70 |
|
used for guidance: |
| 71 |
|
|
| 72 |
|
\begin{verbatim} |
| 73 |
|
verification/adjust_nlfs.cs-32x32x1 |
| 74 |
|
verification/adjustment.cs-32x32x1 |
| 75 |
|
verification/aim.5l_cs |
| 76 |
|
verification/global_ocean.cs32x15 |
| 77 |
|
verification/hs94.cs-32x32x5 |
| 78 |
|
\end{verbatim} |
| 79 |
|
|
| 80 |
|
|
| 81 |
|
|
| 82 |
|
|
| 83 |
|
\subsection{Generating Topology Files} |
| 84 |
|
|
| 85 |
\subsection{Key Variables} |
\subsection{Key Variables} |
| 86 |
|
|
| 87 |
The descriptions of the variables are divided up into scalars, |
The descriptions of the variables are divided up into scalars, |
| 88 |
one-dimensional arrays indexed to the tile number, and two-dimensional |
one-dimensional arrays indexed to the tile number, and two and three |
| 89 |
arrays indexed to tile number and neighboring tile. This division |
dimensional arrays indexed to tile number and neighboring tile. This |
| 90 |
actually reflects the functionality of these variables, not just the |
division actually reflects the functionality of these variables: the |
| 91 |
whim of some FORTRAN enthusiast. |
scalars are common to every part of the topology, the tile-indexed |
| 92 |
|
arrays to individual tiles, and the arrays indexed to tile and |
| 93 |
|
neighbor to relationships between tiles and their neighbors. |
| 94 |
|
|
| 95 |
\subsubsection{Scalars} |
\subsubsection{Scalars} |
| 96 |
|
|
| 97 |
The number of tiles in a particular topology is set with the parameter |
The number of tiles in a particular topology is set with the parameter |
| 98 |
{\em NTILES}, and the maximum number of neighbors of any tiles by |
\texttt{NTILES}, and the maximum number of neighbors of any tiles by |
| 99 |
{\em MAX\_NEIGHBOURS}. These parameters are used for defining the size of |
\texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the |
| 100 |
the various one and two dimensional arrays that store tile parameters |
size of the various one and two dimensional arrays that store tile |
| 101 |
indexed to the tile number. |
parameters indexed to the tile number.\\ |
| 102 |
|
|
| 103 |
The scalar parameters {\em exch2\_domain\_nxt} and |
The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt} |
| 104 |
{\em exch2\_domain\_nyt} express the number of tiles in the x and y global |
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number |
| 105 |
indices. For example, the default setup of six tiles has |
of tiles in the x and y global indices. For example, the default |
| 106 |
{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}. A topology of |
setup of six tiles has \texttt{exch2\_domain\_nxt=6} and |
| 107 |
twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will |
\texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in |
| 108 |
have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}. Note |
gridpoints) tiles, four (2x2) per subdomain, will have |
| 109 |
that these parameters express the tile layout to allow global data files that |
\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}. |
| 110 |
are tile-layout-neutral and have no bearing on the internal storage of the |
Note that these parameters express the tile layout to allow global |
| 111 |
arrays. The tiles are internally stored in a range from {\em 1,bi} (in the |
data files that are tile-layout-neutral and have no bearing on the |
| 112 |
x axis) and y-axis variable {\em bj} is generally ignored within the package. |
internal storage of the arrays. The tiles are internally stored in a |
| 113 |
|
range from \texttt{1,bi} (in the x axis) and y-axis variable |
| 114 |
\subsubsection{One-Dimensional Arrays} |
\texttt{bj} is generally ignored within the package. |
| 115 |
|
|
| 116 |
The following arrays are of size {\em NTILES}, are indexed to the tile number, |
\subsubsection{Arrays Indexed to Tile Number} |
| 117 |
and the indices are omitted in their descriptions. |
|
| 118 |
|
The following arrays are of size \texttt{NTILES}, are indexed to the |
| 119 |
The arrays {\em exch2\_tnx} and {\em exch2\_tny} |
tile number, and the indices are omitted in their descriptions. |
| 120 |
express the x and y dimensions of each tile. At present for each tile |
|
| 121 |
{\em exch2\_tnx = sNx} |
The arrays \varlink{exch2\_tnx}{exch2_tnx} and |
| 122 |
and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}. Future releases of |
\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each |
| 123 |
MITgcm are to allow varying tile sizes. |
tile. At present for each tile \texttt{exch2\_tnx=sNx} and |
| 124 |
|
\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future |
| 125 |
The location of the tiles' Cartesian origin within a subdomain are determined |
releases of MITgcm are to allow varying tile sizes. |
| 126 |
by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}. These variables |
|
| 127 |
are used to relate the location of the edges of the tiles to each other. As |
The location of the tiles' Cartesian origin within a subdomain are |
| 128 |
an example, in the default six-tile topology (the degenerate case) |
determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and |
| 129 |
each index in these arrays are |
\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to |
| 130 |
set to 0. The twenty-four, 32x32 cube face case discussed above will have |
relate the location of the edges of the tiles to each other. As an |
| 131 |
values of 0 or 16, depending on the quadrant the tile falls within the |
example, in the default six-tile topology (the degenerate case) each |
| 132 |
subdomain. {\em exch2\_myFace} contains the number of the |
index in these arrays are set to 0. The twenty-four, 32x32 cube face |
| 133 |
cubeface/subdomain of each tile, numbered 1-6 in the case of the standard |
case discussed above will have values of 0 or 16, depending on the |
| 134 |
cube topology. |
quadrant the tile falls within the subdomain. The array |
| 135 |
|
\varlink{exch2\_myFace}{exch2_myFace} contains the number of the |
| 136 |
The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to |
cubeface/subdomain of each tile, numbered 1-6 in the case of the |
| 137 |
{\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within |
standard cube topology. |
| 138 |
the global address space, similar to that used by global files. |
|
| 139 |
|
The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and |
| 140 |
The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge}, |
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to |
| 141 |
and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge |
\varlink{exch2\_tbasex}{exch2_tbasex} and |
| 142 |
of a subdomain, 0 if not. The values are used within the topology generator |
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the |
| 143 |
to determine the orientation of neighboring tiles and to indicate whether |
global address space, similar to that used by global files. |
| 144 |
a tile lies on the corner of a subdomain. The latter case indicates |
|
| 145 |
special exchange and numerical handling for the singularities at the eight |
The arrays \varlink{exch2\_isWedge}{exch2_isWedge}, |
| 146 |
corners of the cube. {\em exch2\_isNedge} contains a count of how many |
\varlink{exch2\_isEedge}{exch2_isEedge}, |
| 147 |
neighboring tiles each tile has, and is used for setting bounds for looping |
\varlink{exch2\_isSedge}{exch2_isSedge}, and |
| 148 |
over neighboring tiles. {\em exch2\_tProc} holds the process rank of each tile, |
\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed |
| 149 |
and is used in interprocess communication. |
tile lies on the edge of a subdomain, 0 if not. The values are used |
| 150 |
|
within the topology generator to determine the orientation of |
| 151 |
\subsubsection{Two-Dimensional Arrays} |
neighboring tiles and to indicate whether a tile lies on the corner of |
| 152 |
|
a subdomain. The latter case indicates special exchange and numerical |
| 153 |
The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and |
handling for the singularities at the eight corners of the cube. |
| 154 |
describe the orientations between the the tiles. |
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of |
| 155 |
|
how many neighboring tiles each tile has, and is used for setting |
| 156 |
The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile |
bounds for looping over neighboring tiles. |
| 157 |
{\em T}'s neighbor tile {\em a}, and {\em exch2\_opposingSend\_record(a,T)} holds |
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each |
| 158 |
the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T. |
tile, and is used in interprocess communication. |
| 159 |
In other words, |
|
| 160 |
|
\subsubsection{Arrays Indexed to Tile Number and Neighbor} |
| 161 |
\begin{verbatim} |
|
| 162 |
exch2_neighbourId( exch2_opposingSend_record(a,T), exch2_neighbourId(a,T) ) = T |
The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$ |
| 163 |
|
\texttt{NTILES} and describe the orientations between the the tiles. |
| 164 |
|
|
| 165 |
|
The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for |
| 166 |
|
each of the $n$ neighboring tiles. The neighbor tiles are indexed |
| 167 |
|
\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north |
| 168 |
|
then south edges, and then top to bottom on the east and west edges. |
| 169 |
|
Maybe throw in a fig here, eh? |
| 170 |
|
|
| 171 |
|
The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c |
| 172 |
|
in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T. |
| 173 |
|
In other words, |
| 174 |
|
\begin{verbatim} |
| 175 |
|
exch2_neighbourId( exch2_opposingSend_record(a,T), |
| 176 |
|
exch2_neighbourId(a,T) ) = T |
| 177 |
\end{verbatim} |
\end{verbatim} |
| 178 |
|
and this provides a back-reference from the neighbor tiles. |
| 179 |
|
|
| 180 |
{\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}. |
The arrays \varlink{exch2\_pi}{exch2_pi}, |
| 181 |
This is to provide a backreference from the neighbor tiles. |
\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi}, |
| 182 |
|
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and |
| 183 |
|
\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in |
| 184 |
|
exchanges between the neighboring tiles. The dimensions of |
| 185 |
|
\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the |
| 186 |
|
neighbor ID \textit{N} and the tile number \textit{T} as explained |
| 187 |
|
above, plus the transformation vector {\em t }, of length two. The |
| 188 |
|
first element of the transformation vector indicates the factor by |
| 189 |
|
which variables representing the same vector component of a tile will |
| 190 |
|
be multiplied, and the second element indicates the transform to the |
| 191 |
|
variable in the other direction. As an example, |
| 192 |
|
\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a |
| 193 |
|
vector variable in tile \texttt{T} to the i-component of tile |
| 194 |
|
\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold |
| 195 |
|
the component of neighbor \texttt{N}'s j-component. |
| 196 |
|
|
| 197 |
|
Under the current cube topology, one of the two elements of |
| 198 |
|
\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T} |
| 199 |
|
and neighbor \texttt{N} will be 0, reflecting the fact that the vector |
| 200 |
|
components are orthogonal. The other element will be 1 or -1, |
| 201 |
|
depending on whether the components are indexed in the same or |
| 202 |
|
opposite directions. For example, the transform dimension of the |
| 203 |
|
arrays for all tile neighbors on the same subdomain will be [1,0], |
| 204 |
|
since all tiles on the same subdomain are oriented identically. |
| 205 |
|
Vectors that correspond to the orthogonal dimension with the same |
| 206 |
|
index direction will have [0,1], whereas those in the opposite index |
| 207 |
|
direction will have [0,-1]. |
| 208 |
|
|
|
// |
|
| 209 |
|
|
| 210 |
|
{\footnotesize |
| 211 |
\begin{verbatim} |
\begin{verbatim} |
|
|
|
|
|
|
|
C exch2_neighbourId :: Tile number for each neighbour entry. |
|
|
C exch2_opposingSend_record :: Record for entry in target tile send |
|
|
C :: list that has this tile and face |
|
|
C :: as its target. |
|
| 212 |
C exch2_pi :: X index row of target to source permutation |
C exch2_pi :: X index row of target to source permutation |
| 213 |
C :: matrix for each neighbour entry. |
C :: matrix for each neighbour entry. |
| 214 |
C exch2_pj :: Y index row of target to source permutation |
C exch2_pj :: Y index row of target to source permutation |
| 226 |
C :: offset vector for face quantities |
C :: offset vector for face quantities |
| 227 |
C :: of each neighbor entry. |
C :: of each neighbor entry. |
| 228 |
\end{verbatim} |
\end{verbatim} |
| 229 |
|
} |
| 230 |
|
|
| 231 |
|
|
| 232 |
|
|
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch