/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Annotation of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph


Revision 1.10 - (hide annotations) (download) (as text)
Mon Mar 15 20:11:56 2004 UTC (21 years, 4 months ago) by afe
Branch: MAIN
Changes since 1.9: +106 -26 lines
File MIME type: application/x-tex
o added more how-to about exch2 configuration, and figures

1 afe 1.10 % $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.9 2004/03/12 20:58:19 afe Exp $
2 afe 1.1 % $Name: $
3    
4     %% * Introduction
5     %% o what it does, citations (refs go into mitgcm_manual.bib,
6     %% preferably in alphabetic order)
7     %% o Equations
8     %% * Key subroutines and parameters
9     %% * Reference material (auto generated from Protex and structured comments)
10     %% o automatically inserted at \section{Reference}
11    
12    
13 afe 1.10 \section{exch2: Extended Cubed Sphere \mbox{Topology}}
14 afe 1.3 \label{sec:exch2}
15    
16 afe 1.1
17     \subsection{Introduction}
18 afe 1.2
19 edhill 1.8 The \texttt{exch2} package is an extension to the original cubed
20 afe 1.9 sphere topological configuration that allows more flexible domain
21     decomposition and parallelization. Cube faces (also called
22     subdomains) may be divided into any number of tiles that divide evenly
23     into the grid point dimensions of the subdomain. Furthermore, the
24     individual tiles may be run on separate processors in different
25     combinations, and whether exchanges between particular tiles occur
26     between different processors is determined at runtime. This
27 afe 1.10 flexibility provides for manual compile-time load balancing across a
28     relatively arbitrary number of processors. \\
29 edhill 1.8
30     The exchange parameters are declared in
31     \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32     and assigned in
33 afe 1.9 \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34     validity of the cube topology depends on the \texttt{SIZE.h} file as
35 afe 1.10 detailed below. Both files are generated by Matlab scripts in
36     \texttt{utils/exch2/matlab-topology-generator}; see Section
37     \ref{sec:topogen} for details on creating alternate topologies. The
38     default files provided in the release configure a cubed sphere
39     topology of six tiles, one per subdomain, each with 32$\times$32 grid
40     points, all running on a single processor. Pregenerated examples of
41     these files with alternate topologies are provided under
42     \texttt{utils/exch2/code-mods} along with the appropriate
43     \texttt{SIZE.h} file for single-processor execution.
44 afe 1.9
45     \subsection{Invoking exch2}
46    
47 afe 1.10 To use exch2 with the cubed sphere, the following conditions must be
48     met: \\
49 afe 1.9
50 afe 1.10 $\bullet$ The exch2 package is included when \texttt{genmake2} is run. The
51 afe 1.9 easiest way to do this is to add the line \texttt{exch2} to the
52     \texttt{profile.conf} file -- see Section \ref{sect:buildingCode}
53     for general details. \\
54    
55 afe 1.10 $\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and
56 afe 1.9 \texttt{w2\_e2setup.F} must reside in a directory containing code
57     linked when \texttt{genmake2} runs. The safest place to put these
58     is the directory indicated in the \texttt{-mods=DIR} command line
59     modifier (typically \texttt{../code}), or the build directory. The
60 afe 1.10 default versions of these files reside in \texttt{pkg/exch2} and are
61     linked automatically if no other versions exist elsewhere in the
62     link path, but they should be left untouched to avoid breaking
63     configurations other than the one you intend to modify.\\
64    
65     $\bullet$ Files containing grid parameters, named
66     \texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through
67     \texttt{006} (one per subdomain), must be in the working directory
68     when the MITgcm executable is run. These files are provided in the
69     example experiments for cubed sphere configurations with
70     32$\times$32 cube sides and are non-trivial to generate -- please
71     contact MITgcm support if you want to generate files for other
72     configurations. \\
73    
74     $\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h}
75     must be placed where \texttt{genmake2} will find it. In particular
76     for the exch2, the domain decompositin specified in \texttt{SIZE.h}
77     must correspond with the particular configuration's topology
78     specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and
79     \texttt{w2\_e2setup.F}. Domain decomposition issues particular to
80     exch2 are addressed in Section \ref{sec:topogen}: ``Generating
81     Topology Files for exch2''; a more general background on the subject
82     relvant to MITgcm is presented in Section
83     \ref{sect:specifying_a_decomposition}: ``Specifying a
84     decomposition''.\\
85 afe 1.9
86     As of the time of writing the following examples use exch2 and may be
87     used for guidance:
88    
89     \begin{verbatim}
90     verification/adjust_nlfs.cs-32x32x1
91     verification/adjustment.cs-32x32x1
92     verification/aim.5l_cs
93     verification/global_ocean.cs32x15
94     verification/hs94.cs-32x32x5
95     \end{verbatim}
96    
97    
98    
99    
100 afe 1.10 \subsection{Generating Topology Files for exch2}
101     \label{sec:topogen}
102    
103     Alternate cubed sphere topologies may be created using the Matlab
104     scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the
105     m-file \texttt{driver} from the Matlab prompt (without passing any
106     function parameters) generates exch2 topology files
107     \texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the
108     working directory and displays via Matlab a figure of the topology.
109     The other m-files in the directory are subroutines of \texttt{driver}
110     and should not be run except for development purposes. \\
111    
112     The parameters that determine the dimensions and topology of the
113     generated configuration are nr, nb, ng, tnx and tny, and all are
114     assigned early in the script.
115    
116     The first three determine the size of the subdomains (cube faces) and
117     hence the size of the overall domain. Each one determines the number
118     of grid points, and therefore the resolution, along the subdomain
119     sides in a ``great circle'' around each axis of the cube. At the time
120     of this writing MITgcm requires these three parameters to be equal,
121     but they provide for future releases of MITgcm to accomodate different
122     resolutions around the axes to allow (for example) greater resolution
123     around the equator.\\
124    
125     The parameters tnx and tny determine the dimensions of the tiles into
126     which the subdomains are decomposed, and must evenly divide the
127     integer assigned to nr, nb and ng. The result is a rectangular tiling
128     of the subdomain. Figure \ref{fig:24tile} shows one possible topology
129     for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for
130     twelve tiles. \\
131    
132     \begin{figure}
133     \begin{center}
134     \resizebox{4in}{!}{
135     \includegraphics{part6/s24t_16x16.ps}
136     }
137     \end{center}
138     \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
139     twenty-four tiles (tnx=16, tny=16)
140     } \label{fig:24tile}
141     \end{figure}
142    
143     \begin{figure}
144     \begin{center}
145     \resizebox{4in}{!}{
146     \includegraphics{part6/s12t_16x32.ps}
147     }
148     \end{center}
149     \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
150     twelve tiles (tnx=16, tny=32)
151     } \label{fig:12tile}
152     \end{figure}
153    
154     Tiles can be selected from the topology to be omitted from being
155     allocated memory and processors. This kind of tuning is useful in
156     ocean modeling for omitting tiles that fall entirely on land. The
157     tiles omitted are specified in the file \texttt{blanklist.txt} by
158     their tile number in the topology, separated by a newline. \\
159    
160    
161    
162    
163    
164 afe 1.4
165     \subsection{Key Variables}
166    
167     The descriptions of the variables are divided up into scalars,
168 edhill 1.8 one-dimensional arrays indexed to the tile number, and two and three
169     dimensional arrays indexed to tile number and neighboring tile. This
170     division actually reflects the functionality of these variables: the
171     scalars are common to every part of the topology, the tile-indexed
172     arrays to individual tiles, and the arrays indexed to tile and
173     neighbor to relationships between tiles and their neighbors.
174 afe 1.4
175     \subsubsection{Scalars}
176    
177     The number of tiles in a particular topology is set with the parameter
178 edhill 1.8 \texttt{NTILES}, and the maximum number of neighbors of any tiles by
179     \texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the
180     size of the various one and two dimensional arrays that store tile
181 afe 1.9 parameters indexed to the tile number.\\
182 edhill 1.8
183     The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
184     and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
185     of tiles in the x and y global indices. For example, the default
186     setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
187     \texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in
188     gridpoints) tiles, four (2x2) per subdomain, will have
189     \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
190     Note that these parameters express the tile layout to allow global
191     data files that are tile-layout-neutral and have no bearing on the
192     internal storage of the arrays. The tiles are internally stored in a
193     range from \texttt{1,bi} (in the x axis) and y-axis variable
194     \texttt{bj} is generally ignored within the package.
195 afe 1.4
196 afe 1.6 \subsubsection{Arrays Indexed to Tile Number}
197 afe 1.4
198 edhill 1.8 The following arrays are of size \texttt{NTILES}, are indexed to the
199     tile number, and the indices are omitted in their descriptions.
200 afe 1.4
201 edhill 1.8 The arrays \varlink{exch2\_tnx}{exch2_tnx} and
202     \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
203     tile. At present for each tile \texttt{exch2\_tnx=sNx} and
204     \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future
205     releases of MITgcm are to allow varying tile sizes.
206    
207     The location of the tiles' Cartesian origin within a subdomain are
208     determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
209     \varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to
210     relate the location of the edges of the tiles to each other. As an
211     example, in the default six-tile topology (the degenerate case) each
212     index in these arrays are set to 0. The twenty-four, 32x32 cube face
213     case discussed above will have values of 0 or 16, depending on the
214     quadrant the tile falls within the subdomain. The array
215     \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
216     cubeface/subdomain of each tile, numbered 1-6 in the case of the
217     standard cube topology.
218    
219     The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
220     \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
221     \varlink{exch2\_tbasex}{exch2_tbasex} and
222     \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
223     global address space, similar to that used by global files.
224    
225     The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
226     \varlink{exch2\_isEedge}{exch2_isEedge},
227     \varlink{exch2\_isSedge}{exch2_isSedge}, and
228     \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
229     tile lies on the edge of a subdomain, 0 if not. The values are used
230     within the topology generator to determine the orientation of
231     neighboring tiles and to indicate whether a tile lies on the corner of
232     a subdomain. The latter case indicates special exchange and numerical
233     handling for the singularities at the eight corners of the cube.
234     \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
235     how many neighboring tiles each tile has, and is used for setting
236     bounds for looping over neighboring tiles.
237     \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
238     tile, and is used in interprocess communication.
239 afe 1.4
240 afe 1.6 \subsubsection{Arrays Indexed to Tile Number and Neighbor}
241 afe 1.4
242 edhill 1.8 The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
243     \texttt{NTILES} and describe the orientations between the the tiles.
244 afe 1.5
245 edhill 1.8 The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
246     each of the $n$ neighboring tiles. The neighbor tiles are indexed
247     \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
248     then south edges, and then top to bottom on the east and west edges.
249     Maybe throw in a fig here, eh?
250    
251     The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
252     in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
253     In other words,
254     \begin{verbatim}
255     exch2_neighbourId( exch2_opposingSend_record(a,T),
256     exch2_neighbourId(a,T) ) = T
257 afe 1.5 \end{verbatim}
258 edhill 1.8 and this provides a back-reference from the neighbor tiles.
259 afe 1.5
260 edhill 1.8 The arrays \varlink{exch2\_pi}{exch2_pi},
261     \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
262     \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
263     \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
264     exchanges between the neighboring tiles. The dimensions of
265     \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
266     neighbor ID \textit{N} and the tile number \textit{T} as explained
267     above, plus the transformation vector {\em t }, of length two. The
268     first element of the transformation vector indicates the factor by
269     which variables representing the same vector component of a tile will
270     be multiplied, and the second element indicates the transform to the
271     variable in the other direction. As an example,
272     \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
273     vector variable in tile \texttt{T} to the i-component of tile
274     \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
275     the component of neighbor \texttt{N}'s j-component.
276    
277     Under the current cube topology, one of the two elements of
278     \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
279     and neighbor \texttt{N} will be 0, reflecting the fact that the vector
280     components are orthogonal. The other element will be 1 or -1,
281     depending on whether the components are indexed in the same or
282     opposite directions. For example, the transform dimension of the
283     arrays for all tile neighbors on the same subdomain will be [1,0],
284     since all tiles on the same subdomain are oriented identically.
285     Vectors that correspond to the orthogonal dimension with the same
286     index direction will have [0,1], whereas those in the opposite index
287     direction will have [0,-1].
288 afe 1.5
289 afe 1.4
290 edhill 1.8 {\footnotesize
291 afe 1.4 \begin{verbatim}
292     C exch2_pi :: X index row of target to source permutation
293     C :: matrix for each neighbour entry.
294     C exch2_pj :: Y index row of target to source permutation
295     C :: matrix for each neighbour entry.
296     C exch2_oi :: X index element of target to source
297     C :: offset vector for cell-centered quantities
298     C :: of each neighbor entry.
299     C exch2_oj :: Y index element of target to source
300     C :: offset vector for cell-centered quantities
301     C :: of each neighbor entry.
302     C exch2_oi_f :: X index element of target to source
303     C :: offset vector for face quantities
304     C :: of each neighbor entry.
305     C exch2_oj_f :: Y index element of target to source
306     C :: offset vector for face quantities
307     C :: of each neighbor entry.
308     \end{verbatim}
309 edhill 1.8 }
310 afe 1.1
311    
312    
313     \subsection{Key Routines}
314    
315    
316    
317     \subsection{References}

  ViewVC Help
Powered by ViewVC 1.1.22