/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.10 by afe, Mon Mar 15 20:11:56 2004 UTC revision 1.27 by jmc, Sat May 2 02:13:18 2009 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere \mbox{Topology}}  \subsection{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsubsection{Introduction}
18    
19  The \texttt{exch2} package is an extension to the original cubed  The \texttt{exch2} package extends the original cubed sphere topology
20  sphere topological configuration that allows more flexible domain  configuration to allow more flexible domain decomposition and
21  decomposition and parallelization.  Cube faces (also called  parallelization.  Cube faces (also called subdomains) may be divided
22  subdomains) may be divided into any number of tiles that divide evenly  into any number of tiles that divide evenly into the grid point
23  into the grid point dimensions of the subdomain.  Furthermore, the  dimensions of the subdomain.  Furthermore, the tiles can run on
24  individual tiles may be run on separate processors in different  separate processors individually or in groups, which provides for
25  combinations, and whether exchanges between particular tiles occur  manual compile-time load balancing across a relatively arbitrary
26  between different processors is determined at runtime.  This  number of processors.
 flexibility provides for manual compile-time load balancing across a  
 relatively arbitrary number of processors. \\  
27    
28  The exchange parameters are declared in  The exchange parameters are declared in
29  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
30  and assigned in  and assigned in
31  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
32  validity of the cube topology depends on the \texttt{SIZE.h} file as  validity of the cube topology depends on the \file{SIZE.h} file as
33  detailed below.  Both files are generated by Matlab scripts in  detailed below.  The default files provided in the release configure a
34  \texttt{utils/exch2/matlab-topology-generator}; see Section  cubed sphere topology of six tiles, one per subdomain, each with
35  \ref{sec:topogen} for details on creating alternate topologies.  The  32$\times$32 grid points, with all tiles running on a single processor.  Both
36  default files provided in the release configure a cubed sphere  files are generated by Matlab scripts in
37  topology of six tiles, one per subdomain, each with 32$\times$32 grid  \file{utils/exch2/matlab-topology-generator}; see Section
38  points, all running on a single processor.  Pregenerated examples of  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
39  these files with alternate topologies are provided under  for details on creating alternate topologies.  Pregenerated examples
40  \texttt{utils/exch2/code-mods} along with the appropriate  of these files with alternate topologies are provided under
41  \texttt{SIZE.h} file for single-processor execution.  \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
42    file for single-processor execution.
43    
44  \subsection{Invoking exch2}  \subsubsection{Invoking exch2}
45    
46  To use exch2 with the cubed sphere, the following conditions must be  To use exch2 with the cubed sphere, the following conditions must be
47  met: \\  met:
48    
49  $\bullet$ The exch2 package is included when \texttt{genmake2} is run.  The  \begin{itemize}
50    easiest way to do this is to add the line \texttt{exch2} to the  \item The exch2 package is included when \file{genmake2} is run.  The
51    \texttt{profile.conf} file -- see Section \ref{sect:buildingCode}    easiest way to do this is to add the line \code{exch2} to the
52    for general details. \\    \file{packages.conf} file -- see Section \ref{sect:buildingCode}
53      \sectiontitle{Building the code} for general
54  $\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and    details.
55    \texttt{w2\_e2setup.F} must reside in a directory containing code  
56    linked when \texttt{genmake2} runs.  The safest place to put these  \item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
57    is the directory indicated in the \texttt{-mods=DIR} command line    \file{w2\_e2setup.F} must reside in a directory containing files
58    modifier (typically \texttt{../code}), or the build directory.  The    symbolically linked by the \file{genmake2} script.  The safest place
59    default versions of these files reside in \texttt{pkg/exch2} and are    to put these is the directory indicated in the \code{-mods=DIR}
60    linked automatically if no other versions exist elsewhere in the    command line modifier (typically \file{../code}), or the build
61    link path, but they should be left untouched to avoid breaking    directory.  The default versions of these files reside in
62    configurations other than the one you intend to modify.\\    \file{pkg/exch2} and are linked automatically if no other versions
63      exist elsewhere in the build path, but they should be left untouched
64  $\bullet$ Files containing grid parameters, named    to avoid breaking configurations other than the one you intend to
65    \texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through    modify.
66    \texttt{006} (one per subdomain), must be in the working directory  
67    when the MITgcm executable is run.  These files are provided in the  \item Files containing grid parameters, named \file{tile00$n$.mitgrid}
68    example experiments for cubed sphere configurations with    where $n$=\code{(1:6)} (one per subdomain), must be in the working
69    32$\times$32 cube sides and are non-trivial to generate -- please    directory when the MITgcm executable is run.  These files are
70    contact MITgcm support if you want to generate files for other    provided in the example experiments for cubed sphere configurations
71    configurations. \\    with 32$\times$32 cube sides -- please contact MITgcm support if you
72      want to generate files for other configurations.
73  $\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h}  
74    must be placed where \texttt{genmake2} will find it.  In particular  \item As always when compiling MITgcm, the file \file{SIZE.h} must be
75    for the exch2, the domain decompositin specified in \texttt{SIZE.h}    placed where \file{genmake2} will find it.  In particular for exch2,
76    must correspond with the particular configuration's topology    the domain decomposition specified in \file{SIZE.h} must correspond
77    specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and    with the particular configuration's topology specified in
78    \texttt{w2\_e2setup.F}.  Domain decomposition issues particular to    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
79    exch2 are addressed in Section \ref{sec:topogen}: ``Generating    decomposition issues particular to exch2 are addressed in Section
80    Topology Files for exch2''; a more general background on the subject    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
81    relvant to MITgcm is presented in Section    and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
82    \ref{sect:specifying_a_decomposition}: ``Specifying a      Multiprocessing}; a more general background on the subject
83    decomposition''.\\    relevant to MITgcm is presented in Section
84      \ref{sect:specifying_a_decomposition}
85      \sectiontitle{Specifying a decomposition}.
86    \end{itemize}
87    
88  As of the time of writing the following examples use exch2 and may be  At the time of this writing the following examples use exch2 and may
89  used for guidance:  be used for guidance:
90    
91  \begin{verbatim}  \begin{verbatim}
92  verification/adjust_nlfs.cs-32x32x1  verification/adjust_nlfs.cs-32x32x1
# Line 97  verification/hs94.cs-32x32x5 Line 99  verification/hs94.cs-32x32x5
99    
100    
101    
102  \subsection{Generating Topology Files for exch2}  \subsubsection{Generating Topology Files for exch2}
103  \label{sec:topogen}  \label{sec:topogen}
104    
105  Alternate cubed sphere topologies may be created using the Matlab  Alternate cubed sphere topologies may be created using the Matlab
106  scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the  scripts in \file{utils/exch2/matlab-topology-generator}. Running the
107  m-file \texttt{driver} from the Matlab prompt (without passing any  m-file
108  function parameters) generates exch2 topology files  \filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m}
109  \texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the  from the Matlab prompt (there are no parameters to pass) generates
110  working directory and displays via Matlab a figure of the topology.  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
111  The other m-files in the directory are subroutines of \texttt{driver}  \file{w2\_e2setup.F} in the working directory and displays a figure of
112  and should not be run except for development purposes. \\  the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:18tile},
113    and \ref{fig:48tile} are examples of the generated diagrams.  The other
114    m-files in the directory are
115    subroutines called from \file{driver.m} and should not be run ``bare'' except
116    for development purposes. \\
117    
118  The parameters that determine the dimensions and topology of the  The parameters that determine the dimensions and topology of the
119  generated configuration are nr, nb, ng, tnx and tny, and all are  generated configuration are \code{nr}, \code{nb}, \code{ng},
120  assigned early in the script.  \code{tnx} and \code{tny}, and all are assigned early in the script. \\
121    
122  The first three determine the size of the subdomains (cube faces) and  The first three determine the height and width of the subdomains and
123  hence the size of the overall domain.  Each one determines the number  hence the size of the overall domain.  Each one determines the number
124  of grid points, and therefore the resolution, along the subdomain  of grid points, and therefore the resolution, along the subdomain
125  sides in a ``great circle'' around each axis of the cube.  At the time  sides in a ``great circle'' around each the three spatial axes of the cube.  At the time
126  of this writing MITgcm requires these three parameters to be equal,  of this writing MITgcm requires these three parameters to be equal,
127  but they provide for future releases of MITgcm to accomodate different  but they provide for future releases  to accomodate different
128  resolutions around the axes to allow (for example) greater resolution  resolutions around the axes to allow subdomains with differing resolutions.\\
129  around the equator.\\  
130    The parameters \code{tnx} and \code{tny} determine the width and height of
131  The parameters tnx and tny determine the dimensions of the tiles into  the tiles into which the subdomains are decomposed, and must evenly
132  which the subdomains are decomposed, and must evenly divide the  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
133  integer assigned to nr, nb and ng.  The result is a rectangular tiling  The result is a rectangular tiling of the subdomain.  Figure
134  of the subdomain.  Figure \ref{fig:24tile} shows one possible topology  \ref{fig:48tile} shows one possible topology for a twenty-four-tile
135  for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for  cube, and figure \ref{fig:6tile} shows one for six tiles. \\
 twelve tiles. \\  
136    
137  \begin{figure}  \begin{figure}
138  \begin{center}  \begin{center}
139   \resizebox{4in}{!}{   \resizebox{6in}{!}{
140    \includegraphics{part6/s24t_16x16.ps}  % \includegraphics{part6/s24t_16x16.ps}
141      \includegraphics{part6/adjust_cs.ps}
142     }
143    \end{center}
144    
145    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
146    divided into six 32$\times$32 subdomains, each of which is divided
147    into eight tiles of width \code{tnx=16} and height \code{tny=8} for a
148    total of forty-eight tiles. The colored borders of the subdomains
149    represent the parameters \code{nr} (red), \code{ng} (green), and
150    \code{nb} (blue).
151    This tiling is used in the example
152    verification/adjustment.cs-32x32x1/
153    with the option (blanklist.txt) to remove the land-only 4 tiles
154    (11,12,13,14) which are filled in red on the plot.
155    } \label{fig:48tile}
156    \end{figure}
157    
158    \begin{figure}
159    \begin{center}
160     \resizebox{6in}{!}{
161    % \includegraphics{part6/s12t_16x32.ps}
162      \includegraphics{part6/polarcap.ps}
163   }   }
164  \end{center}  \end{center}
165  \caption{Plot of cubed sphere topology with a 32$\times$32 grid and  \caption{Plot of a non-square cubed sphere topology with
166  twenty-four tiles (tnx=16, tny=16)  6 subdomains of different size (nr=90,ng=360,nb=90),
167  } \label{fig:24tile}  divided into one to four tiles each
168     (\code{tnx=90, tny=90}), resulting in a total of 18 tiles.
169    } \label{fig:18tile}
170  \end{figure}  \end{figure}
171    
172  \begin{figure}  \begin{figure}
173  \begin{center}  \begin{center}
174   \resizebox{4in}{!}{   \resizebox{4in}{!}{
175    \includegraphics{part6/s12t_16x32.ps}  % \includegraphics{part6/s6t_32x32.ps}
176      \includegraphics{part6/s6t_32x32.ps}
177   }   }
178  \end{center}  \end{center}
179  \caption{Plot of cubed sphere topology with a 32$\times$32 grid and  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
180  twelve tiles (tnx=16, tny=32)  divided into six 32$\times$32 subdomains with one tile each
181  } \label{fig:12tile}  (\code{tnx=32, tny=32}).  This is the default configuration.
182      }
183    \label{fig:6tile}
184  \end{figure}  \end{figure}
185    
186    
187  Tiles can be selected from the topology to be omitted from being  Tiles can be selected from the topology to be omitted from being
188  allocated memory and processors.  This kind of tuning is useful in  allocated memory and processors.  This tuning is useful in ocean
189  ocean modeling for omitting tiles that fall entirely on land.  The  modeling for omitting tiles that fall entirely on land.  The tiles
190  tiles omitted are specified in the file \texttt{blanklist.txt} by  omitted are specified in the file
191  their tile number in the topology, separated by a newline. \\  \filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt}
192    by their tile number in the topology, separated by a newline. \\
193    
194    
195    
196    
197    \subsubsection{exch2, SIZE.h, and Multiprocessing}
198    \label{sec:exch2mpi}
199    
200    Once the topology configuration files are created, the Fortran
201    \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
202    Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
203      a decomposition} provides a general description of domain
204    decomposition within MITgcm and its relation to \file{SIZE.h}. The
205    current section specifies constraints that the exch2 package imposes
206    and describes how to enable parallel execution with MPI.
207    
208    As in the general case, the parameters \varlink{sNx}{sNx} and
209    \varlink{sNy}{sNy} define the size of the individual tiles, and so
210    must be assigned the same respective values as \code{tnx} and
211    \code{tny} in \file{driver.m}.
212    
213    The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
214    have no special bearing on exch2 and may be assigned as in the general
215    case. The same holds for \varlink{Nr}{Nr}, the number of vertical
216    levels in the model.
217    
218    The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
219    \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
220    tiles and how they are distributed on processors.  When using exch2,
221    the tiles are stored in the $x$ dimension, and so
222    \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
223    configured by exch2 cannot be split up accross processors without
224    regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well.
225    
226    The number of tiles MITgcm allocates and how they are distributed
227    between processors depends on \varlink{nPx}{nPx} and
228    \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
229    processor and \varlink{nPx}{nPx} is the number of processors.  The
230    total number of tiles in the topology minus those listed in
231    \file{blanklist.txt} must equal \code{nSx*nPx}.  Note that in order to
232    obtain maximum usage from a given number of processors in some cases,
233    this restriction might entail sharing a processor with a tile that
234    would otherwise be excluded because it is topographically outside of
235    the domain and therefore in \file{blanklist.txt}.  For example,
236    suppose you have five processors and a domain decomposition of
237    thirty-six tiles that allows you to exclude seven tiles.  To evenly
238    distribute the remaining twenty-nine tiles among five processors, you
239    would have to run one ``dummy'' tile to make an even six tiles per
240    processor.  Such dummy tiles are \emph{not} listed in
241    \file{blanklist.txt}.
242    
243    The following is an example of \file{SIZE.h} for the six-tile
244    configuration illustrated in figure \ref{fig:6tile}
245    running on one processor:
246    
247    \begin{verbatim}
248          PARAMETER (
249         &           sNx =  32,
250         &           sNy =  32,
251         &           OLx =   2,
252         &           OLy =   2,
253         &           nSx =   6,
254         &           nSy =   1,
255         &           nPx =   1,
256         &           nPy =   1,
257         &           Nx  = sNx*nSx*nPx,
258         &           Ny  = sNy*nSy*nPy,
259         &           Nr  =   5)
260    \end{verbatim}
261    
262    The following is an example for the forty-eight-tile topology in
263    figure \ref{fig:48tile} running on six processors:
264    
265    \begin{verbatim}
266          PARAMETER (
267         &           sNx =  16,
268         &           sNy =   8,
269         &           OLx =   2,
270         &           OLy =   2,
271         &           nSx =   8,
272         &           nSy =   1,
273         &           nPx =   6,
274         &           nPy =   1,
275         &           Nx  = sNx*nSx*nPx,
276         &           Ny  = sNy*nSy*nPy,
277         &           Nr  =   5)
278    \end{verbatim}
279    
280    
281  \subsection{Key Variables}  \subsubsection{Key Variables}
282    
283  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
284  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and
285  dimensional arrays indexed to tile number and neighboring tile.  This  three-dimensional arrays indexed to tile number and neighboring tile.
286  division actually reflects the functionality of these variables: the  This division reflects the functionality of these variables: The
287  scalars are common to every part of the topology, the tile-indexed  scalars are common to every part of the topology, the tile-indexed
288  arrays to individual tiles, and the arrays indexed to tile and  arrays to individual tiles, and the arrays indexed by tile and
289  neighbor to relationships between tiles and their neighbors.  neighbor to relationships between tiles and their neighbors. \\
290    
291  \subsubsection{Scalars}  Scalars:
292    
293  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
294  \texttt{NTILES}, and the maximum number of neighbors of any tiles by  \code{NTILES}, and the maximum number of neighbors of any tiles by
295  \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the  \code{MAX\_NEIGHBOURS}.  These parameters are used for defining the
296  size of the various one and two dimensional arrays that store tile  size of the various one and two dimensional arrays that store tile
297  parameters indexed to the tile number.\\  parameters indexed to the tile number and are assigned in the files
298    generated by \file{driver.m}.\\
299    
300  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
301  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
302  of tiles in the x and y global indices.  For example, the default  of tiles in the $x$ and $y$ global indices.  For example, the default
303  setup of six tiles has \texttt{exch2\_domain\_nxt=6} and  setup of six tiles (Fig. \ref{fig:6tile}) has
304  \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in  \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
305  gridpoints) tiles, four (2x2) per subdomain, will have  topology of forty-eight tiles, eight per subdomain (as in figure
306  \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.  \ref{fig:48tile}), will have \code{exch2\_domain\_nxt=12} and
307  Note that these parameters express the tile layout to allow global  \code{exch2\_domain\_nyt=4}.  Note that these parameters express the
308  data files that are tile-layout-neutral and have no bearing on the  tile layout in order to allow global data files that are tile-layout-neutral.
309  internal storage of the arrays.  The tiles are internally stored in a  They have no bearing on the internal storage of the arrays.  The tiles
310  range from \texttt{1,bi} (in the x axis) and y-axis variable  are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the
311  \texttt{bj} is generally ignored within the package.  $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to
312    equal \code{1} throughout the package. \\
313  \subsubsection{Arrays Indexed to Tile Number}  
314    Arrays indexed to tile number:
315  The following arrays are of size \texttt{NTILES}, are indexed to the  
316  tile number, and the indices are omitted in their descriptions.  The following arrays are of length \code{NTILES} and are indexed to
317    the tile number, which is indicated in the diagrams with the notation
318    \textsf{t}$n$.  The indices are omitted in the descriptions. \\
319    
320  The arrays \varlink{exch2\_tnx}{exch2_tnx} and  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
321  \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
322  tile.  At present for each tile \texttt{exch2\_tnx=sNx} and  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
323  \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
324  releases of MITgcm are to allow varying tile sizes.  Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
325    Multiprocessing}.  Future releases of MITgcm may allow varying tile
326  The location of the tiles' Cartesian origin within a subdomain are  sizes. \\
327  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and  
328  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to  The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
329  relate the location of the edges of the tiles to each other.  As an  \varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles'
330  example, in the default six-tile topology (the degenerate case) each  Cartesian origin within a subdomain  
331  index in these arrays are set to 0.  The twenty-four, 32x32 cube face  and locate the edges of different tiles relative to each other.  As
332  case discussed above will have values of 0 or 16, depending on the  an example, in the default six-tile topology (Fig. \ref{fig:6tile})
333  quadrant the tile falls within the subdomain.  The array  each index in these arrays is set to \code{0} since a tile occupies
334  \varlink{exch2\_myFace}{exch2_myFace} contains the number of the  its entire subdomain.  The twenty-four-tile case discussed above will
335  cubeface/subdomain of each tile, numbered 1-6 in the case of the  have values of \code{0} or \code{16}, depending on the quadrant of the
336  standard cube topology.  tile within the subdomain.  The elements of the arrays
337    \varlink{exch2\_txglobalo}{exch2_txglobalo} and
 The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and  
338  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
339  \varlink{exch2\_tbasex}{exch2_tbasex} and  \varlink{exch2\_tbasex}{exch2_tbasex} and
340  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the
341  global address space, similar to that used by global files.  global address space, similar to that used by global output and input
342    files. \\
343    
344    The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
345    the subdomain of each tile, in a range \code{(1:6)} in the case of the
346    standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
347    figures \ref{fig:6tile} and
348    \ref{fig:48tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
349    contains a count of the neighboring tiles each tile has, and sets
350    the bounds for looping over neighboring tiles.
351    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
352    tile, and is used in interprocess communication.  \\
353    
354    
355  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
356  \varlink{exch2\_isEedge}{exch2_isEedge},  \varlink{exch2\_isEedge}{exch2_isEedge},
357  \varlink{exch2\_isSedge}{exch2_isSedge}, and  \varlink{exch2\_isSedge}{exch2_isSedge}, and
358  \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
359  tile lies on the edge of a subdomain, 0 if not.  The values are used  indexed tile lies on the edge of its subdomain, \code{0} if
360  within the topology generator to determine the orientation of  not.  The values are used within the topology generator to determine
361  neighboring tiles and to indicate whether a tile lies on the corner of  the orientation of neighboring tiles, and to indicate whether a tile
362  a subdomain.  The latter case indicates special exchange and numerical  lies on the corner of a subdomain.  The latter case requires special
363  handling for the singularities at the eight corners of the cube.  exchange and numerical handling for the singularities at the eight
364  \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of  corners of the cube. \\
365  how many neighboring tiles each tile has, and is used for setting  
366  bounds for looping over neighboring tiles.  
367  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each  Arrays Indexed to Tile Number and Neighbor:
368  tile, and is used in interprocess communication.  
369    The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
370  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  \code{NTILES} and describe the orientations between the the tiles. \\
371    
372  The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$  The array \code{exch2\_neighbourId(a,T)} holds the tile number
373  \texttt{NTILES} and describe the orientations between the the tiles.  \code{Tn} for each of the tile number \code{T}'s neighboring tiles
374    \code{a}.  The neighbor tiles are indexed
375  The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for  \code{(1:exch2\_nNeighbours(T))} in the order right to left on the
376  each of the $n$ neighboring tiles.  The neighbor tiles are indexed  north then south edges, and then top to bottom on the east then west
377  \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north  edges.  \\
378  then south edges, and then top to bottom on the east and west edges.  
379  Maybe throw in a fig here, eh?   The \code{exch2\_opposingSend\_record(a,T)} array holds the
380    index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}
381  The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c  that holds the tile number \code{T}, given
382  in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.  \code{Tn=exch2\_neighborId(a,T)}.  In other words,
 In other words,  
383  \begin{verbatim}  \begin{verbatim}
384     exch2_neighbourId( exch2_opposingSend_record(a,T),     exch2_neighbourId( exch2_opposingSend_record(a,T),
385                        exch2_neighbourId(a,T) ) = T                        exch2_neighbourId(a,T) ) = T
386  \end{verbatim}  \end{verbatim}
387  and this provides a back-reference from the neighbor tiles.  This provides a back-reference from the neighbor tiles. \\
388    
389  The arrays \varlink{exch2\_pi}{exch2_pi},  The arrays \varlink{exch2\_pi}{exch2_pi} and
390  \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},  \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices
391  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and  in exchanges between the neighboring tiles.  These transformations are
392  \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in  necessary in exchanges between subdomains because a horizontal dimension
393  exchanges between the neighboring tiles.  The dimensions of  in one subdomain
394  \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the  may map to other horizonal dimension in an adjacent subdomain, and
395  neighbor ID \textit{N} and the tile number \textit{T} as explained  may also have its indexing reversed. This swapping arises from the
396  above, plus the transformation vector {\em t }, of length two.  The  ``folding'' of two-dimensional arrays into a three-dimensional
397  first element of the transformation vector indicates the factor by  cube. \\
398  which variables representing the same vector component of a tile will  
399  be multiplied, and the second element indicates the transform to the  The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
400  variable in the other direction.  As an example,  are the neighbor ID \code{N} and the tile number \code{T} as explained
401  \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a  above, plus a vector of length \code{2} containing transformation
402  vector variable in tile \texttt{T} to the i-component of tile  factors \code{t}.  The first element of the transformation vector
403  \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold  holds the factor to multiply the index in the same dimension, and the
404  the component of neighbor \texttt{N}'s j-component.  second element holds the the same for the orthogonal dimension.  To
405    clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis
406  Under the current cube topology, one of the two elements of  index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor
407  \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}  \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s
408  and neighbor \texttt{N} will be 0, reflecting the fact that the vector  $x$ index to the neighbor \code{N}'s $y$ index. \\
409  components are orthogonal.  The other element will be 1 or -1,  
410  depending on whether the components are indexed in the same or  One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a
411  opposite directions.  For example, the transform dimension of the  given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting
412  arrays for all tile neighbors on the same subdomain will be [1,0],  the fact that the two axes are orthogonal.  The other element will be
413  since all tiles on the same subdomain are oriented identically.  \code{1} or \code{-1}, depending on whether the axes are indexed in
414  Vectors that correspond to the orthogonal dimension with the same  the same or opposite directions.  For example, the transform vector of
415  index direction will have [0,1], whereas those in the opposite index  the arrays for all tile neighbors on the same subdomain will be
416  direction will have [0,-1].  \code{(1,0)}, since all tiles on the same subdomain are oriented
417    identically.  An axis that corresponds to the orthogonal dimension
418    with the same index direction in a particular tile-neighbor
419    orientation will have \code{(0,1)}.  Those with the opposite index
420    direction will have \code{(0,-1)} in order to reverse the ordering. \\
421    
422    The arrays \varlink{exch2\_oi}{exch2_oi},
423    \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
424    \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
425    neighbor and specify the relative offset within the subdomain of the
426    array index of a variable going from a neighboring tile \code{N} to a
427    local tile \code{T}.  Consider \code{T=1} in the six-tile topology
428    (Fig. \ref{fig:6tile}), where
429    
 {\footnotesize  
430  \begin{verbatim}  \begin{verbatim}
431  C      exch2_pi          :: X index row of target to source permutation         exch2_oi(1,1)=33
432  C                        :: matrix for each neighbour entry.                     exch2_oi(2,1)=0
433  C      exch2_pj          :: Y index row of target to source permutation         exch2_oi(3,1)=32
434  C                        :: matrix for each neighbour entry.                     exch2_oi(4,1)=-32
 C      exch2_oi          :: X index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oj          :: Y index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oi_f        :: X index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
 C      exch2_oj_f        :: Y index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
435  \end{verbatim}  \end{verbatim}
436  }  
437    The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor,
438    which is \code{Tn=6}.  The axes of \code{T} and \code{Tn} have the
439    same orientation and their $x$ axes have the same origin, and so an
440    exchange between the two requires no changes to the $x$ index.  For
441    the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the
442    \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on
443    \code{Tn}.  The eastern edge of \code{T} shows the reverse case
444    (\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges
445    with \code{x=0} on \code{Tn=2}. \\
446    
447     The most interesting case, where \code{exch2\_oi(1,1)=33} and
448    \code{Tn=3}, involves a reversal of indices.  As in every case, the
449    offset \code{exch2\_oi} is added to the original $x$ index of \code{T}
450    multiplied by the transformation factor \code{exch2\_pi(t,N,T)}.  Here
451    \code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal
452    to the $x$ axis of \code{Tn}.  \code{exch2\_pi(2,1,1)=-1} since the
453    $x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the
454    index is reversed.  The result is that the index of the northern edge
455    of \code{T}, which runs \code{(1:32)}, is transformed to
456    \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to
457    get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn}
458    relative to \code{T}.  This transformation may seem overly convoluted
459    for the six-tile case, but it is necessary to provide a general
460    solution for various topologies. \\
461    
462    
463    
464  \subsection{Key Routines}  Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c},
465    \varlink{exch2\_ithi\_c}{exch2_ithi_c},
466    \varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and
467    \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
468    bounds of the edge segment of the neighbor tile \code{N}'s subdomain
469    that gets exchanged with the local tile \code{T}.  To take the example
470    of tile \code{T=2} in the forty-eight-tile topology
471    (Fig. \ref{fig:48tile}): \\
472    
473    \begin{verbatim}
474           exch2_itlo_c(4,2)=17
475           exch2_ithi_c(4,2)=17
476           exch2_jtlo_c(4,2)=0
477           exch2_jthi_c(4,2)=33
478    \end{verbatim}
479    
480    Here \code{N=4}, indicating the western neighbor, which is
481    \code{Tn=1}.  \code{Tn} resides on the same subdomain as \code{T}, so
482    the tiles have the same orientation and the same $x$ and $y$ axes.
483    The $x$ axis is orthogonal to the western edge and the tile is 16
484    points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c}
485    indicate the column beyond \code{Tn}'s eastern edge, in that tile's
486    halo region. Since the border of the tiles extends through the entire
487    height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
488    \code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in
489    either direction to cover part of the halo. \\
490    
491    For the north edge of the same tile \code{T=2} where \code{N=1} and
492    the neighbor tile is \code{Tn=5}:
493    
494  \subsection{References}  \begin{verbatim}
495           exch2_itlo_c(1,2)=0
496           exch2_ithi_c(1,2)=0
497           exch2_jtlo_c(1,2)=0
498           exch2_jthi_c(1,2)=17
499    \end{verbatim}
500    
501    \code{T}'s northern edge is parallel to the $x$ axis, but since
502    \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s
503    northern edge exchanges with \code{Tn}'s western edge.  The western
504    edge of the tiles corresponds to the lower bound of the $x$ axis, so
505    \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the
506    western halo region of \code{Tn}. The range of
507    \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
508    width of \code{T}'s northern edge, expanded by one into the halo. \\
509    
510    
511    \subsubsection{Key Routines}
512    
513    Most of the subroutines particular to exch2 handle the exchanges
514    themselves and are of the same format as those described in
515    \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere
516    communication}.  Like the original routines, they are written as
517    templates which the local Makefile converts from \code{RX} into
518    \code{RL} and \code{RS} forms. \\
519    
520    The interfaces with the core model subroutines are
521    \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and
522    \code{EXCH\_XY\_RX}.  They override the standard exchange routines
523    when \code{genmake2} is run with \code{exch2} option.  They in turn
524    call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and
525    \code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector
526    quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two
527    and three-dimensional scalar quantities.  These subroutines set the
528    dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE}
529    for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle
530    the singularities at the cube corners. \\
531    
532    The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and
533    \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine
534    needs to pass both the $u$ and $v$ components of the physical vectors.
535    This swapping arises from the topological folding discussed above, where the
536    $x$ and $y$ axes get swapped in some cases, and is not an
537    issue with the scalar case. These subroutines call
538    \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
539    the work using the variables discussed above. \\
540    
541    \subsubsection{Experiments and tutorials that use exch2}
542    \label{sec:pkg:exch2:experiments}
543    
544    \begin{itemize}
545    \item{Held Suarez tutorial, in tutorial\_held\_suarez\_cs verification directory,
546    described in section \ref{sect:eg-hs} }
547    \end{itemize}
Legend:
Removed from v.1.10  
changed lines
  Added in v.1.27
  ViewVC Help
Powered by ViewVC 1.1.22