/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.6 by afe, Tue Feb 3 19:43:38 2004 UTC revision 1.13 by afe, Wed Mar 17 19:49:22 2004 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere Exchange}  \section{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The exch2 package is an extension to the original cubed sphere exchanges  The \texttt{exch2} package extends the original cubed
20  to allow more flexible domain decomposition and parallelization.  Cube faces  sphere topology configuration to allow more flexible domain
21  (subdomains) may be divided into whatever number of tiles that divide evenly  decomposition and parallelization.  Cube faces (also called
22  into the grid point dimensions of the subdomain.  Furthermore, the individual  subdomains) may be divided into any number of tiles that divide evenly
23  tiles may be run on separate processors in different combinations,  into the grid point dimensions of the subdomain.  Furthermore, the
24  and whether exchanges between particular tiles occur between different  individual tiles can run on separate processors in different
25  processors is determined at runtime.  combinations, and whether exchanges between particular tiles occur
26    between different processors is determined at runtime.  This
27  The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and  flexibility provides for manual compile-time load balancing across a
28  assigned in {\em w2\_e2setup.F}, both in the  relatively arbitrary number of processors. \\
29  {\em pkg/exch2} directory.  The validity of the cube topology depends  
30  on the {\em SIZE.h} file as detailed below.  Both files are generated by  The exchange parameters are declared in
31  Matlab scripts and  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32  should not be edited.  The default files provided in the release set up  and assigned in
33  a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34  points, running on a single processor.    validity of the cube topology depends on the \file{SIZE.h} file as
35    detailed below.  The default files provided in the release configure a
36    cubed sphere topology of six tiles, one per subdomain, each with
37    32$\times$32 grid points, all running on a single processor.  Both
38    files are generated by Matlab scripts in
39    \file{utils/exch2/matlab-topology-generator}; see Section
40    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
41    for details on creating alternate topologies.  Pregenerated examples
42    of these files with alternate topologies are provided under
43    \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
44    file for single-processor execution.
45    
46    \subsection{Invoking exch2}
47    
48    To use exch2 with the cubed sphere, the following conditions must be
49    met: \\
50    
51    $\bullet$ The exch2 package is included when \file{genmake2} is run.
52      The easiest way to do this is to add the line \code{exch2} to the
53      \file{profile.conf} file -- see Section
54      \ref{sect:buildingCode} \sectiontitle{Building the code} for general
55      details. \\
56    
57    $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
58      \file{w2\_e2setup.F} must reside in a directory containing code
59      linked when \file{genmake2} runs.  The safest place to put these
60      is the directory indicated in the \code{-mods=DIR} command line
61      modifier (typically \file{../code}), or the build directory.  The
62      default versions of these files reside in \file{pkg/exch2} and are
63      linked automatically if no other versions exist elsewhere in the
64      link path, but they should be left untouched to avoid breaking
65      configurations other than the one you intend to modify.\\
66    
67    $\bullet$ Files containing grid parameters, named
68      \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),
69      must be in the working directory when the MITgcm executable is run.
70      These files are provided in the example experiments for cubed sphere
71      configurations with 32$\times$32 cube sides and are non-trivial to
72      generate -- please contact MITgcm support if you want to generate
73      files for other configurations. \\
74    
75    $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
76      be placed where \file{genmake2} will find it.  In particular for
77      exch2, the domain decomposition specified in \file{SIZE.h} must
78      correspond with the particular configuration's topology specified in
79      \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
80      decomposition issues particular to exch2 are addressed in Section
81      \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
82      and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and MPI}; a more
83      general background on the subject relevant to MITgcm is presented in
84      Section \ref{sect:specifying_a_decomposition}
85      \sectiontitle{Specifying a decomposition}.\\
86    
87    As of the time of writing the following examples use exch2 and may be
88    used for guidance:
89    
90    \begin{verbatim}
91    verification/adjust_nlfs.cs-32x32x1
92    verification/adjustment.cs-32x32x1
93    verification/aim.5l_cs
94    verification/global_ocean.cs32x15
95    verification/hs94.cs-32x32x5
96    \end{verbatim}
97    
98    
99    
100    
101    \subsection{Generating Topology Files for exch2}
102    \label{sec:topogen}
103    
104    Alternate cubed sphere topologies may be created using the Matlab
105    scripts in \file{utils/exch2/matlab-topology-generator}. Running the
106    m-file
107    \filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m}
108    from the Matlab prompt (there are no parameters to pass) generates
109    exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
110    \file{w2\_e2setup.F} in the working directory and displays a figure of
111    the topology via Matlab.  The other m-files in the directory are
112    subroutines of \file{driver.m} and should not be run ``bare'' except
113    for development purposes. \\
114    
115    The parameters that determine the dimensions and topology of the
116    generated configuration are \code{nr}, \code{nb}, \code{ng},
117    \code{tnx} and \code{tny}, and all are assigned early in the script. \\
118    
119    The first three determine the size of the subdomains and
120    hence the size of the overall domain.  Each one determines the number
121    of grid points, and therefore the resolution, along the subdomain
122    sides in a ``great circle'' around an axis of the cube.  At the time
123    of this writing MITgcm requires these three parameters to be equal,
124    but they provide for future releases  to accomodate different
125    resolutions around the axes to allow (for example) greater resolution
126    around the equator.\\
127    
128    The parameters \code{tnx} and \code{tny} determine the dimensions of
129    the tiles into which the subdomains are decomposed, and must evenly
130    divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
131    The result is a rectangular tiling of the subdomain.  Figure
132    \ref{fig:24tile} shows one possible topology for a twentyfour-tile
133    cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
134    
135    \begin{figure}
136    \begin{center}
137     \resizebox{4in}{!}{
138      \includegraphics{part6/s24t_16x16.ps}
139     }
140    \end{center}
141    
142    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
143    divided into six 32$\times$32 subdomains, each of which is divided into four tiles
144    (\code{tnx=16, tny=16}) for a total of twentyfour tiles.
145    } \label{fig:24tile}
146    \end{figure}
147    
148    \begin{figure}
149    \begin{center}
150     \resizebox{4in}{!}{
151      \includegraphics{part6/s12t_16x32.ps}
152     }
153    \end{center}
154    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
155    divided into six 32$\times$32 subdomains of two tiles each
156     (\code{tnx=16, tny=32}).
157    } \label{fig:12tile}
158    \end{figure}
159    
160    \begin{figure}
161    \begin{center}
162     \resizebox{4in}{!}{
163      \includegraphics{part6/s6t_32x32.ps}
164     }
165    \end{center}
166    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
167    divided into six 32$\times$32 subdomains with one tile each
168    (\code{tnx=32, tny=32}).  This is the default configuration.
169      }
170    \label{fig:6tile}
171    \end{figure}
172    
173    
174    Tiles can be selected from the topology to be omitted from being
175    allocated memory and processors.  This tuning is useful in ocean
176    modeling for omitting tiles that fall entirely on land.  The tiles
177    omitted are specified in the file
178    \filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt}
179    by their tile number in the topology, separated by a newline. \\
180    
181    
182    
183    
184    \subsection{exch2, SIZE.h, and multiprocessing}
185    \label{sec:exch2mpi}
186    
187    Once the topology configuration files are created, the Fortran
188    \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
189    Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
190    a decomposition} provides a general description of domain
191    decomposition within MITgcm and its relation to \file{SIZE.h}. The
192    current section specifies certain constraints the exch2 package
193    imposes as well as describes how to enable parallel execution with
194    MPI. \\
195    
196    As in the general case, the parameters \varlink{sNx}{sNx} and
197    \varlink{sNy}{sNy} define the size of the individual tiles, and so
198    must be assigned the same respective values as \code{tnx} and
199    \code{tny} in \file{driver.m}.\\
200    
201    The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
202    have no special bearing on exch2 and may be assigned as in the general
203    case. The same holds for \varlink{Nr}{Nr}, the number of vertical
204    levels in the model.\\
205    
206    The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
207    \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
208    tiles and how they are distributed on processors.  When using exch2,
209    the tiles are stored in single dimension, and so
210    \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
211    configured by exch2 cannot be split up accross processors without
212    regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
213    
214    The number of tiles MITgcm allocates and how they are distributed
215    between processors depends on \varlink{nPx}{nPx} and
216    \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
217    processor and \varlink{nPx}{nPx} the number of processors.  The total
218    number of tiles in the topology minus those listed in
219    \file{blanklist.txt} must equal \code{nSx*nPx}. \\
220    
221    The following is an example of \file{SIZE.h} for the twelve-tile
222    configuration illustrated in figure \ref{fig:12tile} running on
223    one processor: \\
224    
225    \begin{verbatim}
226          PARAMETER (
227         &           sNx =  16,
228         &           sNy =  32,
229         &           OLx =   2,
230         &           OLy =   2,
231         &           nSx =  12,
232         &           nSy =   1,
233         &           nPx =   1,
234         &           nPy =   1,
235         &           Nx  = sNx*nSx*nPx,
236         &           Ny  = sNy*nSy*nPy,
237         &           Nr  =   5)
238    \end{verbatim}
239    
240    The following is an example for the twentyfour-tile topology in figure
241    \ref{fig:24tile} running on six processors:
242    
243    \begin{verbatim}
244          PARAMETER (
245         &           sNx =  16,
246         &           sNy =  16,
247         &           OLx =   2,
248         &           OLy =   2,
249         &           nSx =   4,
250         &           nSy =   1,
251         &           nPx =   6,
252         &           nPy =   1,
253         &           Nx  = sNx*nSx*nPx,
254         &           Ny  = sNy*nSy*nPy,
255         &           Nr  =   5)
256    \end{verbatim}
257    
258    
259    
260    
261    
262  \subsection{Key Variables}  \subsection{Key Variables}
263    
264  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
265  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and three
266  dimensional  dimensional arrays indexed to tile number and neighboring tile.  This
267  arrays indexed to tile number and neighboring tile.  This division  division reflects the functionality of these variables: The
268  actually reflects  the functionality of these variables: the scalars  scalars are common to every part of the topology, the tile-indexed
269  are common to every part of the topology, the tile-indexed arrays to  arrays to individual tiles, and the arrays indexed by tile and
270  individual tiles, and the arrays indexed to tile and neighbor to  neighbor to relationships between tiles and their neighbors. \\
 relationships between tiles and their neighbors.  
271    
272  \subsubsection{Scalars}  \subsubsection{Scalars}
273    
274  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
275  {\em NTILES}, and the maximum number of neighbors of any tiles by  \code{NTILES}, and the maximum number of neighbors of any tiles by
276  {\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of  \code{MAX\_NEIGHBOURS}.  These parameters are used for defining the
277  the various one and two dimensional arrays that store tile parameters  size of the various one and two dimensional arrays that store tile
278  indexed to the tile number.  parameters indexed to the tile number and are assigned in the files
279    generated by \file{driver.m}.\\
280  The scalar parameters {\em exch2\_domain\_nxt} and  
281  {\em exch2\_domain\_nyt} express the number of tiles in the x and y global  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
282  indices.  For example, the default setup of six tiles has  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
283  {\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of  of tiles in the $x$ and $y$ global indices.  For example, the default
284  twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will  setup of six tiles (Fig. \ref{fig:6tile}) has \code{exch2\_domain\_nxt=6} and
285  have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note  \code{exch2\_domain\_nyt=1}.  A topology of twenty-four square tiles,
286  that these parameters express the tile layout to allow global data files that  four per subdomain (as in figure \ref{fig:24tile}), will have
287  are tile-layout-neutral and have no bearing on the internal storage of the  \code{exch2\_domain\_nxt=12} and \code{exch2\_domain\_nyt=2}.  Note
288  arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the  that these parameters express the tile layout to allow global data
289  x axis) and y-axis variable {\em bj} is generally ignored within the package.  files that are tile-layout-neutral and have no bearing on the internal
290    storage of the arrays.  The tiles are internally stored in a range
291    from \code{(1:\varlink{bi}{bi})} the $x$ axis, and $y$ axis variable
292    \varlink{bj}{bj} is generally ignored within the package. \\
293    
294  \subsubsection{Arrays Indexed to Tile Number}  \subsubsection{Arrays Indexed to Tile Number}
295    
296  The following arrays are of size {\em NTILES}, are indexed to the tile number,  The following arrays are of length \code{NTILES}, are indexed to the
297  and the indices are omitted in their descriptions.  tile number, and the indices are omitted in their descriptions. \\
   
 The arrays {\em exch2\_tnx} and {\em exch2\_tny}  
 express the x and y dimensions of each tile.  At present for each tile  
 {\em exch2\_tnx = sNx}  
 and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of  
 MITgcm are to allow varying tile sizes.  
   
 The location of the tiles' Cartesian origin within a subdomain are determined  
 by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These variables  
 are used to relate the location of the edges of the tiles to each other.  As  
 an example, in the default six-tile topology (the degenerate case)  
 each index in these arrays are  
 set to 0.  The twenty-four, 32x32 cube face case discussed above will have  
 values of 0 or 16, depending on the quadrant the tile falls within the  
 subdomain.  {\em exch2\_myFace} contains the number of the  
 cubeface/subdomain of each tile, numbered 1-6 in the case of the standard  
 cube topology.    
   
 The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to  
 {\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within  
 the global address space, similar to that used by global files.    
   
 The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge},  
 and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge  
 of a subdomain, 0 if not.  The values are used within the topology generator  
 to determine the orientation of neighboring tiles and to indicate whether  
 a tile lies on the corner of a subdomain.  The latter case indicates  
 special exchange and numerical handling for the singularities at the eight  
 corners of the cube.  {\em exch2\_isNedge} contains a count of how many  
 neighboring tiles each tile has, and is used for setting bounds for looping  
 over neighboring tiles.  {\em exch2\_tProc} holds the process rank of each tile,  
 and is used in interprocess communication.  
298    
299  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
300    \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
301    each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
302    \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
303    section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
304    multiprocessing}.  Future releases of MITgcm are to allow varying tile
305    sizes. \\
306    
307    The location of the tiles' Cartesian origin within a subdomain are
308    determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
309    \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
310    relate the location of the edges of different tiles to each other.  As
311    an example, in the default six-tile topology (Fig. \ref{fig:6tile})
312    each index in these arrays is set to \code{0} since a tile occupies
313    its entire subdomain.  The twentyfour-tile case discussed above will
314    have values of \code{0} or \code{16}, depending on the quadrant the
315    tile falls within the subdomain.  The elements of the arrays
316    \varlink{exch2\_txglobalo}{exch2_txglobalo} and
317    \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
318    \varlink{exch2\_tbasex}{exch2_tbasex} and
319    \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
320    global address space, similar to that used by global files. \\
321    
322    The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
323    the subdomain of each tile, in a range \code{(1:6)} in the case of the
324    standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
325    figures \ref{fig:12tile} and
326    \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
327    contains a count of how many neighboring tiles each tile has, and is
328    used for setting bounds for looping over neighboring tiles.
329    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
330    tile, and is used in interprocess communication.  \\
331    
332    
333    The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
334    \varlink{exch2\_isEedge}{exch2_isEedge},
335    \varlink{exch2\_isSedge}{exch2_isSedge}, and
336    \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
337    indexed tile lies on the edge of a subdomain, \code{0} if not.  The
338    values are used within the topology generator to determine the
339    orientation of neighboring tiles, and to indicate whether a tile lies
340    on the corner of a subdomain.  The latter case requires special
341    exchange and numerical handling for the singularities at the eight
342    corners of the cube. \\
343    
 The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and  
 describe the orientations between the the tiles.  
344    
345  The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
 {\em T}'s neighbor tile {\em a}, and {\em exch2\_opposingSend\_record(a,T)} holds  
 the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.  
 In other words,  
346    
347  \begin{verbatim}    The following arrays are all of size
348  exch2_neighbourId( exch2_opposingSend_record(a,T), exch2_neighbourId(a,T) ) = T  \code{MAX\_NEIGHBOURS}$\times$\code{NTILES} and describe the
349    orientations between the the tiles. \\
350    
351    The array \code{exch2\_neighbourId(a,T)} holds the tile number
352    \code{Tn} for each of the tile number \code{T}'s neighboring tiles
353    \code{a}.  The neighbor tiles are indexed \code{(1:MAX\_NEIGHBOURS)}
354    in the order right to left on the north then south edges, and then top
355    to bottom on the east and west edges.  Maybe throw in a fig here, eh?
356    \\
357    
358    \sloppy
359    The \code{exch2\_opposingSend\_record(a,T)} array holds the index
360    \code{b} in \texttt{exch2\_neighbourId(b,Tn)} that holds the tile
361    number \code{T}.  In other words,
362    \begin{verbatim}
363       exch2_neighbourId( exch2_opposingSend_record(a,T),
364                          exch2_neighbourId(a,T) ) = T
365  \end{verbatim}  \end{verbatim}
366    This provides a back-reference from the neighbor tiles. \\
367    
368  % {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}.  The arrays \varlink{exch2\_pi}{exch2_pi} and
369  % alternate version  \varlink{exch2\_pj}{exch2_pj} specify the transformations of variables
370    in exchanges between the neighboring tiles.  These transformations are
371    necessary in exchanges between subdomains because a physical vector
372    component in one direction may map to one in a different direction in
373    an adjacent subdomain, and may be have its indexing reversed. This
374    swapping arises from the ``folding'' of two-dimensional arrays into a
375    three-dimensional cube.
376    
377    The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
378    are the neighbor ID \code{N} and the tile number \code{T} as explained
379    above, plus a vector of length 2 containing transformation factors
380    \code{t}.  The first element of the transformation vector indicates
381    the factor \code{t} by which variables representing the same
382    \emph{physical} vector component of a tile \code{T} will be multiplied
383    in exchanges with neighbor \code{N}, and the second element indicates
384    the transform to the physical vector in the other direction.  To
385    clarify (hopefully), \code{exch2\_pi(1,N,T)} holds the transform of
386    the $i$ component of a vector variable in tile \code{T} to the $i$
387    component of tile \code{T}'s neighbor \code{N}, and
388    \code{exch2\_pi(2,N,T)} holds the transform of \code{T}'s $i$
389    components to the neighbor \code{N}'s $j$ component. \\
390    
391    Under the current cube topology, one of the two elements of
392    \code{exch2\_pi} or \code{exch2\_pj} for a given tile \code{T} and
393    neighbor \code{N} will be \code{0}, reflecting the fact that the two
394    vector components are orthogonal.  The other element will be \code{1}
395    or \code{-1}, depending on whether the components are indexed in the
396    same or opposite directions.  For example, the transform vector of the
397    arrays for all tile neighbors on the same subdomain will be
398    \code{(1,0)}, since all tiles on the same subdomain are oriented
399    identically.  A vector direction that corresponds to the orthogonal
400    dimension with the same index direction in a particular tile-neighbor
401    orientation will have \code{(0,1)}, whereas those in the opposite
402    index direction will have \code{(0,-1)}. \\
403    
404    
405    \varlink{exch2\_oi}{exch2_oi},
406    \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
407    \varlink{exch2\_oj\_f}{exch2_oj_f}
408    
 This is to provide a backreference from the neighbor tiles.  
409    
410    
 //  
   
 \begin{verbatim}  
411    
412    This needs some diagrams. \\
413    
414    
415    {\footnotesize
416    \begin{verbatim}
417  C      exch2_pi          :: X index row of target to source permutation  C      exch2_pi          :: X index row of target to source permutation
418  C                        :: matrix for each neighbour entry.              C                        :: matrix for each neighbour entry.            
419  C      exch2_pj          :: Y index row of target to source permutation  C      exch2_pj          :: Y index row of target to source permutation
# Line 143  C      exch2_oj_f        :: Y index elem Line 431  C      exch2_oj_f        :: Y index elem
431  C                        :: offset vector for face quantities            C                        :: offset vector for face quantities          
432  C                        :: of each neighbor entry.                      C                        :: of each neighbor entry.                    
433  \end{verbatim}  \end{verbatim}
434    }
435    
436    
437    
Legend:
Removed from v.1.6  
changed lines
  Added in v.1.13
  ViewVC Help
Powered by ViewVC 1.1.22