home news images pubs c3dTeam
whatIsCart3D?
surfaceModeling
meshGen?
flowSolvers?
flowCart
overview
reorder
mgPrep
running
Input/Output files
postprocess?
mailList?
betaTest?
licensing?

   Running flowCart


Input/Output files
Command line options and usage
Multi-processor running (shared memory and MPI)
Flux Function Options:
Robust Mode
Limiter Choice
How do I restart a calculation?
More information on flowCart I/O

 
Input/Output files:
flowCart requires an input.cntl file, a mesh, and a Mesh.c3d.Info file to get running. If you want to map the solution back to the surface triangulation, then you'll also need to provide the configuration triangulation file which you used to create the mesh (in cubes).  Meshes can be any one of the following:
 
File type - default name
Contents
Platform and solution approach
Mesh.c3d Unordered mesh from cubes Single CPU, no multigrid or grid sequencing
Mesh.R.c3d Reordered mesh ready for partitioning Multiple CPU, no multigrid or grid sequencing
Mesh.mg.c3d Reordered, multilevel mesh file Multiple CPU, with multigrid or grid sequencing

Most often, you're feeding flowCart a "Mesh.mg.c3d" mesh. For more information on flowCart I/O command line options and files, click here.  If you're using the new (Cart3D v1.4)  "$__Force_Moment_Processing" section in input.cntl you will also have to have a GMP-style Config.xml file, but flowCart will automatically produce one of these if the only component you're interested in is "entire".

   Command line options and Usage:

Usage:

 % flowCart -
 
   flowCart - 
    Usage: flowCart [ argument list ]
 Options:

          -- Runtime Options--
   -N %d         Max Number of MultiGrid cycles to advance
   -choice %d    1=do_mg_new, 2=jameson, 3=orig_do_mg_cycle
   -mg %d        Number of multigrid levels
   -gs %d        Use grid sequencing on # levels for startup (no mg)
   -cfl   %f     CFL number
   -rampUp %f    ramp up factor from .01*CFL, default: off
   -tm  %f       cut-cell Grad mod (1stOrd=0. -> 1.0=2ndOrd) def: 1.0
   -limiter %d   0=None, 1=BJ, 2=VanLeer, 3=SinLim, 4=VanAlbada, 5=MinLim
   -buffLim      Buffer limiters to smear shocks, Default: <false>
   -flux  %d     Flux function 0=VanLeer, 1=VanLeer-Hanel, 2=Colella, 3=HLLC)
   -no_fmg       no full multigrid (start MG at finestLevel)
   -subcell      Use subcell resolution on finest mesh
   -gr           Use grads in MGrestrict (auto ON if gradEval all stages)
   -nPart %d     Number of Sub Domains for partitioning
   -order %d     SubDomain ordering, 0=No Reordering, 1=RCM, 2=MLD
   -y_is_spanwise Default assumes z_is_spanwise direction

          -- I/O Options --
   -v           verbose mode ON
   -mem         Report memory usage (auto on with -v)
   -i %s        Input  file name, default:<input.cntl>
   -T           Dump surf triangulation in Tecplot format <surfName.dat>
   -clic        Dump surf trianglulation in Clic format <surfName.triq>
   -his         write history files <history.dat,forces.dat>
   -binaryIO    Write postprocessing data in binary (plotfiles etc.)
   -Xcut %d     Num of X=const cut planes <disjointCutPlanes.dat>
   -Ycut %d     Num of Y=const cut planes <disjointCutPlanes.dat>
   -Zcut %d     Num of Z=const cut planes <disjointCutPlanes.dat>
   -version     Dump version info and exit
   -Dmatrix     Dump i,j formatted connectivity Matrices
   -Dcut        Dump tecplottable file <cutcells.dat>
    
   -no_ckpt     Suppress checkpointing
   -restart     Restart into any # of partitions <Restart.file>
   -Derr        Output tau error estimates for adapt
top

  Multi-processor running:
 
flowCart uses OpenMP or MPI for explicit communication between subdomains in multi-CPU/multi-core runs. This means that you can run it on any machine or cluster which supports either communication protocol. Since it explicitly decomposes the domain, it achieves extremely good scalability as is typical of message passing codes. to the right is a plot of showing parallel speedup (current Oct 2003) for a typical case on about 4.7M cells. Here is a more recent comparison on NASA's Columbia system
More  information on the programming paradigm and details of the parallelization are availible in:
scalability OpenMP & MPI

  • To run on a multi-cpu or multi-core system with OpenMP (shared memory) follow these steps (csh): 
1% setenv OMP_NUM_THREADS nProc #  ...nProc is the integer 
#    number of CPU's you want
2% flowCart -[flowCart Options]
# ...run the executable, flowCart will interrogate 
#     the environmetn and set the number of subdomains
#     to nProc automatically

  • To run on a multi-cpu or multi-core system with MPI do the following (csh):
1% mpiexec -np nProc flowCart -[flowCart Options] #  ...nProc is the integer 
#    number of CPU's you want

Note:
We frequently get questions about MPI running. mpi_flowCart is built using mpich2 and we distribute both static and dynamiclly linked executables. There often turn out to be basic issues with the underlying mpich2 installation. If you're new to MPI, there are a couple of things you should start with before jumping in...
  1. Run flowCart (shared memory) on more than 1 thread using OpenMP.
  2. Does "% mpiexec -np 2 /bin/date" return 2 copies of the date?
  3. Can you run any (other) mpi code?
  4. Can you run any of the mpich2 demos?

(top)
  Flux Function Options:
flowCart is designed to allow user-selectable flux functions. Currently, van Leer, Colella'89 and HLLC (beta-test) are availible. Of these van Leer ("-flux 0") and van Leer-Hänel ("-flux 1") are the most well excercised, and since they're positive flux functions, they're certain to be the most robust. Most users should choose "-flux 0" (van Leer) as their default. You can specify the flu function on the command line using "-flux 0" or by using the  FluxFun tag in the $_Solver_Control_Information category of the input control file and set it to "0". Flux functions are ordered roughly in order of decreasing smearing of contact discontinuities. HLLC gets an exact contact speed, while  this is the classic achillies heel of van Leer. Take a 2D airfoil (like the naca0012 in $CART3D/cases/samples/naca0012) and experiment.

top

  Robust Mode:

Starting with verions v1.3, flowCart has a new "robust mode" of operation. In general, the code is very robust running with the standard Runge-Kutta (RK) scheme sketched in the sample input.cntl file, This scheme works great most of the time, but its a compromise between robustness and speed. As is clear from the RK tags, typically you can get away with only evaluating the gradient and limiters at the first stage of the scheme, saving  this work on subsequent stages. In taking this shortcut, we give up some of the remarkable positivity property of some flux-vector-splittings. By being more careful - evaluating the gradient at every stage in the RK scheme, and re-evaluating it during multigrid restrictions, the scheme can be made extremely robust.
To invoke "robust mode" simply set up your RK scheme in the input.cntl file to evaluate the gradient (and limiter) at every stage in the RK, like this:
Entries in $__Solver_Control_Information block of "input.cntl" to invoke "robust mode"
RK    0.0695      1
RK    0.1602      1
RK    0.2898      1
RK    0.5060      1 
RK    1.0         1

Use "robust mode" sparingly. Nearly all calculations will run with the standard scheme (which is about 50% cheaper). Virtually all the simulations that you'll find on this website were run with the standard scheme.  
top

  Limiter Choice:

Starting with Cart3D_v1.3 the limiters have been completely re-coded to be better behaved than in earlier distributions. They are now linearity preserving regardless of mesh stretching, and there is a choice of 5 of them. Having said that, you should almost always use limiter 2 (van Leer). This is the most aggressive smooth limiter and is not excessively dissipative - especially in the k-exact implementation in flowCart. Furthermore, its among the fastest to compute. The limiter choices are (in order of increasing dissipation): (0) no limiter, (1) Barth-Jespersen (2) van Leer, (3) sin limiter, (4) van Albada (5) Minmod. To understand the differences between them,
Look at the figure below. The plot on the left shows the limiter value (0-1) vs. the normalized acceleration or deceleration of the data. (sketch on the left). Phi = 1 means "no limiting". The parameter f is equivalently either del+/delC or del-/delC. The Barth-Jespersen (BJ) limiter essentially follows the monotonicity boundary for a 2nd order monotone scheme, so you can think of this as an effective "upper bound" for the limiter value. More aggressive, and you're not TVD. Below this curve, and you're limiting more than is strictly necessary. All the limiters go through Phi=0 at f=0.5. This says that they recommend no limiting when the data is linear, which is necessary for a 2nd order scheme.
Relative dissipation of limiters
Basically, we have BJ as the most aggressive, and minmod as the most dissipative. Everybody else falls in between. This graph helps you understand how much dissipation you're buying when you choose something other than "-limiter 1" on the command line, or in input.cntl. About the only time you should even consider something else is for high mach number cases since the more aggressive limiters may "over compress" the shocks leading to excessive "staircasing".
I've made a small quicktime movie comparing the "sin limiter" to Barth-Jespersen. (The "sin limiter" is in green, and BJ is shown with the red line.) Notice how, in an effort to be smooth, the sin limiter retards the slope  pre-emptively as  u_i approaches either of its neighbors. Van Albada is even more dissipative, and you can see that it gives up its slope substantially more quickly than sin(). BJ, or course, holds onto its slope as long as it can without overshoots, but the transitions are abrupt.
(top)

  How do I restart a calculation?
    You can restart a calculation using the parameters in your input.cntl file, and your can override most of these, if you'd like, using command line options (preferred). Both are scanned upon restart.

    1.  When you run a flowCart simulation, a checkpoint file is automatically produced (you can suppress this with the "-no_ckpt" command line option, if you konw you dont want to restart).  This checkpoint file is automatically named using the scheme ckPtFileName = check.[nCycles], so if you ran 150 cycles the restart file would be named "check.00150". You can use this file to restart the calculation by soft-linking the special name "Restart.file" to the checkpoint file, and invoking the "-restart" command line flag. e.g.

       
      %ln -s check.00150 Restart.file # create a softlink
      %flowCart -restart -N 200 # run flowCart an additional 50 cycles
    2.  When you restart a computation, you are free to change both the number of partitions or the number of levels of multigrid that you're using. To change the number of partitions, just reset your OMP_NUM_THREADS environment variable in OpenMP, (or the "% mpiexec -np %d" input in MPI), and flowCart will check it when it begins. To change the number of multigrid levels, use the -mg %d flag on the command line. 
(top)

last update Dec. 2008, M. Aftosmis