4.1 Command Line Options

A CHARM++ program accepts the following command line options:

+pN

Run the program with N processors. The default is 1.

+ss

Print summary statistics about chare creation. This option prints the total number of chare creation requests, and the total number of chare creation requests processed across all processors.

+cs

Print statistics about the number of create chare messages requested and processed, the number of messages for chares requested and processed, and the number of messages for branch office chares requested and processed, on a per processor basis. Note that the number of messages created and processed for a particular type of message on a given node may not be the same, since a message may be processed by a different processor from the one originating the request.

user_options

Options that are be interpreted by the user program may be included mixed with the system options. However, user_options cannot start with +. The user_options will be passed as arguments to the user program via the usual argc/argv construct to the main entry point of the main chare. CHARM++ system options will not appear in argc/argv.

4.1.1 Additional Network Options

The following ++ command line options are available in the network version:

++local

Run charm program only on local machines. No remote shell invocation is needed in this case. It starts node programs right on your local machine. This could be useful if you just want to run small program on only one machine, for example, your laptop.

++mpiexec

Use the cluster's mpiexec job launcher instead of the built in rsh/ssh method.

This will pass -n $P to indicate how many processes to launch. An executable named something other than mpiexec can be used with the additional argument ++remote-shell runmpi, with `runmpi' replaced by the necessary name.

Use of this option can potentially provide a few benefits:

• Faster startup compared to the SSH/RSH approach charmrun would otherwise use
• No need to generate a nodelist file
• Multi-node job startup on clusters that do not allow connections from the head/login nodes to the compute nodes

At present, this option depends on the environment variables for some common MPI implementations. It supports OpenMPI (OMPI_COMM_WORLD_RANK and OMPI_COMM_WORLD_SIZE) and M(VA)PICH (MPIRUN_RANK and MPIRUN_NPROCS or PMI_RANK and PMI_SIZE).

++debug

Run each node under gdb in an xterm window, prompting the user to begin execution.

++debug-no-pause

Run each node under gdb in an xterm window immediately (i.e. without prompting the user to begin execution).

If using one of the ++debug or ++debug-no-pause options, the user must ensure the following:

1. The DISPLAY environment variable points to your terminal. SSH's X11 forwarding does not work properly with CHARM++.

2. The nodes must be authorized to create windows on the host machine (see man pages for xhost and xauth).

3. xterm, xdpyinfo, and gdb must be in the user's path.

4. The path must be set in the .cshrc file, not the .login file, because rsh does not run the .login file.

++maxrsh

Maximum number of rsh's to run at a time.

++nodelist

File containing list of nodes.

++ppn

number of pes per node

++help

print help messages

++runscript

script to run node-program with

++xterm

which xterm to use

++in-xterm

Run each node in an xterm window

++display

X Display for xterm

++debugger

which debugger to use

++remote-shell

which remote shell to use

++useip

Use IP address provided for charmrun IP

++usehostname

Send nodes our symbolic hostname instead of IP address

++server-auth

CCS Authentication file

++server-port

Port to listen for CCS requests

++server

Enable client-server (CCS) mode

++nodegroup

which group of nodes to use

++verbose

Print diagnostic messages

++timeout

seconds to wait per host connection

++p

number of processes to create

4.1.2 Multicore Options

On multicore platforms, operating systems (by default) are free to move processes and threads among cores to balance load. This however sometimes can degrade the performance of Charm++ applications due to the extra overhead of moving processes and threads, especailly when Charm++ applications has already implemented its own dynamic load balancing.

Charm++ provides the following runtime options to set the processor affinity automatically so that processes or threads no longer move. When cpu affinity is supported by an operating system (tested at Charm++ configuration time), same runtime options can be used for all flavors of Charm++ versions including network and MPI versions, smp and non-smp versions.

+setcpuaffinity

set cpu affinity automatically for processes (when Charm++ is based on non-smp versions) or threads (when smp)

+excludecore <core #>

does not set cpu affinity for the given core number. One can use this option multiple times to provide a list of core numbers to avoid.

+pemap L[-U[:S[.R]

][,...]] Bind the execution threads to the sequence of cores described by the arguments using the operating system's CPU affinity functions.

A single number identifies a particular core. Two numbers separated by a dash identify an inclusive range (lower bound and upper bound). If they are followed by a colon and another number (a stride), that range will be stepped through in increments of the additional number. Within each stride, a dot followed by a run will indicate how many cores to use from that starting point.

For example, the sequence 0-8:2,16,20-24 includes cores 0, 2, 4, 6, 8, 16, 20, 21, 22, 23, 24. On a 4-way quad-core system, if one wanted to use 3 cores from each socket, one could write this as 0-15:4.3.

+commap p[,q,...]

Bind communication threads to the listed cores, one per process.