Charm++ Version 6.2

       Copyright (C) 1989-2010 Regents of the University of Illinois

INTRODUCTION
============

Charm++ is a message-passing parallel language and runtime system.
It is implemented as a set of libraries for C++, is efficient,
and is portable to a wide variety of parallel machines.
Source code is provided, and non-commercial use is free.


GETTING THE LATEST CHARM SOURCE
===============================

Charm's devleopment is tracked in a publicly accessible Git repository
at git://charm.cs.uiuc.edu/charm.git. The development branch is named
`charm', and stable releases are tagged `charm-version'
(e.g. `charm-6.2.0').

To obtain a copy of the repository,

      git clone git://charm.cs.uiuc.edu/charm.git

To switch from the default development branch to a release,

      git checkout charm-6.2.0

We also maintain an anonymous CVS front-end to the Git repository,
accessible as follows:

      cvs -d ':pserver:anonymous@charm.cs.uiuc.edu:12401/charm.git' co charm

The trailing `charm' specifies the branch or tag to obtain.

PICKING A VERSION
=================

First, you need to decide which version of charm++ to use. The "build" 
script in charm source directory takes several command line options to
compile Charm++. The command line syntax is:

build <target> <version> [options ...] [--basedir=dir] [--libdir=dir] [--incdir=dir] [charmc-options ...]

for detailed help messages, use -h or --help to the build script, i.e.
./build --help

REQUIRED:
---------
<target> specifies the parts of Charm++ to compile.  The most often used 
  <target> is "charm++", which will compile the key Charm++ executables and 
  runtime libraries.  Other common targets are "AMPI" and "FEM".
<versions> defines the CPU, OS and Communication layer of the machines.  See 
  "How to choose a <version>" below for details.

OPTIONAL:
---------
<options> defines more detailed information of the compilations, including 
  compilers, features to support, etc.  See "How to choose <option>"
  below.
[--libdir=dir] specify additional lib paths for building Charm++.
[--incdir=dir] specify additional include paths for building Charm++.
[--basedir=dir] a shortcut to specify additional include and lib paths for 
               building Charm++, the include path is dir/include and lib path 
               is dir/lib.


Running build script, a directory of the name of combination of version and 
options like "<version>-<option1>-<option2>-..." will be created and 
the build script will compile Charm++ under this directory.

For example, on an ordinary Linux PC:

   ./build charm++ net-linux

will build charm++ in the directory: net-linux/.  The communication
defaults to UDP packets and the compiler to gcc.

For a more complex example, on a Scyld workstation with the Intel C++ 
compiler, where you want the communication to happen over TCP sockets:

   ./build charm++ net-linux scyld icc tcp

will build charm++ in the directory: net-linux-scyld-tcp-icc/.

You can specify multiple options, however you can use at most one compiler 
option. The sequence of the options are not important given in build script, 
only one directory name will be generated, following the rules:
1. compiler option will be at the end;
2. other options are sorted alphabetically.

**** How to choose a <version> ****

Here is the table for choosing correct version. The default setting of compiler
in Charm version is gcc/g++. However, one can use <options> to specify other 
compilers. See the detailed explanation of the <options> below.
(Note: this isn't a complete list.  Run ./build for a complete listing)

Charm version          OS        Communication    Default Compiler  
-------------       ---------    --------------   --------------------
net-linux            PC Linux       UDP/Myrinet   GNU compiler
net-sol              Solaris        UDP           GNU compiler
net-win32            Win32          UDP           MS Visual C++
net-cygwin           Win32/cygwin   UDP           GNU compiler
mpi-sp               IBM A/IX       MPI           A/IX xlC Compiler 
mpi-origin           Origin2000     MPI           SGI C++ compiler
net-ppc-darwin       MacOS X        UDP           GNU C++ compiler
net-linux-ia64       IA64 Linux     UDP/Myrinet   GNU compiler
net-linux-amd64      Opteron Linux  UDP           GNU compiler
net-irix             IRIX	    UDP           GNU compiler
net-axp              Alpha          UDP           GNU compiler
net-hp               HP-UX          UDP           GNU compiler
mpi-linux            PC Linux       MPI           GNU compiler
mpi-ppc-darwin       MacOS X        MPI           GNU C++ compiler
mpi-linux-ia64       IA64 Linux     MPI           GNU compiler
mpi-linux-x86_64     Opteron Linux  MPI           GNU compiler
mpi-axp              Alpha          MPI           GNU compiler
mpi-linux-axp        Alpha Linux    MPI           GNU compiler
origin2000           Origin2000  shared-mem       SGI C++ compiler
t3e                  Cray T3E    shared-mem       Cray C++ compiler


To choose <version>, your choice is determined by three options:

1.)  The way a parallel program written in Charm++ will communicate:

	"net-" Charm++ communicates using the regular TCP/IP stack
(UDP packets), which works everywhere but is fairly slow.  Use this
option for networks of workstations, clusters, or single-machine 
development and testing.

	"mpi-" Charm++ communicates using MPI calls.  Use this for
machines with a good MPI implementation (such as the Origin 2000).

	"exemplar", "ncube-2", "paragon-red", "sp3", and "t3e" Charm++
communicates using direct calls to the machine's communication primitives.

	"sim-" and "uth-" are not actively maintained.  These are
single-processor versions: "uth-" simulates processors as user-level
threads; "sim-" switches between processors and counts communications.


2.)  Your operating system:

	"linux"   Linux 
	"win32"   MS Windows with MS Visual C++ compiler
	"cygwin"  MS Windows with Cygnus' Cygwin Unix layer
	"irix"    SGI IRIX
	"origin"  SGI Origin 2000 IRIX
	"sol"     Solaris
	"sun"     SunOS
	"rs6k"    IBM R/S 6000 A/IX 
	"sp"      IBM SP A/IX
	"hp"      Hewlett-Packard HP-UX
	"axp"     DEC Alpha DECUNIX
	

3.)  Some operating systems have other options, such as:
	"-x86"     For Solaris, use PC hardware (instead of Sun).
	"-axp"     For Linux, use Alpha hardware (instead of PC).
        "-ia64"    Use Itanium(tm) instructions (instead of x86).
        "-x86_64"  Use AMD Opteron instructions (instead of x86).

Your Charm++ version is made by concatenating the options, e.g.:

"net-linux"     Charm++ for a network of Linux workstations, compiled
                using g++.
"mpi-origin"    Charm++ for SGI Origin 2000, compiled using SGI CC.


**** How to choose <options> ****

<version> above defines the most important OS, CPU and Communication of 
your machine, and most of time, it use the GNU gcc as default compiler. 
To use different compiler or demand additional special feature support, you 
need to choose <options> from the following list:

* gcc3  - GNU GCC/G++ version 3
* acc  - HP aC++ compiler
* cc  - For Sun WorkShop C++ compilers;
* cc64 - For 64 bits Sun WorkShop C++ or IBM xlC compilers;
* cxx - DIGITAL C++ compiler;
* kcc - KAI C++ compiler;
* pgcc - Portland Group's C++ compiler;
* icc - Intel C/C++ compiler for Linux IA32
* ecc - Intel C/C++ compiler for Linux IA64
* mpcc - SUN Solaris C++ compiler for MPI.

* scyld - support Beowulf Scyld based on bproc;
* clustermatic - for Clustermatic Beowulf cluster based on bproc;
* gm - support MyriCom's Myrinet GM library;
* vmi - support NCSA's VMI library;

* tcp - for net- version, default communication is via UDP. Using option
        tcp will switch to TCP. TCP version of CHarm++ is usually slower
        than UDP, but it is more reliable.
* smp - Enable direct SMP support.  An "smp" version communicates using
        shared memory within a machine; but normal message passing across 
        machines. Because of locking, "smp" may slightly impact non-SMP 
        performance. Try your application to decide if enabling smp mode 
        improves performance.

* bluegene - compile Charm++ as running on Blue Gene emulator.
* help - show supported options for a version. For example, for net-linux, 
         running:
         > ./build charm++ net-linux help
         will give:
         supported options: gcc3 gm icc kcc pgcc scyld smp bluegene tcp


BUILDING THE SOURCE
===================

If you have downloaded a binary version of Charm++, you can skip
this step-- Charm++ should already be compiled.  For win32/win64 systems,
see README.win; for Cygwin version, see README.cygwin; for net- version, 
see README.net.

Once you have decided on a version, unpack Charm++, cd into charm,
and run

     > ./build <target> <version> <opts>

<target> is one of
	"charm++"  The basic Charm++ language.
	"AMPI"     An implementation of MPI on top of Charm++
	"FEM"      A Finite-Element framework on top of Charm++
	"Tau"      TAU's performance profiling/tracing

<version> is described above

<opts> are build-time options (such as the compiler or "smp"), 
or command line options passed to the charmc compile script.
Common compile time options such as -g, -O, -Ipath, -Lpath, -llib are 
accepted.

For example, on a Linux machine, you would run
     > ./build charm++ net-linux -O

This will construct a net-linux directory, link over all
the Charm++ source code into net-linux/tmp, build the entire
Charm++ runtime system in net-linux/tmp, and link sample programs 
into net-linux/pgms.

Several #define's control the compilation of Charm++.  Some of these
#define's can be found in src/<version>/conv-mach.h.  #define's can
also be specified on the command line, using the -D option.  For
example,
    > ./build charm++ net-linux -O -DCMK_OPTIMIZE=1

CMK_OPTIMIZE: Turn on optimizations in Charm++/Converse. This disables most of
the run-time checking performed by Converse and Charm++ runtime. This option
should be used only after the program has been debugged. Also, this option
disables Converse/Charm++ tracing mechanisms such as projections and summary.

When Charm++ is built successfully, the diretory structure under the
target directory will look like:

net-linux/
   |
   ---  bin/			# all executables
   |
   ---  doc/			# documentations
   |
   ---  include/		# header files
   |
   ---  lib/			# libraries
   |
   ---  lib_so/			# dynamic libraries
   |
   ---  examples/		# all example programs
   |
   ---  tests/			# all test programs
   |
   ---  tmp/			# Charm++ build directory

BUILDING A PROGRAM
==================

To make a sample program, cd into pgms/charm++/queens/.
This program solves the N-queens problem-- find how many ways there 
are to arrange N queens on an NxN chess board such that none may 
attack another.

To build the program, type make.  You should get an
executable named "pgm".


RUNNING A PROGRAM
==================

Following the previous example, to run the program on two processors, type

     > ./charmrun ./pgm 12 6 +p2

This should run for a few seconds, and print out:
There are 14200 Solutions to 12 queens. Finish time=4.030000

Charmrun is used to provide a uniform interface to run charm programs.
On some platforms, charmrun is just a shell script which calls the 
platform-specific start program, such as mpirun on mpi versions.

For net- version, charmrun is an executable which invokes rsh or ssh to start 
node programs on remote machines. You should set up a ~/.nodelist that 
enumerates all the machines you want to run jobs on, otherwise it will
create a default ~/.nodelist for you that contains only localhost. Here is a 
typical .nodelist file:

group main ++shell /bin/ssh
host <machinename>

The default remote shell program is rsh, but you can define different remote 
shell you like to start remote processes in the ++shell option. You should 
also make sure that you can rsh or ssh to these machines without password 
authentication. Just type following command to verify:
     > rsh <machinename> date
If this gives you current date immediately, your running environment with this 
node has been setup correctly.

Now, for test running purpose, net- version charmrun comes with an easy-to-use 
"++local" options. No remote shell invocation is needed in this case. It starts
node programs right on your local machine. This could be useful if you just 
want to run program on only one machine, for example, your laptop. This
can save you all the hassle of setting up rsh/ssh or charmd daemons.
To use this option, just type:
     
     > ./charmrun ++local ./pgm 12 100 +p2

However, for best performance, you should launch one node program per processor.

For more detailed information, please check the "INSTALLATION MANUAL" and "RUN MANUAL" 
under doc/install.


Build Charm++ in Dynamic libraries
=============================

In order to compile Charm++ into dynamic libraries, one need to specify
"-build-shared" option as one of the Charm compiler script "charmc" 
at link time. For example, to compile Charm++ under net-linux/tmp, run

make charm++ OPTS='-O -build-shared'

Charm++'s dynamic libraries are compiled into lib_so/ directory. 
Typically, they are with ".so" suffix.

Note, "-build-shared" option is automatically turned on when building 
Charm++ using "build" script. So you don't need to pass "-build-shared" 
option to "build".

One can compile a Charm++ applicaiton linking against Charm++ dynamic 
libraries, run charmc with "-charm-shared" as one of the link options.
For example:

charmc -o pgm pgm.o -charm-shared

You can then run the program as usual.
Note, linking against Charm++ dynamic libraries produces much smaller size
binaries and takes much less linking time.

FOR MORE INFORMATION
====================

The Charm++ web page, with documentation, more programs,
and the latest version of Charm++, is at
	http://charm.cs.uiuc.edu/

The Charm++ mailing list, for questions, comments, suggestions, 
improvements, or bug reports is
	charm@cs.uiuc.edu


AUTHORS
=======

Charm++ was created and is maintained by the Parallel Programming Lab,
in the Computer Science department at the University of Illinois at
Urbana-Champaign.  Our managing professor is Dr. L.V. Kale; students
have included (in rough time order) Wennie Shu, Kevin Nomura, Wayne
Fenton, Balkrishna Ramkumar, Vikram Saletore, Amitabh B. Sinha, Manish
Gupta, Attila Gursoy, Balkrishna Ramkumar, Amitabh B. Sinha, Nimish
Shah, Sanjeev Krishnan, Jayant DeSouza, Parthasarathy Ramachandran,
Jeff Wright, Michael Lang, Jackie Wang, Fang Hu, Michael Denardo,
Joshua Yelon, Narain Jagathesan, Zehra Sura, Krishnan Varadarajan,
Sameer Paranjpye, Milind Bhandarkar, Robert Brunner, Jayant Desouza,
Terry Wilmarth, Orion Lawlor, Karthik Mahesh, Neelam Saboo, and Chee
Wai Lee.  Current developers include Gengbin Zheng, Eric Bohm, Filippo
Gioachin, Abhinav Bhatele, Isaac Dooley, Aaron Becker, Chao Mei, David
Kunzman, Esteban Meneses, Lukasz Wesolowski, Pritish Jetley, Ramprasad
Venkataraman, Ryan Mokos, Phil Miller, and Jonathan Lifflander.