\section{Introduction}

This manual describes \charmpp, an object oriented portable parallel
programming language based on \CC. Its program structure, execution
model, interface language constructs and runtime system calls are
described here\footnote{For a description of the underlying design
philosophy please refer to the following papers :\\
    L. V. Kale and Sanjeev Krishnan,
    {\em ``\charmpp : Parallel Programming with Message-Driven Objects''},
    in ``Parallel Programming Using \CC'',
    MIT Press, 1995. \\
    L. V. Kale and Sanjeev Krishnan,
    {\em ``\charmpp : A Portable Concurrent Object Oriented System
    Based On \CC''},
    Proceedings of the Conference on Object Oriented Programming,
    Systems, Languages and Applications (OOPSLA), September 1993.
}.

\charmpp\ has continuously evolved since the OOPSLA 1993 paper.  The earlier
versions modified the \CC\ syntax to support \charmpp\ primitives, and
contained a full-fledged \charmpp\ translator that parsed the \charmpp\
syntactic extensions as well as the \CC\ syntax to produce a \CC\ program,
which was later compiled using a \CC\ compiler.  The current version does not
augment the \CC{} syntax, and does not use a \charmpp\ translator as in previous
versions. Instead, the older constructs are converted to calls into the runtime
library, several new constructs are added, and minimal language constructs are
used to describe the interfaces.

\charmpp\ is an explicitly parallel language based on \CC\ with a runtime
library for supporting parallel computation called the Charm kernel.  It
provides a clear separation between sequential and parallel objects.  The
execution model of \charmpp\ is message driven, thus helping one write programs
that are latency-tolerant.  \charmpp\ supports dynamic load balancing while
creating new work as well as periodically, based on object migration.  Several
dynamic load balancing strategies are provided.  \charmpp\ supports both
irregular as well as regular, data-parallel applications.  It is based on the
{\sc Converse} interoperable runtime system for parallel programming.

Currently the parallel platforms supported by \charmpp\ are the IBM SP, SGI
Origin2000, Cray T3E, Intel Paragon, a single workstation or a network of
workstations from Sun Microsystems (Solaris), IBM RS-6000 (AIX) SGI (IRIX 5.3
or 6.4), HP (HP-UX), and Intel x86 (Linux, Windows NT).  \charmpp\ programs can
run without changing the source on all these platforms.  Please see the
\charmpp{}/\converse{} Installation and Usage Manual for details about
installing, compiling and running \charmpp\ programs.

\subsection{Overview}

\charmpp\ is an object oriented parallel language. What sets \charmpp\ apart
from traditional programming models such as message passing and shared variable
programming is that the execution model of \charmpp\ is message-driven.
Therefore, computations in \charmpp\ are triggered based on arrival of
associated messages. These computations in turn can fire off more messages to
other (possibly remote) processors that trigger more computations on those
processors.

At the heart of any \charmpp\ program is a scheduler that repetitively chooses
a message from the available pool of messages, and executes the computations
associated with that message.

The programmer-visible entities in a \charmpp\ program are:

\begin{itemize}
\item Concurrent Objects : called {\em chares}\footnote{
      Chare (pronounced {\bf ch\"ar}, \"a as in c{\bf a}rt) is Old 
      English for chore.
      }
\item Communication Objects : Messages
\item Chare Groups and Nodegroups
\item Chare Arrays
\item Readonly data
\end{itemize}

\charmpp\ starts a program by creating a single \index{chare} instance of each
{\em mainchare} on processor 0, and invokes constructor methods of these
chares.  Typically, these chares then creates a number of other \index{chare}
chares, possibly on other processors, which can simultaneously work to solve
the problem at hand.

Each \index{chare}chare contains a number of \index{entry method}{\em entry
methods}, which are methods that can be invoked from remote processors. The
\charmpp\ runtime system needs to be explicitly told about these methods, via
an {\em interface} in a separate file.  The syntax of this interface
specification file is described in the later sections.

\charmpp\ provides system calls to asynchronously create remote \index{chare}
chares and to asynchronously invoke entry methods on remote chares by sending
\index{message} messages to those chares. This asynchronous
\index{message}message passing is the basic interprocess communication
mechanism in \charmpp. However, \charmpp\ also permits wide variations on this
mechanism to make it easy for the programmer to write programs that adapt to
the dynamic runtime environment.  These possible variations include
prioritization (associating priorities with method invocations), conditional
\index{message packing}message packing and unpacking (for reducing messaging
overhead), \index{quiescence}quiescence detection (for detecting completion of
some phase of the program), and dynamic load balancing (during remote object
creation). In addition, several libraries are built on top of \charmpp\ that
can simplify otherwise arduous parallel programming tasks.

The following sections provide detailed information about various features of
\charmpp\ programming system.

\subsection{History}

The {\sc Charm} software was developed as a group effort of the Parallel
Programming Laboratory at the University of Illinois at Urbana-Champaign.
Researchers at the Parallel Programming Laboratory keep \charmpp\ updated for
the new machines, new programming paradigms, and for supporting and simplifying
development of emerging applications for parallel processing.  The earliest
prototype, Chare Kernel(1.0), was developed in the late eighties. It consisted
only of basic remote method invocation constructs available as a library.  The
second prototype, Chare Kernel(2.0), a complete re-write with major design
changes.  This included C language extensions to denote Chares, messages and
asynchronous remote method invocation.  {\sc Charm}(3.0) improved on this
syntax, and contained important features such as information sharing
abstractions, and chare groups (called Branch Office Chares).  {\sc Charm}(4.0)
included \charmpp\ and was released in fall 1993.  \charmpp\ in its initial
version consisted of syntactic changes to \CC\ and employed a special
translator that parsed the entire \CC\ code while translating the syntactic
extensions.  {\sc Charm}(4.5)  had a major change that resulted from a
significant shift in the research agenda of the Parallel Programming
Laboratory. The message-driven runtime system code of the \charmpp\ was
separated from the actual language implementation, resulting in an
interoperable parallel runtime system called {\sc
Converse}. The \charmpp\ runtime system was
retargetted on top of {\sc Converse}, and popular programming paradigms such as
MPI and PVM were also implemented on {\sc Converse}. This allowed
interoperability between these paradigms and \charmpp. This release also
eliminated the full-fledged \charmpp\ translator by replacing syntactic
extensions to \CC\ with \CC\ macros, and instead contained a small language and
a translator for describing the interfaces of \charmpp\ entities to the runtime
system.  This version of \charmpp, which, in earlier releases was known as {\em
Interface Translator \charmpp}, is the default version of \charmpp\ now, and
hence referred simply as {\bf \charmpp}.  In early 1999, the runtime system of
\charmpp\ was formally named the Charm Kernel, and was rewritten in \CC.
Several new features were added. The interface language underwent significant
changes, and the macros that replaced the syntactic extensions in original
\charmpp, were replaced by natural \CC\ constructs. Late 1999, and early
2000 reflected several additions to \charmpp{}, when a load balancing
framework and migratable objects were added to \charmpp{}.
