### Example: Setting Up Completion Detection

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Configures completion detection for 10 producers, specifying callbacks for start, all produced, and finish events, with a priority of 0.

```c++
detector.start_detection(10,
                         CkCallback(CkIndex_chare1::start_test(), thisProxy),
                         CkCallback(CkIndex_chare1::produced_test(), thisProxy),
                         CkCallback(CkIndex_chare1::finish_test(), thisProxy),
                         0);
```

--------------------------------

### Quick Start Build Charm++

Source: https://github.com/charmplusplus/charm/blob/main/README.md

Run the top-level build script for a quick start. This is recommended for first-time users.

```bash
./build
```

--------------------------------

### Example Charm++ Build Command

Source: https://github.com/charmplusplus/charm/blob/main/README.md

An example of building the basic Charm++ runtime for a Linux x86_64 system with optimizations enabled.

```bash
./build charm++ netlrts-linux-x86_64 -O
```

--------------------------------

### Start Parallel Write Session (Basic)

Source: https://github.com/charmplusplus/charm/blob/main/doc/libraries/manual.md

Prepares to write data into a specified file within a given window (size and offset). A `SessionReadyMsg` is sent to the `ready` callback upon setup. All data is written and synced before the `complete` callback is invoked. This function should only be called from a single PE.

```c++
void Ck::IO::startSession(Ck::IO::File file, size_t size, size_t offset, CkCallback ready,
                 CkCallback complete)
```

--------------------------------

### taskGraphSolver setup Method

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/ck-libs/taskGraph/taskGraph.html

Placeholder for the setup method within the taskGraphSolver class.

```C++
void setup()
```

--------------------------------

### Set GKlib Installation Prefix

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/ck-libs/metis/GKlib/BUILD.txt

Specify a custom installation directory using 'make config'. This example sets the prefix to '~/local'.

```bash
$ make config prefix=~/local
```

--------------------------------

### Run XBraid Example with AMPI

Source: https://github.com/charmplusplus/charm/blob/main/doc/ampi/05-examples.md

Execute an XBraid example using charmrun with specified process grid and simulation parameters. Use ++local for local execution.

```bash
./charmrun +p2 ./ex-02 -pgrid 1 1 8 -ml 15 -nt 128 -nx 33 33 -mi 100 +vp8 ++local
```

--------------------------------

### Charm++ Load Balancer Example - lbexample.ci

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Defines the main module and chares for a simple load balancing example in Charm++'s CI language.

```charmci
/*** lbexample.ci ***/
mainmodule lbexample {
  readonly CProxy_Main mainProxy;
  readonly int nElements;

  mainchare Main {
    entry Main(CkArgMsg *m);
    entry void done(void);
  };

  array [1D] LBExample {
    entry LBExample(void);
    entry void doWork();
  };
};
```

--------------------------------

### Example POSE poser specification in .ci file

Source: https://github.com/charmplusplus/charm/blob/main/doc/pose/manual.md

A practical example of a poser specification in a .ci file, demonstrating constructor and event method declarations.

```none
poser Worker : sim adapt4 chpt {
  entry Worker(WorkerCreationMsg *);
  entry [event] void doWork(WorkMsg *);
  ...
};
```

--------------------------------

### Example Output of Supported Charm++ Options

Source: https://github.com/charmplusplus/charm/blob/main/README.md

This is an example of the output you might receive when querying for supported build options. It lists available compilers and functional options.

```text
Supported compilers: clang craycc gcc icc iccstatic msvc pgcc xlc xlc64 icx
Supported options: common cuda flang gfortran ifort local nolb omp ooc papi perftools persistent pgf90 pxshm smp syncft sysvshm tcp tsan
```

--------------------------------

### Charm++ Array Creation Specifying All Indices

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

This example explicitly sets the start, end, and step indices for a 1D Charm++ array. It demonstrates the full configuration of array bounds and spacing, equivalent to the default behavior for dense arrays.

```c++
// Specify all three indices
CkArrayOptions options;
options.setStart(CkArrayIndex1D(0))
       .setEnd(CkArrayIndex1D(nElements))
       .setStep(CkArrayIndex1D(1));
```

--------------------------------

### Starting a CCS Server

Source: https://github.com/charmplusplus/charm/blob/main/doc/converse/manual.md

Instructions on how to start a Converse program as a CCS server using charmrun with specific options.

```APIDOC
## Starting a CCS Server

A Converse program is started using

```bash
$ charmrun pgmname +pN charmrun-opts pgm-opts
```

charmrun also accepts the CCS options:

`++server`: open a CCS server on any TCP port number

`++server-port`=$port$: open the given TCP port as a CCS server

`++server-auth`=$authfile$: accept authenticated queries

As the parallel job starts, it will print a line giving the IP address and TCP port number of the new CCS server. The format is: “ccs: Server IP = $ip$, Server port = $port$ $”, where $ip$ is a dotted decimal version of the server IP address, and $port$ is the decimal port number.
```

--------------------------------

### Build UCX from Source

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Steps to clone, configure, build, and install UCX from its GitHub repository. Ensure UCX is installed in a location accessible by Charm++.

```bash
git clone https://github.com/openucx/ucx.git
cd ucx
./autogen.sh
./contrib/configure-release --prefix=$HOME/ucx/build
make -j8
make install
```

--------------------------------

### Charm++ Load Balancer Example - lbexample.C

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Implements the main chare and array chare for a load balancing example, demonstrating work distribution and synchronization using AtSync().

```c++
/*** lbexample.C ***/
#include <stdio.h>
#include "lbexample.decl.h"

/*readonly*/ CProxy_Main mainProxy;
/*readonly*/ int nElements;

#define MAX_WORK_CNT 50
#define LB_INTERVAL 5

/*mainchare*/
class Main : public CBase_Main
{
private:
  int count;
public:
  Main(CkArgMsg* m)
  {
    /*....Initialization....*/
    mainProxy = thisProxy;
    CProxy_LBExample arr = CProxy_LBExample::ckNew(nElements);
    arr.doWork();
  };

  void done(void)
  {
    count++;
    if(count==nElements){
      CkPrintf("All done");
      CkExit();
    }
  };
};

/*array [1D]*/
class LBExample : public CBase_LBExample
{
private:
  int workcnt;
public:
  LBExample()
  {
    workcnt=0;
    /* May initialize some variables to be used in doWork */
    //Must be set to true to make AtSync work
    usesAtSync = true;
  }

  LBExample(CkMigrateMessage *m) { /* Migration constructor -- invoked when chare migrates */ }

  /* Must be written for migration to succeed */
  void pup(PUP::er &p){
    p|workcnt;
    /* There may be some more variables used in doWork */
  }

  void doWork()
  {
    /* Do work proportional to the chare index to see the effects of LB */

    workcnt++;
    if(workcnt==MAX_WORK_CNT)
      mainProxy.done();

    if(workcnt%LB_INTERVAL==0)
      AtSync();
    else
      doWork();
  }

  void ResumeFromSync(){
    doWork();
  }
};

#include "lbexample.def.h"
```

--------------------------------

### Run MiniXYCE

Source: https://github.com/charmplusplus/charm/blob/main/doc/ampi/05-examples.md

Example command to run the MiniXYCE application with simulation parameters.

```bash
./charmrun +p3 ./miniXyce.x +vp3 -circuit ../tests/cir1.net -t_start 1e-6 -pf params.txt
```

--------------------------------

### Install Charm++ with CMake (Basic)

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Installs Charm++ using CMake, requiring a build directory to be created first.

```bash
cd charm
mkdir build-cmake
cd build-cmake
cmake ..
make -j4
```

--------------------------------

### Build OpenPMIx

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Steps to download, configure, build, and install OpenPMIx, specifying the libevent installation path. This is recommended for PMIx support in Charm++.

```bash
wget https://github.com/openpmix/openpmix/releases/download/v3.1.5/pmix-3.1.5.tar.gz
tar -xf pmix-3.1.5.tar.gz
cd pmix-3.1.5
./configure --prefix=$HOME/pmix-3.1.5/build --with-libevent=$HOME/libevent-2.1.12-stable/build
make -j
make install
```

--------------------------------

### Run MiniFE

Source: https://github.com/charmplusplus/charm/blob/main/doc/ampi/05-examples.md

Example command to run the MiniFE application with specified parameters.

```bash
./charmrun +p4 ./miniFE.x nx=30 ny=30 nz=30 +vp32
```

--------------------------------

### Run MFEM Example with AMPI

Source: https://github.com/charmplusplus/charm/blob/main/doc/ampi/05-examples.md

Execute an MFEM parallel example using charmrun. Runtime options can be added to control visualization and speed up execution.

```bash
./charmrun +p4 ./ex15p -m ../data/amr-quad.mesh +vp16
```

--------------------------------

### Build libevent for OpenPMIx

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Steps to download, configure, build, and install libevent, a dependency for OpenPMIx. Ensure the installation path is correctly specified.

```bash
wget https://github.com/libevent/libevent/releases/download/release-2.1.12-stable/libevent-2.1.12-stable.tar.gz
tar -xf libevent-2.1.12-stable.tar.gz
cd libevent-2.1.12-stable
./configure --prefix=$HOME/libevent-2.1.12-stable/build
make -j
make install
```

--------------------------------

### Fortran Example: Creating a Voxel Field

Source: https://github.com/charmplusplus/charm/blob/main/doc/mblock/manual.md

Example demonstrating the creation of a field for voxData in Fortran, including setting up dimensions and calculating offsets.

```fortran
!In Fortran
double precision, allocatable :: voxData(:,:,:)
integer :: size(3), ni,nj,nk
integer :: fid, err

!Find the dimensions of the grid interior
MBLK_Get_blocksize(size,err);

!Add ghost region width to the interior dimensions
size=size+4;  ! 4 because of the 2-deep region on both sides

!Allocate and initialize the grid
allocate(voxData(size(1),size(2),size(3)))
voxData=0.0

!Create a field for voxData
call MBLK_Create_field(
       &size,1, MBLK_DOUBLE,3,
       &offsetof(grid(1,1,1),grid(3,3,3)),
       &offsetof(grid(1,1,1),grid(2,1,1)),fid,err)
```

--------------------------------

### Initialize Converse System

Source: https://github.com/charmplusplus/charm/blob/main/doc/converse/manual.md

Use `ConverseInit` to start the Converse system. It initializes all processors and can execute in different modes affecting program flow and scheduler invocation. The `fn` parameter is a user-supplied start function.

```c++
typedef void (*CmiStartFn)(int argc, char **argv);
void ConverseInit(int argc, char *argv[], CmiStartFn fn, int usched, int
initret);
```

--------------------------------

### Fortran Driver Example for MBLK_Register

Source: https://github.com/charmplusplus/charm/blob/main/doc/mblock/manual.md

Example of calling MBLK_Register from a Fortran driver subroutine. It demonstrates how to pass a Fortran derived type and its PUP subroutine to register it for migration.

```fortran
!- Fortran driver subroutine
use my_block_mod
interface
  subroutine pup_my_block(p,m)
    use my_block_mod
    INTEGER :: p
    TYPE(my_block) :: m
  end subroutine
end interface
TYPE(my_block) :: m
INTEGER :: myId,err
MBLK_Register(m,pup_my_block,myId,err)
```

--------------------------------

### Example Program: Main Execution Flow

Source: https://github.com/charmplusplus/charm/blob/main/doc/convext/manual.md

Demonstrates initializing Convext, firing tasks, awaiting responses, and processing them using CmsAwaitResponses and CmsGetResponse.

```c++
#include "cms.h"

#define MAX 10

typedef struct {
    float a;
} Task;

typedef struct {
    float result;
} Response;

Task t;

int worker(Task *t, Response **r)
{
    /* do work and generate a single response */
    int i;
    Task *t1;
    int k;

    CmiPrintf("%d: in worker %f \n", CmiMyPe(), t->a);
    *r = (Response *) malloc(sizeof(Response));
    (*r)->result = t->a * t->a;
    return sizeof(Response);
}

int consumer(Response * r, int refnum)
{
    CmiPrintf("consumer: response with refnum = %d is %f\n", refnum,
              r->result);
}

main(int argc, char *argv[])
{
    int i, j, k, ref;
    /* 2nd parameter is the max number of tasks
     * fired before "awaitResponses"
     */
    CmsInit((CmsWorkerFn)worker, 20);
    if (CmiMyPe() == 0) { /* I am the manager */
        CmiPrintf("manager inited\n");
        for (i = 0; i < 3; i++) { /* number of iterations or phases */
          /* prepare the next generation of problems to solve */
          /* then, fire the next batch of tasks for the worker */
            for (j = 0; j < 5; j++) {
                t.a = 10 * i + j;
                ref = j;  /* a ref number to associate with the task, */
                /* so that the reponse for this task can be identified. */
                CmsFireTask(ref, &t, sizeof(t));
            }
          /* Now wait for the responses */
            CmsAwaitResponses();  /* allows proc 0 to be used as a worker. */
            /* Now extract the resoneses from the system */
            for (j = 0; j < 5; j++) {
                Response *r = (Response *) CmsGetResponse(j);
                CmiPrintf("Response %d is: %f \n", j, r->result);
            }
          /* End of one mast-slave phase */
            CmiPrintf("End of phase %d\n", i);
        }
    }

    CmiPrintf("Now the consumerFunction mode\n");

    if (CmiMyPe() == 0) { /* I am the manager */
       for (i = 0; i < 3; i++) {
           t.a = 5 + i;
           CmsFireTask(i, &t, sizeof(t));
       }
       CmsProcessResponses((CmsConsumerFn)consumer);
       /* Also allows proc. 0 to be used as a worker.
        * In addition, responses will be processed on processor 0
        * via the "consumer" function as soon as they are available
        */
    }
    CmsExit();
}
```

--------------------------------

### Get FFT Grid Dimensions

Source: https://github.com/charmplusplus/charm/blob/main/doc/libraries/manual.md

Retrieves the start and end extents for input (Z) and output (X) grids associated with an FFT proxy.

```c++
if (hasX) {
  Charm_getOutputExtents(gridStart[MY_X], gridEnd[MY_X],
                        gridStart[MY_Y], gridEnd[MY_Y],
                        gridStart[MY_Z], gridEnd[MY_Z],
                        fftProxy);
}

if (hasZ) {
  Charm_getInputExtents(gridStart[MY_X], gridEnd[MY_X],
                        gridStart[MY_Y], gridEnd[MY_Y],
                        gridStart[MY_Z], gridEnd[MY_Z],
                        fftProxy);
}

for(int i = 0; i < 3; i++) {
  gridLength[i] = gridEnd[i] - gridStart[i];
}

```

--------------------------------

### C++ Converse Program Example

Source: https://github.com/charmplusplus/charm/blob/main/doc/convext/manual.md

This example demonstrates how to initialize Converse, register handlers, and send prioritized messages using CldEnqueue. It shows message creation, setting handlers, and invoking the load balancer.

```c++
#include <stdio.h>
#include "converse.h"
#define CHARES 10

void startup(int argc, char *argv[]);
void registerAndInitialize();

typedef struct pemsgstruct
{
  char header[CmiExtHeaderSizeBytes];
  int pe, id, pfnx;
  int queuing, priobits;
  unsigned int prioptr;
} pemsg;

CpvDeclare(int, MyHandlerIndex);
CpvDeclare(int, InfoFnIndex);
CpvDeclare(int, PackFnIndex);

int main(int argc, char *argv[])
{
  ConverseInit(argc, argv, startup, 0, 0);
  CsdScheduler(-1);
}

void startup(int argc, char *argv[])
{
  pemsg *msg;
  int i;

  registerAndInitialize();
  for (i=0; i<CHARES; i++) {
    msg = (pemsg *)malloc(sizeof(pemsg));
    msg->pe = CmiMyPe();
    msg->id = i;
    msg->pfnx = CpvAccess(PackFnIndex);
    msg->queuing = CQS_QUEUEING_FIFO;
    msg->priobits = 0;
    msg->prioptr = 0;
    CmiSetHandler(msg, CpvAccess(MyHandlerIndex));
    CmiPrintf("[%d] sending message %d\n", msg->pe, msg->id);
    CldEnqueue(CLD_ANYWHERE, msg, CpvAccess(InfoFnIndex));
    /*    CmiSyncSend(i, sizeof(pemsg), &msg); */
  }
}

void MyHandler(pemsg *msg)
{
  CmiPrintf("Message %d created on %d handled by %d.\n", msg->id, msg->pe, 
         CmiMyPe());
}

void InfoFn(pemsg *msg, CldPackFn *pfn, int *len, int *queuing, int *priobits,
         unsigned int *prioptr)
{
  *pfn = (CldPackFn)CmiHandlerToFunction(msg->pfnx);
  *len = sizeof(pemsg);
  *queuing = msg->queuing;
  *priobits = msg->priobits;
  prioptr = &(msg->prioptr);
}

void PackFn(pemsg **msg)
{
}

void registerAndInitialize()
{
  CpvInitialize(int, MyHandlerIndex);
  CpvAccess(MyHandlerIndex) = CmiRegisterHandler(MyHandler);
  CpvInitialize(int, InfoFnIndex);
  CpvAccess(InfoFnIndex) = CldRegisterInfoFn((CldInfoFn)InfoFn);
  CpvInitialize(int, PackFnIndex);
  CpvAccess(PackFnIndex) = CldRegisterPackFn((CldPackFn)PackFn);
}

```

--------------------------------

### Initialize and Start POSE Simulation

Source: https://github.com/charmplusplus/charm/blob/main/doc/pose/manual.md

Initialize the POSE system and start the simulation. POSE_init() can be called with or without a time parameter to control termination via inactivity detection or an end time, respectively. POSE_start() begins the simulation after all posers are created.

```c++
POSE_init();
```

```c++
POSE_start();
```

--------------------------------

### Get Rank of a PE within its Node

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Returns the rank of the given PE within its node. This is the PE's index relative to the start of its node.

```c++
int CkRankOf(int pe)
```

--------------------------------

### Example nodelist file configuration

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Defines three groups of machines (kale-sun, kale-sol, main) with specific host configurations and qualifiers.

```none
group kale-sun ++cpus 1
  host charm.cs.illinois.edu ++shell ssh
  host dp.cs.illinois.edu
  host grace.cs.illinois.edu
  host dagger.cs.illinois.edu
group kale-sol
  host beauty.cs.illinois.edu ++cpus 2
group main
  host localhost
```

--------------------------------

### Get Rank of PE on Node

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Returns the rank number of the PE on the node where the call is made. PEs within a node are ranked starting from zero.

```c++
int CkMyRank()
```

--------------------------------

### Build PDF Manual with Sphinx and LaTeX

Source: https://github.com/charmplusplus/charm/blob/main/doc/README.rst

Installs Sphinx, navigates to the documentation directory, builds the LaTeX version of the manual, and then compiles it to PDF using make. Opens the PDF with evince.

```bash
$ pip install sphinx
$ cd charm/doc
$ sphinx-build -b latex . latex/
$ cd latex
$ make
$ evince charm.pdf
```

--------------------------------

### Define Charm++ Interface Function

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Interface functions are used to invoke Charm++ libraries from pure MPI programs. This example shows how to create a chare and start the Charm++ scheduler.

```c++
void HelloStart(int elems)
{
  if(CkMyPe() == 0) {
    CkPrintf("HelloStart - Starting lib by calling constructor of MainHello\n");
    CProxy_MainHello mainhello = CProxy_MainHello::ckNew(elems);
  }
  StartCharmScheduler(-1);
}
```

--------------------------------

### Compile RAJA with CUDA and OpenMP Backends

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Build the RAJA library with both CUDA for GPUs and OpenMP for multicore CPUs. This is necessary for the 'vecadd' example. Ensure to specify the correct installation prefix and paths.

```bash
mkdir build && install
cd build
cmake -DENABLE_CUDA=On -DCMAKE_INSTALL_PREFIX=<path to RAJA install folder> ../
make -j
make install
```

--------------------------------

### Run liveViz with Charm++ program

Source: https://github.com/charmplusplus/charm/blob/main/doc/libraries/manual.md

Example command to compile and run a Charm++ program with liveViz enabled, and then launch the liveViz client.

```bash
cd charm/examples/charm++/wave2d
make
./charmrun ./wave2d +p2 ++server ++server-port 1234
~/ccs_tools/bin/liveViz localhost 1234
```

--------------------------------

### Bulk Creation of Sparse 2D Charm++ Array (Set at Construction)

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

This example demonstrates creating a sparse 2D Charm++ array by specifying the start, end, and step indices directly within the CkArrayOptions constructor. It populates only odd indices from (1,1) to (10,10).

```c++
// Set at construction
CkArrayOptions options(CkArrayIndex2D(1,1),
                       CkArrayIndex2D(10,10),
                       CkArrayIndex(2,2));
```

--------------------------------

### Scalable and Batch Startup

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Configure startup strategies for multi-core systems. `++scalable-start` uses one SSH session per node for efficient launch. `++batch` launches nodes in sets to avoid overloading.

```bash
++scalable-start
```

```bash
++batch
```

--------------------------------

### Test Balancer Strategy Implementation (C++)

Source: https://github.com/charmplusplus/charm/blob/main/doc/convext/manual.md

Provides the core logic for the 'test' load balancer. Includes functions for getting the strategy name, distributing tokens, handling messages, enqueuing messages, and module initialization. Use this as a starting point for custom load balancers.

```cpp
#include <stdio.h>
#include "converse.h"
#define PERIOD 100
#define MAXMSGBFRSIZE 100000

const char *CldGetStrategy(void)
{
  return "test";
}

CpvDeclare(int, CldHandlerIndex);
CpvDeclare(int, CldBalanceHandlerIndex);
CpvDeclare(int, CldRelocatedMessages);
CpvDeclare(int, CldLoadBalanceMessages);
CpvDeclare(int, CldMessageChunks);

void CldDistributeTokens()
{
  int destPe = (CmiMyPe()+1)%CmiNumPes(), numToSend;

  numToSend = CldLoad() / 2;
  if (numToSend > CldCountTokens())
    numToSend = CldCountTokens() / 2;
  if (numToSend > 0)
    CldMultipleSend(destPe, numToSend);
  CcdCallFnAfter((CcdVoidFn)CldDistributeTokens, NULL, PERIOD);
}

void CldBalanceHandler(void *msg)
{
  CmiGrabBuffer((void **)&msg);
  CldRestoreHandler(msg);
  CldPutToken(msg);
}

void CldHandler(void *msg)
{
  CldInfoFn ifn; CldPackFn pfn;
  int len, queueing, priobits; unsigned int *prioptr;

  CmiGrabBuffer((void **)&msg);
  CldRestoreHandler(msg);
  ifn = (CldInfoFn)CmiHandlerToFunction(CmiGetInfo(msg));
  ifn(msg, &pfn, &len, &queueing, &priobits, &prioptr);
  CsdEnqueueGeneral(msg, queueing, priobits, prioptr);
}

void CldEnqueue(int pe, void *msg, int infofn)
{
  int len, queueing, priobits; unsigned int *prioptr;
  CldInfoFn ifn = (CldInfoFn)CmiHandlerToFunction(infofn);
  CldPackFn pfn;

  if ((pe == CLD_ANYWHERE) && (CmiNumPes() > 1)) {
    ifn(msg, &pfn, &len, &queueing, &priobits, &prioptr);
    CmiSetInfo(msg,infofn);
    CldPutToken(msg);
  }
  else if ((pe == CmiMyPe()) || (CmiNumPes() == 1)) {
    ifn(msg, &pfn, &len, &queueing, &priobits, &prioptr);
    CmiSetInfo(msg,infofn);
    CsdEnqueueGeneral(msg, queueing, priobits, prioptr);
  }
  else {
    ifn(msg, &pfn, &len, &queueing, &priobits, &prioptr);
    if (pfn) {
      pfn(&msg);
      ifn(msg, &pfn, &len, &queueing, &priobits, &prioptr);
    }
    CldSwitchHandler(msg, CpvAccess(CldHandlerIndex));
    CmiSetInfo(msg,infofn);
    if (pe==CLD_BROADCAST)
      CmiSyncBroadcastAndFree(len, msg);
    else if (pe==CLD_BROADCAST_ALL)
      CmiSyncBroadcastAllAndFree(len, msg);
    else CmiSyncSendAndFree(pe, len, msg);
  }
}

void CldModuleInit()
{
  char *argv[] = { NULL };
  CpvInitialize(int, CldHandlerIndex);
  CpvAccess(CldHandlerIndex) = CmiRegisterHandler(CldHandler);
  CpvInitialize(int, CldBalanceHandlerIndex);
  CpvAccess(CldBalanceHandlerIndex) = CmiRegisterHandler(CldBalanceHandler);
  CpvInitialize(int, CldRelocatedMessages);
  CpvInitialize(int, CldLoadBalanceMessages);
  CpvInitialize(int, CldMessageChunks);
  CpvAccess(CldRelocatedMessages) = CpvAccess(CldLoadBalanceMessages) =
    CpvAccess(CldMessageChunks) = 0;
  CldModuleGeneralInit(argv);
  if (CmiNumPes() > 1)
    CldDistributeTokens();
}

```

--------------------------------

### Set up VS 2015 Environment

Source: https://github.com/charmplusplus/charm/wiki/Building-Charm-on-Windows

Use this command to set up the environment for Visual Studio 2015. This command configures the necessary build tools and paths for the specified version.

```batch
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64

```

--------------------------------

### Compile a Charm++ Program

Source: https://github.com/charmplusplus/charm/blob/main/README.md

Navigate to the example program directory and use 'make' to compile it. This assumes a Makefile is present.

```bash
make
```

--------------------------------

### Install METIS

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/ck-libs/metis/BUILD.txt

Installs the built METIS library. The 'prefix' option can be used during 'make config' to specify an installation directory.

```bash
$ make install
```

--------------------------------

### Install OpenMP Headers

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/conv-libs/openmp_llvm/runtime/src/CMakeLists.txt

Installs the main OpenMP header file (omp.h) and optionally OMPT and Fortran module headers to the specified installation path.

```cmake
install(
  FILES
  ${CMAKE_CURRENT_BINARY_DIR}/omp.h
  DESTINATION ${LIBOMP_HEADERS_INSTALL_PATH}
)
if(${LIBOMP_OMPT_SUPPORT})
  install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ompt.h DESTINATION ${LIBOMP_HEADERS_INSTALL_PATH})
endif()
if(${LIBOMP_FORTRAN_MODULES})
  install(FILES
    ${CMAKE_CURRENT_BINARY_DIR}/omp_lib.h
    ${CMAKE_CURRENT_BINARY_DIR}/omp_lib.mod
    ${CMAKE_CURRENT_BINARY_DIR}/omp_lib_kinds.mod
    DESTINATION ${LIBOMP_HEADERS_INSTALL_PATH}
  )
endif()
```

--------------------------------

### Configure METIS with Custom Installation Prefix

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/ck-libs/metis/BUILD.txt

Sets a custom installation prefix for METIS during the configuration step. This path will be used when 'make install' is executed.

```bash
$ make config prefix=~/myroot/
```

--------------------------------

### Uninstall METIS

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/ck-libs/metis/BUILD.txt

Removes all files that were installed by 'make install'.

```bash
$ make uninstall
```

--------------------------------

### Build HTML Manual with Sphinx

Source: https://github.com/charmplusplus/charm/blob/main/doc/README.rst

Installs Sphinx, navigates to the documentation directory, and builds the HTML version of the manual. Opens the generated index file in Firefox.

```bash
$ pip install sphinx
$ cd charm/doc
$ sphinx-build . html/
$ firefox html/index.html
```

--------------------------------

### Install Charm++ with Spack (Disable SMP)

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Installs Charm++ using Spack while disabling SMP support.

```bash
$ spack install charmpp ~smp
```

--------------------------------

### Build Charm++ from Source

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Use this command to build Charm++ from source. Replace placeholders with your target framework, architecture, and desired options. Consult `./build --help` for a full list of options.

```bash
$ ./build <TARGET> <TARGET ARCHITECTURE> [OPTIONS]
```

--------------------------------

### User Consumer Function Example

Source: https://github.com/charmplusplus/charm/blob/main/doc/convext/manual.md

Example of a user-written consumer function that processes responses received by the system.

```c++
int consumer(Response * r, int refnum)
{
    CmiPrintf("consumer: response with refnum = %d is %f\n", refnum,
              r->result);
}
```

--------------------------------

### Download and Prepare Autobuild Script

Source: https://github.com/charmplusplus/charm/wiki/CDash-integration

Download the autobuild script and make it executable. This is the first step for running manual tests.

```bash
# Download autobuild script
$ wget https://raw.githubusercontent.com/UIUC-PPL/charm/master/cdash/run_autobuild.sh
$ chmod a+x run_autobuild.sh
```

--------------------------------

### Example Checkpoint Initiation

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Initiates a checkpoint to the 'log' directory using a predefined callback. Ensure the callback is properly constructed before calling.

```c++
/* ... */ CkCallback cb(CkIndex_Hello::SayHi(), helloProxy);
CkStartCheckpoint("log", cb);
```

--------------------------------

### Install Charm++ with Spack (MPI + OpenMP)

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Installs the MPI version of Charm++ with integrated OpenMP support using Spack.

```bash
$ spack install charmpp backend=mpi +omp
```

--------------------------------

### Install Charm++ with Spack (Default)

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Installs Charm++ using Spack with the default netlrts network backend and SMP support.

```bash
$ spack install charmpp
```

--------------------------------

### RDMA Get Operation with Callback

Source: https://github.com/charmplusplus/charm/blob/main/doc/converse/manual.md

Initiates an RDMA Get operation and registers a callback function to be invoked upon completion.

```APIDOC
## CmiGetCb

### Description
Initiates an RDMA Get operation and registers a callback function `fn` with associated parameter `param` to be executed upon completion.

### Method
`void CmiGetCb(unsigned int sourceId, unsigned int targetId, void *Saddr, void *Taddr, unsigned int size, CmiRdmaCallbackFn fn, void *param)`

### Parameters
- **sourceId** (unsigned int) - The ID of the source machine.
- **targetId** (unsigned int) - The ID of the target machine.
- **Saddr** (void *) - Pointer to the source memory location.
- **Taddr** (void *) - Pointer to the target memory location.
- **size** (unsigned int) - The size of the memory region to transfer in bytes.
- **fn** (CmiRdmaCallbackFn) - The callback function to invoke on completion.
- **param** (void *) - User-defined parameter to pass to the callback function.
```

--------------------------------

### Ck::IO::startSession

Source: https://github.com/charmplusplus/charm/blob/main/doc/libraries/manual.md

Initiates a write session for a specified file, preparing a window for data writing. Two variants are available: one for general session setup and another that includes committing user-specified data.

```APIDOC
## Ck::IO::startSession (Variant 1)

### Description
Prepare to write data into `file`, in the window defined by `size` and `offset` (both specified in bytes). When the session is set up, a `SessionReadyMsg` (wraps a `Ck::IO::Session session`) will be sent to the `ready` callback. When all of the data has been written and synced, an empty `CkReductionMsg` will be sent to the `complete` callback. Should only be called from a single PE, once per session.

### Method Signature
```c++
void Ck::IO::startSession(Ck::IO::File file, size_t size, size_t offset, CkCallback ready, CkCallback complete)
```

### Parameters
- `file` (Ck::IO::File) - The file to start a session on.
- `size` (size_t) - The size of the data window in bytes.
- `offset` (size_t) - The starting offset of the data window in bytes.
- `ready` (CkCallback) - The callback to invoke when the session is ready.
- `complete` (CkCallback) - The callback to invoke when all data has been written and synced.
```

```APIDOC
## Ck::IO::startSession (Variant 2)

### Description
Prepare to write data into `file`, in the window defined by `size` and `offset` (both specified in bytes). When the session is set up, a `SessionReadyMsg` (wraps a `Ck::IO::Session session`) will be sent to the `ready` callback. When all of the data has been written and synced, an additional write of `commitData` (of size `commitSize`) will be made to the file at the specified offset (`commitOffset`) to “commit” the session’s work. When that write has completed, an empty `CkReductionMsg` will be sent to the `complete` callback. Should only be called from a single PE, once per session.

### Method Signature
```c++
void Ck::IO::startSession(Ck::IO::File file, size_t size, size_t offset, CkCallback ready, const char *commitData, size_t commitSize, size_t commitOffset, CkCallback complete)
```

### Parameters
- `file` (Ck::IO::File) - The file to start a session on.
- `size` (size_t) - The size of the data window in bytes.
- `offset` (size_t) - The starting offset of the data window in bytes.
- `ready` (CkCallback) - The callback to invoke when the session is ready.
- `commitData` (const char *) - Pointer to the data to commit.
- `commitSize` (size_t) - The size of the commit data in bytes.
- `commitOffset` (size_t) - The offset at which to write the commit data.
- `complete` (CkCallback) - The callback to invoke when the commit write has completed.
```

--------------------------------

### Install Charm++ with Spack (Specific Version)

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Installs a specific version of Charm++ (e.g., 6.8.1) or the develop branch using Spack.

```bash
$ spack install charmpp@develop
```

--------------------------------

### Charm++ 'Hello World' Implementation (hello.cpp)

Source: https://github.com/charmplusplus/charm/blob/main/doc/quickstart.md

Implements the mainchare and the 'Hello' chare array declared in hello.ci. It demonstrates creating a chare array, invoking entry methods, and exiting the application.

```c++
#include "hello.decl.h" // created from hello.ci file above

/*readonly*/ CProxy_Main mainProxy;
constexpr int nElem = 8;

/*mainchare*/
class Main : public CBase_Main
{
public:
  Main(CkArgMsg* m)
  {
    //Start computation
    CkPrintf("Running Hello on %d processors with %d elements.\n", CkNumPes(), nElem);
    CProxy_Hello arr = CProxy_Hello::ckNew(nElem); // Create a new chare array with nElem elements
    mainProxy = thisProxy;
    arr[0].SayHi(0);
  };

  void done()
  {
    // Finish computation
    CkPrintf("All done.\n");
    CkExit();
  };
};

/*array [1D]*/
class Hello : public CBase_Hello
{
public:
  Hello() {}

  void SayHi()
  {
    // thisIndex stores the element’s array index
    CkPrintf("PE %d says: Hello world from element %d.\n", CkMyPe(), thisIndex);
    if (thisIndex < nElem - 1) {
      thisProxy[thisIndex + 1].SayHi(); // Pass the hello on
    } else {
      mainProxy.done(); // We've been around once -- we're done.
    }
  }
};

#include "hello.def.h" // created from hello.ci file above
```

--------------------------------

### Example predict Function Implementation

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

An example implementation of the predict function, demonstrating a simple linear model for predicting future load.

```c++
double predict(double x, double *param) {return (param[0]*x + param[1]);}
```

--------------------------------

### Build Charm++ with Visual Studio

Source: https://github.com/charmplusplus/charm/wiki/Building-Charm-on-Windows

Example commands for building Charm++ using Visual Studio with different communication layers and modes. These commands are executed from the Charm++ source directory.

```shell
./build charm++ multicore-win-x86_64

```

```shell
./build charm++ netlrts-win-x86_64

```

```shell
./build charm++ netlrts-win-x86_64 smp

```

```shell
./build charm++ mpi-win-x86_64

```

```shell
./build charm++ mpi-win-x86_64 smp

```

--------------------------------

### User Worker Function Example

Source: https://github.com/charmplusplus/charm/blob/main/doc/convext/manual.md

Example of a user-written worker function. It allocates and computes response data, returning a pointer to it and its size.

```c++
int worker(Task *t, Response **r)
{
    /* do work and generate a single response */
    int i;
    Task *t1;
    int k;

    CmiPrintf("%d: in worker %f \n", CmiMyPe(), t->a);
    *r = (Response *) malloc(sizeof(Response));
    (*r)->result = t->a * t->a;
    return sizeof(Response);
}
```

--------------------------------

### Compile Charm++ Example

Source: https://github.com/charmplusplus/charm/blob/main/doc/quickstart.md

Use the 'charmc' compiler wrapper to first compile the Charm interface file and then the C++ implementation file to create an executable.

```console
charm/bin/charmc hello.ci # creates hello.def.h and hello.decl.h
charm/bin/charmc hello.cpp -o hello
```

--------------------------------

### Set up VS 2017/2019/2022 Environment

Source: https://github.com/charmplusplus/charm/wiki/Building-Charm-on-Windows

Use these commands to set up the environment for Visual Studio 2017, 2019, or 2022. These commands configure the necessary build tools and paths.

```batch
"C:\Program Files\Microsoft Visual Studio\20XX\BuildTools\VC\Auxiliary\Build\vcvars64.bat"

```

```batch
"C:\Program Files (x86)\Microsoft Visual Studio\20XX\BuildTools\VC\Auxiliary\Build\vcvars64.bat"

```

--------------------------------

### Compile and Run Charm++ Hello World

Source: https://context7.com/charmplusplus/charm/llms.txt

Compile the Charm++ program using `charmc` to generate necessary header files and the executable. Then, run the application using `charmrun` on a specified number of processors.

```console
# Compile
$ charmc hello.ci                   # generates hello.decl.h and hello.def.h
$ charmc hello.cpp -o hello -language charm++

# Run on 2 PEs
$ ./charmrun +p2 ./hello
# Output:
# Running Hello on 2 PEs with 8 elements.
# PE 0: Hello from element 0
# ...
# All done.
```

--------------------------------

### Charm++ Interface File (.ci) Example

Source: https://github.com/charmplusplus/charm/blob/main/doc/f90charm/manual.md

Defines a Charm++ mainmodule with a 1D chare array 'Hello' and its entry methods. Use this to declare parallel chare arrays and their entry points.

```charmci
// ## Just replace Hello throughout with your chare's name. ##
// ## and add your chare's entry points below where indicated ##
// ## Everything else remains the same ##
mainmodule hello {
  // declare readonly variables which once set is available to all
  // Chares across processors.
  readonly int chunkSize;

  array [1D] Hello {
    entry Hello();

    // Note how your Fortran function takes the above defined
    // message instead of a list of parameters.
    entry void SayHi(int a, double b, int n, int arr[n]);

    // Other entry points go here
    entry [reductiontarget] void MyReduction(int result);
  };
};

```

--------------------------------

### Build OpenMPI from Source

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Steps to download, configure, build, and install OpenMPI from source. Ensure OpenMPI is available for Charm++ compilation.

```bash
$ wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.gz
$ tar -xvf openmpi-4.0.1.tar.gz
$ cd openmpi-4.0.1
$ ./configure --enable-install-libpmix --prefix=$HOME/openmpi-4.0.1/build
$ make -j24
$ make install all
```

--------------------------------

### Build Documentation with Doxygen

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/conv-libs/openmp_llvm/runtime/README.txt

Use this command to generate HTML documentation from Doxygen format. Ensure Doxygen is installed.

```bash
% doxygen doc/doxygen/config
```

--------------------------------

### Initiate Receiver-Side Persistent Communication Setup

Source: https://github.com/charmplusplus/charm/blob/main/doc/converse/manual.md

At the receiver, this function creates a temporary handle for initiating persistent communication setup. This handle should then be sent to the sender.

```c++
PersistentReq CmiCreateReceiverPersistent(int maxBytes);
```

--------------------------------

### Runtime Load Balancer Configuration

Source: https://context7.com/charmplusplus/charm/llms.txt

Examples of how to select and configure load balancers at runtime. The first command uses `GreedyLB`, while the second uses `TreeLB` with a JSON configuration file.

```console
# Select load balancer at runtime (TreeLB with GreedyRefine is recommended)
$ ./charmrun +p8 ./lbsim +balancer GreedyLB

# Use TreeLB with JSON config file treelb.json
$ ./charmrun +p8 ./lbsim +TreeLBFile treelb.json

```

--------------------------------

### Split Charm++ Initialization in User-Driven Mode

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

Use CharmBeginInit to initialize the runtime and create main chares, returning control to user code. Use CharmFinishInit to complete initialization by sending messages for readonlies and groups.

```c++
void CharmBeginInit(int argc, char **argv);
void CharmFinishInit();
```

--------------------------------

### Install Metis Executables

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/ck-libs/metis/programs/CMakeLists.txt

Installs the built Metis executables to the 'bin' directory if the METIS_INSTALL option is enabled. This makes the programs available in the system's PATH.

```cmake
if(METIS_INSTALL)
  install(TARGETS gpmetis ndmetis mpmetis m2gmetis graphchk cmpfillin
    RUNTIME DESTINATION bin)
endif()
```

--------------------------------

### Install Metis Library

Source: https://github.com/charmplusplus/charm/blob/main/src/libs/ck-libs/metis/libmetis/CMakeLists.txt

This snippet defines the installation rules for the 'ckmetis' target, specifying the destination directories for libraries, runtime files, and archives when METIS_INSTALL is enabled.

```cmake
if(METIS_INSTALL)
  install(TARGETS ckmetis
    LIBRARY DESTINATION lib
    RUNTIME DESTINATION lib
    ARCHIVE DESTINATION lib)
endif()
```

--------------------------------

### Example CPM Program

Source: https://github.com/charmplusplus/charm/blob/main/doc/convext/manual.md

A sample C program demonstrating CPM initialization and invoking a remote function. It includes the generated .cpm.h file and calls `CpmInitializeThisModule`.

```c++
#include "myprog.cpm.h"

CpmInvokable print_integer(int n)
{
  CmiPrintf("%d\n", n);
}

user_main(int argc, char **argv)
{
  int i;
  CpmModuleInit();
  CpmInitializeThisModule();
  if (CmiMyPe()==0)
    for (i=1; i<CmiNumPes(); i++)
      Cpm_print_integer(CpmSend(i), rand());
}

main(int argc, char **argv)
{
  ConverseInit(argc, argv, user_main, 0, 0);
}
```

--------------------------------

### Get Command-Line Help for Charm++ Programs

Source: https://github.com/charmplusplus/charm/blob/main/doc/faq/manual.md

Execute './charmrun ./pgm --help' to display the command-line arguments available for a specific Charm++ program. Charmrun arguments are documented separately.

```bash
./charmrun ./pgm --help
```

--------------------------------

### Example function Implementation for Parameter Learning

Source: https://github.com/charmplusplus/charm/blob/main/doc/charm++/manual.md

An example implementation of the function method, which is used internally for learning model parameters. It calculates the function value and its derivatives.

```c++
void function(double x, double *param, double &y, double *dyda) {
    y = predict(x, param);
    dyda[0] = x;
    dyda[1] = 1;
  }
```

--------------------------------

### Run CharmMPI Program with Virtualization

Source: https://github.com/charmplusplus/charm/wiki/CharmMPI-Onboarding-Tutorial

Utilize the `+vp` parameter to specify the total number of virtualized ranks for load balancing. This example sets 64 virtual processors for 8 logical nodes.

```sh
"$CHARMBIN/charmrun" ./myprogram +n8 +vp64
```