### Start All Celeborn Services

Source: https://github.com/apache/celeborn/blob/main/docs/deploy.md

Use this script to start all Celeborn services if the installation paths are identical across nodes and SSH login is possible. Otherwise, start services manually.

```bash
$CELEBORN_HOME/sbin/start-all.sh
```

--------------------------------

### Master Service Startup Log Example

Source: https://github.com/apache/celeborn/blob/main/docs/deploy.md

Example output from the Master's log indicating a successful startup and worker registration.

```log
22/10/08 19:29:11,805 INFO [main] Dispatcher: Dispatcher numThreads: 64
22/10/08 19:29:11,875 INFO [main] TransportClientFactory: mode NIO threads 64
22/10/08 19:29:12,057 INFO [main] Utils: Successfully started service 'Master' on port 9097.
22/10/08 19:29:12,113 INFO [main] Master: Metrics system enabled.
22/10/08 19:29:12,125 INFO [main] HttpServer: master: HttpServer started on port 9098.
22/10/08 19:29:12,126 INFO [main] Master: Master started.
22/10/08 19:29:57,842 INFO [dispatcher-event-loop-19] Master: Registered worker
Host: 192.168.15.140
RpcPort: 37359
PushPort: 38303
FetchPort: 37569
ReplicatePort: 37093
SlotsUsed: 0()
LastHeartbeat: 0
Disks: {/mnt/disk1=DiskInfo(maxSlots: 6679, committed shuffles 0, running applications 0, shuffleAllocations: Map(), mountPoint: /mnt/disk1, usableSpace: 448284381184, avgFlushTime: 0, activeSlots: 0) status: HEALTHY dirs , /mnt/disk3=DiskInfo(maxSlots: 6716, committed shuffles 0, running applications 0, shuffleAllocations: Map(), mountPoint: /mnt/disk3, usableSpace: 450755608576, avgFlushTime: 0, activeSlots: 0) status: HEALTHY dirs , /mnt/disk2=DiskInfo(maxSlots: 6713, committed shuffles 0, running applications 0, shuffleAllocations: Map(), mountPoint: /mnt/disk2, usableSpace: 450532900864, avgFlushTime: 0, activeSlots: 0) status: HEALTHY dirs , /mnt/disk4=DiskInfo(maxSlots: 6712, committed shuffles 0, running applications 0, shuffleAllocations: Map(), mountPoint: /mnt/disk4, usableSpace: 450456805376, avgFlushTime: 0, activeSlots: 0) status: HEALTHY dirs }
WorkerRef: null
```

--------------------------------

### Install Dependencies with Pnpm

Source: https://github.com/apache/celeborn/blob/main/web/README.md

Run this command in the `celeborn/web` directory to install all required project dependencies.

```sh
pnpm install
```

--------------------------------

### Configure Example Metrics Source

Source: https://github.com/apache/celeborn/blob/main/docs/monitoring.md

Example configuration parameter to activate an 'ExampleSource' for Celeborn metrics. This demonstrates how to add custom or additional metric sources.

```properties
"celeborn.metrics.conf.*.source.jvm.class"="org.apache.celeborn.common.metrics.source.ExampleSource"
```

--------------------------------

### Setup Ubuntu Local Environment

Source: https://github.com/apache/celeborn/blob/main/cpp/README.md

Execute the setup script for the Celeborn C++ development environment on Ubuntu. This script handles C++ dependencies only.

```bash
cd scripts
bash setup-ubuntu.sh
```

--------------------------------

### Start Spark Shell with Celeborn

Source: https://github.com/apache/celeborn/blob/main/docs/README.md

Configure spark.shuffle.manager and spark.shuffle.service.enabled, then start the spark-shell.

```shell
cd $SPARK_HOME

./bin/spark-shell \
--conf spark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManager \
--conf spark.shuffle.service.enabled=false
```

--------------------------------

### Start Celeborn Services

Source: https://github.com/apache/celeborn/blob/main/README.md

Commands to start Celeborn master and worker services. Assumes Celeborn is installed in the same path on all nodes and SSH login is possible.

```bash
$CELEBORN_HOME/sbin/start-master.sh
```

```bash
$CELEBORN_HOME/sbin/start-worker.sh
```

--------------------------------

### Start Celeborn Master

Source: https://github.com/apache/celeborn/blob/main/docs/README.md

Navigate to the Celeborn home directory and start the master service using the start-master.sh script.

```shell
cd $CELEBORN_HOME
./sbin/start-master.sh
```

--------------------------------

### Start Development Server

Source: https://github.com/apache/celeborn/blob/main/web/README.md

Use this command to compile the project and enable hot-reloading for development.

```sh
pnpm dev
```

--------------------------------

### Example Dynamic Configurations

Source: https://github.com/apache/celeborn/blob/main/docs/developers/configuration.md

Illustrates SYSTEM, TENANT, and TENANT_USER level configurations for buffer size, showing how specific settings override broader ones.

```yaml
# SYSTEM level configuration
- level: SYSTEM
  config:
    celeborn.worker.flusher.buffer.size: 256K # sets buffer size of worker to 256 KiB

# TENANT level configuration
- tenantId: tenantId1
  level: TENANT
  config:
    celeborn.worker.flusher.buffer.size: 128K # sets buffer size of tenantId1 to 128 KiB
  users:
    # TENANT_USER level configuration
    - name: user1
      config:
        celeborn.worker.flusher.buffer.size: 96K # sets buffer size of tenantId1 and user1 to 128 KiB

```

--------------------------------

### Worker Tags Configuration Example (YAML)

Source: https://github.com/apache/celeborn/blob/main/docs/worker_tags.md

This example demonstrates how to assign worker tags and configure system-wide tag expressions using a YAML file with the FileSystem store backend. It shows SYSTEM, TENANT, and USER level configurations.

```yaml
-  level: SYSTEM
   config:
     celeborn.tags.preferClientTagsExpr: false
     celeborn.tags.tagsExpr: 'env=production'
   tags:
     env=production:
       - 'host1:1111'
       - 'host2:2222'
     env=staging:
       - 'host3:3333'
     region=us-east:
       - 'host1:1111'
       - 'host3:3333'
     region=us-west:
       - 'host2:2222'
-  tenantId: tenant_01
   level: TENANT
   config:
     celeborn.tags.preferClientTagsExpr: false
     celeborn.tags.tagsExpr: 'env=production,region=us-east'
   users:
     - name: Jerry
       config:
         celeborn.tags.preferClientTagsExpr: true

```

--------------------------------

### Run MapReduce Word Count Example

Source: https://github.com/apache/celeborn/blob/main/docs/README.md

Execute the MapReduce word count example using the Hadoop jar. This command is used to verify the Celeborn and MapReduce integration after configuration.

```shell
cd $HADOOP_HOME

./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar wordcount /someinput /someoutput
```

--------------------------------

### Run Flink Word Count Example

Source: https://github.com/apache/celeborn/blob/main/docs/README.md

Deploy the example word count job to a running Flink cluster configured with Celeborn. This command assumes you are in the Flink home directory.

```shell
cd $FLINK_HOME

./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH
```

--------------------------------

### Start Celeborn Master Service

Source: https://github.com/apache/celeborn/blob/main/docs/deploy.md

Manually start the Celeborn master service.

```bash
$CELEBORN_HOME/sbin/start-master.sh
```

--------------------------------

### Verify Celeborn CLI Installation

Source: https://github.com/apache/celeborn/blob/main/docs/celeborn_cli.md

Command to check if the Celeborn CLI is installed and working correctly by displaying its version.

```shell
celeborn-cli -V
```

--------------------------------

### Install Helm Unit Test Plugin

Source: https://github.com/apache/celeborn/blob/main/docs/developers/helm-charts.md

Install the helm unittest plugin to enable running unit tests for Helm charts. Ensure you use the specified version for compatibility.

```shell
helm plugin install https://github.com/helm-unittest/helm-unittest.git --version=0.5.1
```

--------------------------------

### Helm Install Celeborn Charts

Source: https://github.com/apache/celeborn/blob/main/docs/deploy_on_k8s.md

Navigate to the Celeborn charts directory and install Celeborn using Helm. Replace <namespace> with your target Kubernetes namespace.

```bash
cd ./charts/celeborn

helm install celeborn -n <namespace> .
```

--------------------------------

### Quota Configuration Example (SQL)

Source: https://github.com/apache/celeborn/blob/main/docs/quota_management.md

This SQL snippet demonstrates how to define quota settings for system default, tenant, and specific users when using the database store backend. The exact table and column names may vary based on your database schema.

```sql
This backend reads [quota](#quota-indicators) settings from a user-specified database.
For more information on using the database store backend, refer to [database config service](../developers/configuration#database-config-service).
Here's an example quota setting sql of above quota examples:

```

--------------------------------

### Worker JVM Profiler Configuration Example

Source: https://github.com/apache/celeborn/blob/main/docs/developers/jvmprofiler.md

Example configuration for the Celeborn worker JVM profiler, enabling it and setting specific options for profiling events, intervals, and memory allocation tracking.

```properties
celeborn.worker.jvmProfiler.enabled true
celeborn.worker.jvmProfiler.options event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s
```

--------------------------------

### Install Celeborn Ratis-shell

Source: https://github.com/apache/celeborn/blob/main/docs/celeborn_ratis_shell.md

Steps to extract the Celeborn binary package and create a symbolic link for the Celeborn home directory.

```shell
tar -C <DST_DIR> -zxvf apache-celeborn-<VERSION>-bin.tgz
ln -s <DST_DIR>/apache-celeborn-<VERSION>-bin <DST_DIR>/celeborn
```

--------------------------------

### Celeborn Master Pod Logs

Source: https://github.com/apache/celeborn/blob/main/docs/deploy_on_k8s.md

Example output showing Celeborn master pod logs during startup, indicating successful binding and service initialization.

```log
** server can't find celeborn-master-0.celeborn-master-svc.default.svc.cluster.local: NXDOMAIN

waiting for master
Server:         172.17.0.10
Address:        172.17.0.10#53

... 

Name:   celeborn-master-0.celeborn-master-svc.default.svc.cluster.local
Address: 10.225.139.80

Server:         172.17.0.10
Address:        172.17.0.10#53

starting org.apache.celeborn.service.deploy.master.Master, logging to /opt/celeborn/logs/celeborn--org.apache.celeborn.service.deploy.master.Master-1-celeborn-master-0.out

... 

23/03/23 14:10:56,081 INFO [main] RaftServer: 0: start RPC server
23/03/23 14:10:56,132 INFO [nioEventLoopGroup-2-1] LoggingHandler: [id: 0x83032bf1] REGISTERED
23/03/23 14:10:56,132 INFO [nioEventLoopGroup-2-1] LoggingHandler: [id: 0x83032bf1] BIND: 0.0.0.0/0.0.0.0:9872
23/03/23 14:10:56,134 INFO [nioEventLoopGroup-2-1] LoggingHandler: [id: 0x83032bf1, L:/0:0:0:0:0:0:0:0:9872] ACTIVE
23/03/23 14:10:56,135 INFO [JvmPauseMonitor0] JvmPauseMonitor: JvmPauseMonitor-0: Started
23/03/23 14:10:56,208 INFO [main] Master: Metrics system enabled.
23/03/23 14:10:56,216 INFO [main] HttpServer: master: HttpServer started on port 9098.
23/03/23 14:10:56,216 INFO [main] Master: Master started.
```

--------------------------------

### Quota Configuration Example (YAML)

Source: https://github.com/apache/celeborn/blob/main/docs/quota_management.md

This YAML file defines quota settings for system default, tenant, and specific users using the filesystem store backend. Ensure the file path is correctly configured in Celeborn.

```yaml
-  level: SYSTEM
   config:
     celeborn.quota.tenant.diskBytesWritten: 1G
     celeborn.quota.tenant.diskFileCount: 100
     celeborn.quota.tenant.hdfsBytesWritten: 1G

-  tenantId: tenant_01
   level: TENANT
   config:
     celeborn.quota.tenant.diskBytesWritten: 10G
     celeborn.quota.tenant.diskFileCount: 1000
     celeborn.quota.tenant.hdfsBytesWritten: 10G
   users:
     - name: Jerry
       config:
         celeborn.quota.tenant.diskBytesWritten: 100G
         celeborn.quota.tenant.diskFileCount: 10000

```

--------------------------------

### Get API Help

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Lists all available API providers and their descriptions for the master.

```APIDOC
## GET /help

### Description
List the available API providers of the master.

### Method
GET

### Endpoint
/help
```

--------------------------------

### Insert Dynamic Configuration Data

Source: https://github.com/apache/celeborn/blob/main/docs/developers/configuration.md

SQL statements to insert sample data for cluster information, system-level configurations, and tenant/user-level configurations. These examples demonstrate setting buffer sizes for workers.

```sql
INSERT INTO celeborn_cluster_info ( `id`, `name`, `namespace`, `endpoint`, `gmt_create`, `gmt_modify` )
VALUES
    ( 1, 'default', 'celeborn-worker', 'celeborn-namespace.endpoint.com', '2024-02-27 22:08:30', '2024-02-27 22:08:30' );

# SYSTEM level configuration
# sets buffer size of celeborn-worker to 256 KiB
INSERT INTO `celeborn_cluster_system_config` ( `id`, `cluster_id`, `config_key`, `config_value`, `type`, `gmt_create`, `gmt_modify` )
VALUES
    ( 1, 1, 'celeborn.worker.flusher.buffer.size', '256K', 'QUOTA', '2024-02-27 22:08:30', '2024-02-27 22:08:30' );

# TENANT/TENANT_USER level configuration
# TENANT: sets buffer size of tenantId1 to 128 KiB
# TENANT_USER: sets buffer size of tenantId1 and user1 to 96 KiB
INSERT INTO `celeborn_cluster_tenant_config` ( `id`, `cluster_id`, `tenant_id`, `level`, `name`, `config_key`, `config_value`, `type`, `gmt_create`, `gmt_modify` )
VALUES
    ( 1, 1, 'tenantId1', 'TENANT', '', 'celeborn.worker.flusher.buffer.size', '128K', 'worker', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 2, 1, 'tenantId1', 'TENANT_USER', 'user1', 'celeborn.worker.flusher.buffer.size', '96K', 'worker', '2024-02-27 22:08:30', '2024-02-27 22:08:30' );
```

--------------------------------

### Configure Library Search Paths

Source: https://github.com/apache/celeborn/blob/main/cpp/CMakeLists.txt

Sets up linker paths for libraries installed in the thirdparty directory, specifically targeting lib64 and lib subdirectories for Linux environments.

```cmake
message(STATUS "Using CMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}")
if (EXISTS ${CMAKE_PREFIX_PATH}/lib64)
    link_directories(${CMAKE_PREFIX_PATH}/lib64)
    link_directories(${CMAKE_PREFIX_PATH}/lib)
elseif (CMAKE_SYSTEM_NAME MATCHES "Linux")
    message(
            WARNING
            "You did not use the recommended way "
            "(using 'thirdparty/build-thirdparty.sh') to build & install "
            "thirdparty libraries.")
endif ()
```

--------------------------------

### Enable Spark Client Profile with SBT

Source: https://github.com/apache/celeborn/blob/main/docs/developers/sbt.md

To enable specific Spark client modules, use the Maven-style profile management with the `-P` flag. This example enables the Spark 3.3 client.

```bash
-Pspark-3.3
```

--------------------------------

### Start Celeborn Worker Service

Source: https://github.com/apache/celeborn/blob/main/docs/deploy.md

Manually start the Celeborn worker service.

```bash
$CELEBORN_HOME/sbin/start-worker.sh
```

--------------------------------

### Configure Custom Artifact Repository for SBT Startup

Source: https://github.com/apache/celeborn/blob/main/docs/developers/sbt.md

Accelerate SBT startup by setting the `DEFAULT_ARTIFACT_REPOSITORY` environment variable to a custom URL, such as a mirror or proxy. Ensure the URL ends with a trailing slash.

```shell
# The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
# Ensure that the URL ends with a trailing slash "/"
export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/
./build/sbt
```

--------------------------------

### Start Celeborn Worker

Source: https://github.com/apache/celeborn/blob/main/docs/README.md

Start a worker by providing the Master's IP and Port to the start-worker.sh script.

```shell
cd $CELEBORN_HOME
./sbin/start-worker.sh celeborn://<Master IP>:<Master Port>
```

--------------------------------

### Download and Set Up Celeborn

Source: https://github.com/apache/celeborn/blob/main/docs/README.md

Decompress the Celeborn binary and set the CELEBORN_HOME environment variable.

```shell
tar -C <DST_DIR> -zxvf apache-celeborn-<VERSION>-bin.tgz
export CELEBORN_HOME=<Decompressed path>
```

--------------------------------

### Run Prebuilt Docker Container

Source: https://github.com/apache/celeborn/blob/main/cpp/README.md

Launch a prebuilt Docker container for C++ development. Mount your local project directory and set the working directory inside the container.

```bash
export PROJECT_DIR=/your/path/to/celeborn/dir
docker run \
    -v ${PROJECT_DIR}:/celeborn \
    -w /celeborn \
    -it --rm \
    --name celeborn-cpp-dev-container \
    jraaaay/celeborn-cpp-dev:0.4 \
    /bin/bash
```

--------------------------------

### Server-Side Helm Installation Dry Run

Source: https://github.com/apache/celeborn/blob/main/charts/celeborn/README.md

Perform a dry run of a Helm installation to see the rendered templates on the server-side before actual deployment.

```shell
helm install --dry-run --debug --generate-name ../celeborn
```

--------------------------------

### Update Worker Start Script URL Schema

Source: https://github.com/apache/celeborn/blob/main/docs/migration.md

The Celeborn Master URL schema has changed from `rss://` to `celeborn://`. Update your worker start scripts accordingly.

```bash
sbin/start-worker.sh celeborn://<master-host>:<master-port>
```

--------------------------------

### Build and Launch Docker Container

Source: https://github.com/apache/celeborn/blob/main/cpp/README.md

Build a custom Docker image for Celeborn C++ development and then launch a container using the built image.

```bash
cd scripts

# build image
bash ./build-docker-image.sh

# launch container with image above
bash ./launch-docker-container.sh
```

--------------------------------

### Prepare Release Candidate Tag and Push

Source: https://github.com/apache/celeborn/blob/main/docs/developers/release.md

This sequence of commands bumps the version, creates a release candidate tag, prepares for the next development version, and pushes the branch and tag to the Apache remote repository. It also includes checking out the release candidate tag after pushing.

```shell
# Bump to the release version
# vim pom.xml <project.version>${RELEASE_VERSION}</project.version>
# vim version.sbt
# Regenerate the openapi-client code
build/sbt "clean;celeborn-openapi-client/generate"
git commit -am "[RELEASE] Bump ${RELEASE_VERSION}"

# Create tag
git tag v${RELEASE_VERSION}-rc${RELEASE_RC_NO}

# Prepare for the next development version
# vim pom.xml <project.version>${RELEASE_VERSION}</project.version>
# vim version.sbt
# Regenerate the openapi-client code
build/sbt "clean;celeborn-openapi-client/generate"
git commit -am "[RELEASE] Bump ${NEXT_VERSION}-SNAPSHOT"

# Push branch to apache remote repo
git push apache

# Push tag to apache remote repo
git push apache v${RELEASE_VERSION}-rc${RELEASE_RC_NO}

# Go back to release candidate tag 
git checkout v${RELEASE_VERSION}-rc${RELEASE_RC_NO}
```

--------------------------------

### Update KEYS File and Commit

Source: https://github.com/apache/celeborn/blob/main/docs/developers/release.md

Checkout the release repository, append your GPG public key to the KEYS file, and commit the changes using SVN.

```shell
svn checkout --depth=files "https://dist.apache.org/repos/dist/release/celeborn" tmp/svn-celeborn

(gpg --list-sigs "${ASF_USERNAME}@apache.org" && gpg --export --armor "${ASF_USERNAME}@apache.org") >> tmp/svn-celeborn/KEYS

svn commit --username "${ASF_USERNAME}" --password "${ASF_PASSWORD}" --message "Update KEYS" tmp/svn-celeborn
```

--------------------------------

### In-Flight Request Calculation Example

Source: https://github.com/apache/celeborn/blob/main/docs/configuration/index.md

This example demonstrates the calculation for total in-flight requests based on worker memory, buffer sizes, and the number of workers. It helps in tuning `spark.celeborn.client.push.maxReqsInFlight` for performance.

```plaintext
slots_per_worker          = 16 GB / 256 KB   -> 65,536
total_slots               = 65,536 * 5       -> 327,680
requests_per_slot         = 256 KB / 64 KB   -> 4
total_inflight_requests   = 327,680 × 4      -> 1,310,720
```

--------------------------------

### Get Worker Information

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Retrieves general information about the worker.

```APIDOC
## GET /workerInfo

### Description
List the worker information of the worker.

### Method
GET

### Endpoint
/workerInfo
```

--------------------------------

### Get Worker Configuration

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Fetches the current configuration settings of the worker.

```APIDOC
## GET /conf

### Description
List the conf setting of the worker.

### Method
GET

### Endpoint
/conf
```

--------------------------------

### Set up Celeborn Ratis-shell Environment

Source: https://github.com/apache/celeborn/blob/main/docs/celeborn_ratis_shell.md

Export the CELEBORN_HOME environment variable and add the Celeborn bin directory to the system's PATH.

```shell
export CELEBORN_HOME=<DST_DIR>/celeborn
export PATH=${CELEBORN_HOME}/bin:$PATH
```

--------------------------------

### Get Thread Dump

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Retrieves the current thread dump of the worker process.

```APIDOC
## GET /threadDump

### Description
List the current thread dump of the worker.

### Method
GET

### Endpoint
/threadDump
```

--------------------------------

### Run All Tests in a Project with SBT Console

Source: https://github.com/apache/celeborn/blob/main/docs/developers/sbt.md

Use the sbt console to select a project and run all its tests. Keep the console open for faster re-runs.

```bash
./build/sbt
> project celeborn-master
> test
```

--------------------------------

### Get thread dump

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Retrieves the current thread dump of the master process.

```APIDOC
## GET /threadDump

### Description
List the current thread dump of the master.

### Method
GET

### Endpoint
/threadDump
```

--------------------------------

### Send and Verify Public Key to Ubuntu Server

Source: https://github.com/apache/celeborn/blob/main/docs/developers/release.md

Upload your public GPG key to the Ubuntu keyserver and then verify its reception. Replace ${PUBLIC_KEY} with your actual public key identifier.

```shell
gpg --keyserver hkp://keyserver.ubuntu.com --send-keys ${PUBLIC_KEY} # send public key to ubuntu server
gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys ${PUBLIC_KEY} # verify
```

--------------------------------

### Publish MapReduce Shade Client

Source: https://github.com/apache/celeborn/blob/main/docs/developers/sbt.md

Use this command to publish the shade client for MapReduce. Ensure the necessary build and testing are completed beforehand.

```shell
$ ./build/sbt -Pmr celeborn-client-mr-shaded/publish
```

--------------------------------

### Get Master Configuration

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Retrieves the current configuration settings of the master node.

```APIDOC
## GET /conf

### Description
List the conf setting of the master.

### Method
GET

### Endpoint
/conf
```

--------------------------------

### Run Individual Tests from Command Line

Source: https://github.com/apache/celeborn/blob/main/docs/developers/sbt.md

Execute individual tests or test suites directly from the command line by enclosing the sbt command and its arguments in quotes. This is slower than using an open sbt console.

```bash
$ ./build/sbt "celeborn-master/testOnly *MasterSuite -- -z \"test single node startup functionality\""
```

--------------------------------

### Get Available API Providers

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Lists the available API providers exposed by the worker.

```APIDOC
## GET /help

### Description
List the available API providers of the worker.

### Method
GET

### Endpoint
/help
```

--------------------------------

### Build for Production

Source: https://github.com/apache/celeborn/blob/main/web/README.md

This command performs type-checking, compilation, and minification for a production-ready build.

```sh
pnpm build
```

--------------------------------

### Get Prometheus Metrics

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Retrieves worker metrics data in Prometheus format. The URL path is configurable.

```APIDOC
## GET /metrics/prometheus

### Description
List the metrics data in prometheus format of the worker. The url path is defined by configure `celeborn.metrics.prometheus.path`.

### Method
GET

### Endpoint
/metrics/prometheus
```

--------------------------------

### Build Celeborn with Aliyun OSS Support

Source: https://github.com/apache/celeborn/blob/main/README.md

Use this command to build Celeborn with Spark 3.4, JDK 8, and Aliyun OSS support enabled.

```shell
./build/make-distribution.sh --sbt-enabled -Pspark-3.4 -Pjdk-8 -Paliyun
```

--------------------------------

### Get Prometheus Metrics

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Retrieves master metrics data in Prometheus format. The URL path is configurable.

```APIDOC
## GET /metrics/prometheus

### Description
List the metrics data in prometheus format of the master. The url path is defined by configure `celeborn.metrics.prometheus.path`.

### Method
GET

### Endpoint
/metrics/prometheus
```

--------------------------------

### Dynamic Configuration Template

Source: https://github.com/apache/celeborn/blob/main/docs/developers/configuration.md

This YAML template outlines the structure for defining SYSTEM, TENANT, and TENANT_USER level dynamic configurations.

```yaml
# SYSTEM level configuration
- level: SYSTEM
  config:
    [config_key]: [config_val]
    ...

# TENANT level configuration
- tenantId: [tenant_id]
  level: TENANT
  config:
    [config_key]: [config_val]
    ...
  users:
    # TENANT_USER level configuration
    - name: [name]
      config:
        [config_key]: [config_val]
        ...

```

--------------------------------

### Get Master Group Information

Source: https://github.com/apache/celeborn/blob/main/docs/restapi.md

Retrieves information about the master group, including leader and follower status.

```APIDOC
## GET /masterGroupInfo

### Description
List master group information of the service. It will list all master's LEADER, FOLLOWER information.

### Method
GET

### Endpoint
/masterGroupInfo
```

--------------------------------

### Get LifecycleManager Host and Port

Source: https://github.com/apache/celeborn/blob/main/docs/developers/integrate.md

Retrieve the host and port of the created LifecycleManager instance. These are needed to establish a connection for the ShuffleClient.

```java
String host = lm.getHost();
int port = lm.getPort();
```

--------------------------------

### Publish Spark 3.5 Shade Client

Source: https://github.com/apache/celeborn/blob/main/docs/developers/sbt.md

Use this command to publish the shade client for Spark 3.5. Ensure the necessary build and testing are completed beforehand.

```shell
$ ./build/sbt -Pspark-3.5 celeborn-client-spark-3-shaded/publish
```

--------------------------------

### Local Helm Template Rendering

Source: https://github.com/apache/celeborn/blob/main/charts/celeborn/README.md

Use this command to render Helm chart templates locally for debugging without installing the chart.

```shell
helm template --debug ../celeborn
```

--------------------------------

### Configure Tenant and Tenant User Quotas

Source: https://github.com/apache/celeborn/blob/main/docs/quota_management.md

These SQL INSERT statements demonstrate how to configure quotas at the tenant level and for specific tenant users. Tenant-level configurations override system defaults, and tenant-user configurations override tenant-level settings.

```sql
INSERT INTO `celeborn_cluster_tenant_config` ( `id`, `cluster_id`, `tenant_id`, `level`, `name`, `config_key`, `config_value`, `type`, `gmt_create`, `gmt_modify` )
VALUES
    ( 1, 1, 'tenant_01', 'TENANT', '', 'celeborn.quota.tenant.diskBytesWritten', '10G', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 2, 1, 'tenant_01', 'TENANT', '', 'celeborn.quota.tenant.diskFileCount', '1000', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 3, 1, 'tenant_01', 'TENANT', '', 'celeborn.quota.tenant.hdfsBytesWritten', '10G', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 4, 1, 'tenant_01', 'TENANT_USER', 'Jerry', 'celeborn.quota.tenant.diskBytesWritten', '100G', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 5, 1, 'tenant_01', 'TENANT_USER', 'Jerry', 'celeborn.quota.tenant.diskFileCount', '10000', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' );
```

--------------------------------

### Create MySQL Tables for Dynamic Configurations

Source: https://github.com/apache/celeborn/blob/main/docs/developers/configuration.md

SQL script to create necessary tables in MySQL for storing dynamic configurations at cluster, system, and tenant/user levels. Ensure tables are created before inserting configuration data.

```sql
CREATE TABLE IF NOT EXISTS celeborn_cluster_info (
  id int NOT NULL AUTO_INCREMENT,
  name varchar(255) NOT NULL COMMENT 'celeborn cluster name',
  namespace varchar(255) DEFAULT NULL COMMENT 'celeborn cluster namespace',
  endpoint varchar(255) DEFAULT NULL COMMENT 'celeborn cluster endpoint',
  gmt_create timestamp NOT NULL,
  gmt_modify timestamp NOT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY `index_cluster_unique_name` (`name`)
);

# SYSTEM level configuration
CREATE TABLE IF NOT EXISTS celeborn_cluster_system_config (
  id int NOT NULL AUTO_INCREMENT,
  cluster_id int NOT NULL,
  config_key varchar(255) NOT NULL,
  config_value varchar(255) NOT NULL,
  type varchar(255) DEFAULT NULL COMMENT 'conf categories, such as quota',
  gmt_create timestamp NOT NULL,
  gmt_modify timestamp NOT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY `index_unique_system_config_key` (`cluster_id`, `config_key`)
);

# TENANT/TENANT_USER level configuration
CREATE TABLE IF NOT EXISTS celeborn_cluster_tenant_config (
  id int NOT NULL AUTO_INCREMENT,
  cluster_id int NOT NULL,
  tenant_id varchar(255) NOT NULL,
  level varchar(255) NOT NULL COMMENT 'config level, valid level is TENANT,USER',
  name varchar(255) DEFAULT NULL COMMENT 'tenant sub user',
  config_key varchar(255) NOT NULL,
  config_value varchar(255) NOT NULL,
  type varchar(255) DEFAULT NULL COMMENT 'conf categories, such as quota',
  gmt_create timestamp NOT NULL,
  gmt_modify timestamp NOT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY `index_unique_tenant_config_key` (`cluster_id`, `tenant_id`, `name`, `config_key`)
);
```