### Connect to Vineyard Client

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/getting-started.rst

Demonstrates how to initialize a connection to the Vineyard daemon using a socket path.

```python
import vineyard
client = vineyard.connect('/var/run/vineyard.sock')
```

--------------------------------

### Install and Launch Vineyard Server

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/getting-started.rst

Commands to install the Vineyard Python package and launch the daemon server. Includes options for handling permission issues by specifying custom socket paths.

```console
$ pip3 install vineyard
$ python3 -m vineyard
$ sudo -E python3 -m vineyard
$ python3 -m vineyard --socket /tmp/vineyard.sock
```

--------------------------------

### Install vineyard Python Package

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Installs the vineyard Python package using pip. This is the simplest way to get started with vineyard if you don't need to build from source.

```shell
pip3 install vineyard
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Installs the Python packages required for building the vineyard documentation using pip. It installs packages listed in `requirements.txt` and `requirements-dev.txt`.

```shell
pip3 install -r requirements.txt -r requirements-dev.txt
```

--------------------------------

### Install and Execute Argo Workflows

Source: https://github.com/v6d-io/v6d/blob/main/docs/tutorials/kubernetes/efficient-data-sharing-in-kubeflow-with-vineyard-csi-driver.rst

Instructions for installing the Argo workflow server and submitting pipeline jobs. Includes loops to run experiments with different data multipliers to measure performance.

```bash
# Install Argo
kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.4.8/install.yaml

# Submit pipeline jobs
for data_multiplier in 4000 5000 6000; do
    argo delete --all -n kubeflow
    argo submit --watch pipeline.yaml -n kubeflow -p data_multiplier=${data_multiplier} -p registry="ghcr.io/v6d-io/v6d/kubeflow-example"
    sleep 60
done
```

--------------------------------

### Install Fluid Python SDK

Source: https://github.com/v6d-io/v6d/blob/main/docs/tutorials/kubernetes/vineyard-on-fluid.rst

Installs the Fluid Python SDK from its GitHub repository using pip. This command fetches and installs the latest development version of the SDK, which is necessary for interacting with the Fluid platform programmatically.

```bash
$ pip install git+https://github.com/fluid-cloudnative/fluid-client-python.git
```

--------------------------------

### Install Vineyard Operator from Source

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/cloud-native/deploy-kubernetes.rst

Manual installation steps including cloning the repository, building the Docker image, and deploying the operator to the cluster.

```bash
git clone https://github.com/v6d-io/v6d.git
cd k8s
make -C k8s docker-build
kind load docker-image vineyardcloudnative/vineyard-operator:latest
make -C k8s deploy
```

--------------------------------

### Install Kedro Vineyard Plugin

Source: https://github.com/v6d-io/v6d/blob/main/python/vineyard/contrib/kedro/README.md

Installs the necessary packages to use the Kedro vineyard plugin. This command installs the vineyard-kedro package, which includes the components for integrating Kedro with vineyard.

```bash
pip3 install vineyard-kedro
```

--------------------------------

### Install Apache Arrow on Ubuntu/Debian

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Installs the Apache Arrow development libraries on Ubuntu or Debian by adding the official Arrow APT repository and then installing the `libarrow-dev` package.

```shell
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb \
    -O /tmp/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
apt install -y -V /tmp/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
apt update -y
apt install -y libarrow-dev
```

--------------------------------

### Schedule Workload with JSON Resource (CLI Example)

Source: https://github.com/v6d-io/v6d/blob/main/k8s/cmd/README.md

This example demonstrates scheduling a workload by providing its definition directly as a JSON string via the `--resource` flag. It also specifies the vineyardd service details.

```shell
vineyardctl schedule workload --resource '{ \
    "apiVersion": "apps/v1", \
    "kind": "Deployment", \
    "metadata": { \
      "name": "web-server" \
    }, \
    "spec": { \
      "selector": { \
      "matchLabels": { \
        "app": "web-store" \
      } \
      }, \
      "replicas": 3, \
      "template": { \
      "metadata": { \
        "labels": { \
        "app": "web-store" \
        } \
      }, \
      "spec": { \
        "affinity": { \
        "podAntiAffinity": { \
          "requiredDuringSchedulingIgnoredDuringExecution": [ \
          { \
            "labelSelector": { \
            "matchExpressions": [ \
              { \
              "key": "app", \
              "operator": "In", \
              "values": [ \
                "web-store" \
              ] \
              } \
            ] \
            }, \
            "topologyKey": "kubernetes.io/hostname" \
          } \
          ] \
        }, \
        "podAffinity": { \
          "requiredDuringSchedulingIgnoredDuringExecution": [ \
          { \
            "labelSelector": { \
            "matchExpressions": [ \
              { \
              "key": "app", \
              "operator": "In", \
              "values": [ \
                "store" \
              ] \
              } \
            ] \
            }, \
            "topologyKey": "kubernetes.io/hostname" \
          } \
          ] \
        } \
        }, \
        "containers": [ \
        { \
          "name": "web-app", \
          "image": "nginx:1.16-alpine" \
        } \
        ] \
      } \
      } \
    } \
    }' \
    --vineyardd-name vineyardd-sample \
    --vineyardd-namespace vineyard-system
```

--------------------------------

### Install KFP Standalone Instance

Source: https://github.com/v6d-io/v6d/blob/main/docs/tutorials/kubernetes/efficient-data-sharing-in-kubeflow-with-vineyard-csi-driver.rst

Commands to deploy a standalone KFP instance on a Kubernetes cluster using kustomize and verify the deployment status.

```bash
export PIPELINE_VERSION=2.0.1

kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"
```

--------------------------------

### Build C++ Executable with Vineyard (CMake)

Source: https://github.com/v6d-io/v6d/blob/main/test/vineyard-cmake-example/CMakeLists.txt

This snippet shows how to configure a C++ executable using CMake to link with the vineyard library. It requires CMake version 3.3 or higher and the vineyard package to be installed. The output is a compiled executable ready to use vineyard features.

```cmake
cmake_minimum_required(VERSION 3.3)

project(vineyard-cmake-example LANGUAGES C CXX)

find_package(vineyard REQUIRED)

add_executable(vineyard-cmake-example ${CMAKE_CURRENT_SOURCE_DIR}/example.cc)
target_include_directories(vineyard-cmake-example PRIVATE ${VINEYARD_INCLUDE_DIRS})
target_link_libraries(vineyard-cmake-example PRIVATE ${VINEYARD_LIBRARIES})
```

--------------------------------

### Install Fluid Platform with Helm

Source: https://github.com/v6d-io/v6d/blob/main/docs/tutorials/kubernetes/vineyard-on-fluid.rst

Installs the Fluid platform on ACK using Helm. This involves adding the Fluid Helm repository, updating it, searching for the development version, and deploying the chart. Ensure Helm is installed and configured.

```bash
# Create the fluid-system namespace
$ kubectl create ns fluid-system

# Add the Fluid repository to the Helm repository
$ helm repo add fluid https://fluid-cloudnative.github.io/charts
# Get the latest Fluid repository
$ helm repo update
# Find the development version in the Fluid repository
$ helm search repo fluid --devel
# Deploy the corresponding version of the Fluid chart on ACK
$ helm install fluid fluid/fluid --devel
```

--------------------------------

### Install Vineyard Operator with Helm

Source: https://github.com/v6d-io/v6d/blob/main/charts/vineyard-operator/README.md

Instructions for installing the vineyard-operator using Helm. It highlights the importance of not using the `--wait` flag and shows how to install in a specific namespace or with custom values.

```shell
helm install vineyard-operator vineyard/vineyard-operator
helm install vineyard-operator vineyard/vineyard-operator \
      --namespace vineyard-system \
      --create-namespace
helm install vineyard-operator vineyard/vineyard-operator \
      --namespace vineyard-system \
      --set controllerManager.manager.image.tag=v0.10.1
```

--------------------------------

### Build HTML Documentation

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Builds the HTML version of the vineyard documentation. This command should be run from the `docs/` directory after installing the necessary dependencies.

```shell
cd docs/  # skip if you are already there
make html
```

--------------------------------

### Run Vineyard Server and Configure Environment

Source: https://github.com/v6d-io/v6d/blob/main/modules/llm-cache/README.md

Instructions for starting the vineyard daemon and setting the PYTHONPATH to include the library modules.

```bash
./build/bin/vineyardd --socket /tmp/vineyard_test.sock
export PYTHONPATH=/INPUT_YOUR_PATH_HERE/v6d/python:$PYTHONPATH
```

--------------------------------

### Start MySQL Docker Container for Hive Metastore

Source: https://github.com/v6d-io/v6d/blob/main/java/hive/README.rst

Starts a MySQL Docker container using docker-compose for the Hive metastore. The configuration, including the password, can be adjusted in `mysql-compose.yaml` and `hive-site.xml`.

```bash
cd v6d/java/hive/docker/dependency/mysql
docker-compose -f mysql-compose.yaml up -d
```

--------------------------------

### Define Installation Macros

Source: https://github.com/v6d-io/v6d/blob/main/CMakeLists.txt

Macros for installing build targets and project headers, including specific patterns for file types and exclusion rules for third-party directories.

```cmake
macro(install_vineyard_target target)
    install(TARGETS ${target}
        ARCHIVE DESTINATION lib
        LIBRARY DESTINATION lib
        RUNTIME DESTINATION bin
    )
endmacro()

macro(install_export_vineyard_target target)
    install(TARGETS ${target}
        EXPORT vineyard-targets
        ARCHIVE DESTINATION lib
        LIBRARY DESTINATION lib
        RUNTIME DESTINATION bin
    )
endmacro()

macro(install_vineyard_headers header_path)
    if(${ARGC} GREATER_EQUAL 2)
        set(install_headers_destination "${ARGV1}")
    else()
        set(install_headers_destination "include/vineyard")
    endif()
    install(DIRECTORY ${header_path}
            DESTINATION ${install_headers_destination}
            FILES_MATCHING
            PATTERN "*.h"
            PATTERN "*.hpp"
            PATTERN "*.vineyard-mod"
            PATTERN "*/thirdparty/*" EXCLUDE
    )
endmacro()
```

--------------------------------

### Install v6d Dependencies on Ubuntu/Debian

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Installs the necessary system dependencies for building vineyard on Ubuntu or Debian-based systems using apt-get. This includes build tools, libraries like boost, gflags, glog, grpc, protobuf, and MPICH.

```shell
apt-get install -y ca-certificates \
                       cmake \
                       doxygen \
                       libboost-all-dev \
                       libcurl4-openssl-dev \
                       libgflags-dev \
                       libgoogle-glog-dev \
                       libgrpc-dev \
                       libgrpc++-dev \
                       libmpich-dev \
                       libprotobuf-dev \
                       libssl-dev \
                       libunwind-dev \
                       libz-dev \
                       protobuf-compiler-grpc \
                       python3-pip \
                       wget
```

--------------------------------

### Install Vineyard Operator via Helm

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/cloud-native/deploy-kubernetes.rst

Recommended method to install the Vineyard Operator using Helm charts from the official repository.

```bash
helm repo add vineyard https://vineyard.oss-ap-southeast-1.aliyuncs.com/charts/
helm repo update
helm install vineyard-operator vineyard/vineyard-operator --namespace vineyard-system --create-namespace
```

--------------------------------

### Configure Vineyard Server

Source: https://github.com/v6d-io/v6d/blob/main/java/hive/README.rst

Command to start the Vineyard server with specified socket and metadata paths.

```bash
vineyardd --socket=./vineyard/vineyard.sock --meta=local
```

--------------------------------

### Install Vineyard IO Target and Headers - CMake

Source: https://github.com/v6d-io/v6d/blob/main/modules/io/CMakeLists.txt

These CMake commands handle the installation of the 'vineyard_io' target and its associated headers. 'install_export_vineyard_target' makes the target available for other projects, while 'install_vineyard_headers' copies the specified header files to the installation directory.

```cmake
install_export_vineyard_target(vineyard_io)
install_vineyard_headers("${CMAKE_CURRENT_SOURCE_DIR}/io" "include/vineyard/io")
```

--------------------------------

### Start Vineyard Server for Mimalloc Benchmark

Source: https://github.com/v6d-io/v6d/blob/main/benchmark/alloc_test/README.md

Starts the vineyard server with a specified amount of shared memory, which is necessary before running the mimalloc benchmark tests. The vineyard socket path needs to be defined.

```bash
export vineyard_socket=[please create a socket file here]
./vineyardd -socket=$(vineyard_socket) -size=8G
```

--------------------------------

### Install libclang Python Package

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Installs the libclang Python package, which is a required dependency for building vineyard. This is done using pip.

```shell
pip3 install libclang
```

--------------------------------

### Install and Deploy Vineyard Cluster via CLI

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/cloud-native/deploy-kubernetes.rst

Commands to install the vineyardctl tool and deploy a vineyard cluster directly into a Kubernetes namespace.

```bash
pip3 install vineyard
python3 -m vineyard.ctl deploy vineyard-cluster --create-namespace
```

--------------------------------

### Configure and Launch Vineyard Environment

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/integration/kedro.md

Commands to install the plugin, launch the local Vineyard server, and configure the environment variable for socket communication.

```bash
pip3 install vineyard-kedro
python3 -m vineyard --socket=/tmp/vineyard.sock
export VINEYARD_IPC_SOCKET=/tmp/vineyard.sock
```

--------------------------------

### Install and Configure Pre-commit Hooks

Source: https://github.com/v6d-io/v6d/blob/main/CONTRIBUTING.rst

Installs the pre-commit tool and configures the necessary hooks for the Vineyard project. This ensures code quality and prevents accidental secret inclusion before committing.

```bash
pip3 install pre-commit
pre-commit install
```

--------------------------------

### Start Vineyard CSI Driver with vineyardctl

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/cloud-native/vineyardctl.md

Starts the Vineyard CSI driver using specific endpoint and node ID configurations. This command is essential for enabling storage orchestration within the Vineyard environment.

```shell
vineyardctl csi --endpoint=unix:///csi/csi.sock --nodeid=csinode1
```

--------------------------------

### Build Python Wheel Distribution

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Packages the vineyard library into an installable Python wheel distribution after the main library has been successfully built from source.

```shell
python3 setup.py bdist_wheel
```

--------------------------------

### Share Objects Between Processes

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/getting-started.rst

Demonstrates passing data between producer and consumer processes using named objects in Vineyard.

```python
import multiprocessing as mp
import vineyard
import numpy as np
import pandas as pd

socket = '/var/run/vineyard.sock'

def produce(name):
   client = vineyard.connect(socket)
   client.put(pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD')),
              persist=True, name=name)

def consume(name):
   client = vineyard.connect(socket)
   print(client.get(name=name).sum())

if __name__ == '__main__':
   name = 'dataset'
   producer = mp.Process(target=produce, args=(name,))
   producer.start()
   consumer = mp.Process(target=consume, args=(name,))
   consumer.start()
   producer.join()
   consumer.join()
```

--------------------------------

### Store and Retrieve Data Objects

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/getting-started.rst

Shows how to store NumPy arrays and Pandas DataFrames in Vineyard memory and retrieve them using their unique object IDs. This process enables zero-copy data access.

```python
import numpy as np
object_id = client.put(np.random.rand(2, 4))
shared_array = client.get(object_id)

import pandas as pd
df = pd.DataFrame({'u': [0, 0, 1, 2, 2, 3], 'v': [1, 2, 3, 3, 4, 4], 'weight': [1.5, 3.2, 4.7, 0.3, 0.8, 2.5]})
object_id = client.put(df)
shared_dataframe = client.get(object_id)
```

--------------------------------

### vineyardctl inject command examples

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/cloud-native/vineyardctl.md

Demonstrates various ways to use the `vineyardctl inject` command. It shows how to output injected manifests in JSON format, pipe the output to `kubectl apply` for deployment, and how to enable resource creation during injection.

```shell
# use json format to output the injected workload
# notice that the output is a json string of all manifests
# it looks like:
# {
#   "workload": "workload json string",
#   "rpc_service": "rpc service json string",
#   "etcd_service": "etcd service json string",
#   "etcd_internal_service": [
#     "etcd internal service json string 1",
#     "etcd internal service json string 2",
#     "etcd internal service json string 3"
#   ],
#   "etcd_pod": [
#     "etcd pod json string 1",
#     "etcd pod json string 2",
#     "etcd pod json string 3"
#   ]
# }
vineyardctl inject -f workload.yaml -o json

# inject the default vineyard sidecar container into a workload
# output all injected manifests and then deploy them
vineyardctl inject -f workload.yaml | kubectl apply -f -

# if you only want to get the injected workload yaml rather than
# all manifests that includes the etcd cluster and the rpc service, 
# you can enable the apply-resources and then the manifests will be
# created during the injection, finally you will get the injected
# workload yaml
vineyardctl inject -f workload.yaml --apply-resources

```

--------------------------------

### Create a New Kedro Project from Template (Bash)

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/integration/kedro.md

This command initiates the creation of a new Kedro project using the 'pandas-iris' starter template. This is a standard way to bootstrap a Kedro project for experimentation or development.

```bash
$ kedro new --starter=pandas-iris
```

--------------------------------

### Install hosseinmoein-dataframe Headers (CMake)

Source: https://github.com/v6d-io/v6d/blob/main/modules/hosseinmoein-dataframe/CMakeLists.txt

Installs the header files from the third-party DataFrame directory into the vineyard contrib include path. This ensures that the DataFrame headers are available when the vineyard library is installed.

```cmake
install(DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/DataFrame/include/DataFrame
        DESTINATION include/vineyard/contrib    # target directory
        FILES_MATCHING                          # install only matched files
        PATTERN "*.h"                           # select header files
        PATTERN "*.hpp"                         # select C++ template header files
)
```

--------------------------------

### Configure Kubernetes Infrastructure for Pipelines

Source: https://github.com/v6d-io/v6d/blob/main/docs/tutorials/kubernetes/efficient-data-sharing-in-kubeflow-with-vineyard-csi-driver.rst

Setup commands to create namespaces, apply data volumes, and configure RBAC policies. These are essential prerequisites for running pipeline workloads in a Kubernetes cluster.

```bash
kubectl create namespace kubeflow
kubectl apply -f prepare-data.yaml
kubectl apply -f rbac.yaml
```

--------------------------------

### Install CMake Configuration Files

Source: https://github.com/v6d-io/v6d/blob/main/CMakeLists.txt

This CMake command installs generated and provided CMake configuration files to the installation directory. These files are crucial for enabling other projects to find and use the v6d library.

```cmake
configure_file("${PROJECT_SOURCE_DIR}/vineyard-config.in.cmake"
               "${PROJECT_BINARY_DIR}/vineyard-config.cmake" @ONLY
)
configure_file("${PROJECT_SOURCE_DIR}/vineyard-config-version.in.cmake"
               "${PROJECT_BINARY_DIR}/vineyard-config-version.cmake" @ONLY
)
install(FILES "${PROJECT_BINARY_DIR}/vineyard-config.cmake"
              "${PROJECT_BINARY_DIR}/vineyard-config-version.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindArrow.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/CheckGCCABI.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindFUSE3.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindGFlags.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindGlog.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindGperftools.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindLibUnwind.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindRdkafka.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/FindPythonExecutable.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/GenerateVineyard.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/GenerateVineyardJava.cmake"
              "${PROJECT_SOURCE_DIR}/cmake/DetermineImplicitIncludes.cmake"
              DESTINATION lib/cmake/vineyard
)
install(EXPORT vineyard-targets
        FILE vineyard-targets.cmake
        DESTINATION lib/cmake/vineyard
)
```

--------------------------------

### Upload Dataset to OSS via ossutil

Source: https://github.com/v6d-io/v6d/blob/main/docs/tutorials/kubernetes/vineyard-on-fluid.rst

Command-line instruction to upload the serialized dataset file to an Alibaba Cloud OSS bucket using the ossutil tool.

```bash
ossutil cp ./df.pkl oss://your-bucket-name/your-path
```

--------------------------------

### Install Dask Cluster using Helm

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/cloud-native/vineyard-operator.rst

Installs a Dask scheduler and workers using Helm. Ensure you have Helm installed and the Dask Helm repository added. The Dask worker image must be built with Vineyard support.

```bash
# install dask scheduler and dask worker.
$ helm repo add dask https://helm.dask.org/
$ helm repo update
```

--------------------------------

### Create New Kedro Project from Template

Source: https://github.com/v6d-io/v6d/blob/main/python/vineyard/contrib/kedro/README.md

This command initiates the creation of a new Kedro project using the 'pandas-iris' starter template. This is a common step when setting up a new Kedro project, especially for demonstration or testing purposes, before building Docker images and deploying to Kubernetes.

```bash
kedro new --starter=pandas-iris
```

--------------------------------

### Install Vineyard Airflow Provider

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/integration/airflow.rst

Installs the necessary Python package to enable Vineyard integration within an Airflow environment.

```bash
pip3 install airflow-provider-vineyard
```

--------------------------------

### Deploy Vineyardd on Kubernetes using vineyardctl

Source: https://github.com/v6d-io/v6d/blob/main/k8s/cmd/README.md

Demonstrates how to deploy the Vineyard daemon on a Kubernetes cluster. Examples cover default deployments, waiting for readiness, using custom images, and configuring persistent storage spill mechanisms via command-line flags and external configuration files.

```shell
# deploy the default vineyard on kubernetes
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy vineyardd

# not to wait for the vineyardd to be ready
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy vineyardd --wait=false

# deploy the vineyardd from a yaml file
vineyardctl --kubeconfig $HOME/.kube/config deploy vineyardd --file vineyardd.yaml

# deploy the vineyardd with customized image
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy vineyardd --image vineyardd:v0.12.2

# deploy the vineyardd with spill mechanism on persistent storage from json string
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy vineyardd \
  --vineyardd.spill.config spill-path \
  --vineyardd.spill.path /var/vineyard/spill \
  --vineyardd.spill.pv-pvc-spec '{
    "pv-spec": {
      "capacity": {
        "storage": "1Gi"
      },
      "accessModes": ["ReadWriteOnce"],
      "storageClassName": "manual",
      "hostPath": {"path": "/var/vineyard/spill"}
    },
    "pvc-spec": {
      "storageClassName": "manual",
      "accessModes": ["ReadWriteOnce"],
      "resources": {"requests": {"storage": "512Mi"}}
    }
  }'

# deploy the vineyardd with spill mechanism on persistent storage from yaml string
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy vineyardd \
  --vineyardd.spill.config spill-path \
  --vineyardd.spill.path /var/vineyard/spill \
  --vineyardd.spill.pv-pvc-spec '
pv-spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: manual
  hostPath:
    path: "/var/vineyard/spill"
pvc-spec:
  storageClassName: manual
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 512Mi'

# deploy the vineyardd with spill mechanism on persistent storage from json file
cat pv-pvc.json | vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy vineyardd \
  --vineyardd.spill.config spill-path \
  --vineyardd.spill.path /var/vineyard/spill -
```

--------------------------------

### Launch Vineyard Server Locally

Source: https://github.com/v6d-io/v6d/blob/main/python/vineyard/contrib/kedro/README.md

Starts a local vineyard server instance. This command is used to set up the vineyard service that Kedro will connect to for data sharing. The socket path can be configured as needed.

```bash
python3 -m vineyard --socket=/tmp/vineyard.sock
```

--------------------------------

### Registering and Dispatching I/O Drivers in Python

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/key-concepts/io-drivers.rst

Demonstrates the internal dispatching logic of the vineyard 'open' function and how drivers are registered for specific URL schemes. It shows the use of a decorator '@registerize' and a '.register' method for associating a driver function with a scheme.

```python
>>> @registerize
>>> def open(path, *args, **kwargs):
>>>     scheme = urlparse(path).scheme

>>>     for reader in open._factory[scheme][::-1]:
>>>         r = reader(path, *args, **kwargs)
>>>         if r is not None:
>>>             return r
>>>     raise RuntimeError('Unable to find a proper IO driver for %s' % path)
>>>
>>> # different driver functions are registered as follows
>>> open.register('file', local_driver)
```

--------------------------------

### Start Vineyard CSI Driver

Source: https://github.com/v6d-io/v6d/blob/main/k8s/cmd/README.md

Starts the vineyard CSI driver. Requires specifying the endpoint and node ID. It manages the state file path.

```shell
vineyardctl csi [flags]

# Example:
vineyardctl csi --endpoint=unix:///csi/csi.sock --nodeid=csinode1
```

--------------------------------

### Start Vineyard Manager with vineyardctl

Source: https://github.com/v6d-io/v6d/blob/main/k8s/cmd/README.md

Commands to start the vineyard operator's manager component. Allows enabling or disabling webhooks and schedulers. Default configuration enables both.

```shell
vineyardctl manager
vineyardctl manager --enable-webhook=false
vineyardctl manager --enable-scheduler=false
vineyardctl manager --enable-webhook=false --enable-scheduler=false
```

--------------------------------

### Generate Project Documentation

Source: https://github.com/v6d-io/v6d/blob/main/CONTRIBUTING.rst

Builds the project documentation locally using Sphinx and Doxygen, then opens the resulting HTML files in a browser.

```bash
cd docs/
make html
open _build/html/index.html
```

--------------------------------

### Run FIO on Kubernetes Pods for DFS Bandwidth Testing (Bash)

Source: https://github.com/v6d-io/v6d/blob/main/modules/llm-cache/tests/scripts/README.md

This script executes FIO on Kubernetes pods to measure the average bandwidth of a distributed file system. It requires FIO to be installed on the pods and the DFS to be mounted. The script takes several arguments to configure the test, including pod labels, FIO action type, number of jobs, maximum pods, target directory, and block size. The output provides insights into the read/write performance of the DFS.

```bash
bash dfs-fio.sh 
Usage: dfs-fio.sh -l <pod-label> -a <fio-action> -j <num-jobs> -m <max-pods> -d <directory> -b <block-size>
  -l <pod-label>     Pod label for filtering specific Pods
  -a <fio-action>    FIO operation type: read, write, randread, randwrite
  -j <num-jobs>      Number of parallel jobs in FIO
  -m <max-pods>      Limit on the maximum number of Pods, should not exceed the total number of Pods
  -d <directory>     Directory path for FIO testing
  -b <block-size>    Block size for the FIO test
```

```bash
bash dfs-fio.sh -l your_pod_label_key=your_pod_label_value -a read -j 1 -m 1 -d /your/dfs/mountpath -b 4992k -n default
```

```bash
bash dfs-fio.sh -l your_pod_label_key=your_pod_label_value -a write -j 1 -m 10 -d /your/dfs/mountpath -b 4992k -n default
```

```bash
bash dfs-fio.sh -l your_pod_label_key=your_pod_label_value -a randread -j 1 -m 10 -d /your/dfs/mountpath -b 4992k -n default
```

--------------------------------

### Install vineyard Target and Headers (CMake)

Source: https://github.com/v6d-io/v6d/blob/main/modules/hosseinmoein-dataframe/CMakeLists.txt

Installs the 'vineyard_hosseinmoein_dataframe' target and all vineyard headers. This makes the compiled library and its associated headers available for use in other projects that link against vineyard.

```cmake
install_export_vineyard_target(vineyard_hosseinmoein_dataframe)
install_vineyard_headers("${CMAKE_CURRENT_SOURCE_DIR}")
```

--------------------------------

### Install CRDs and Kubernetes Roles for Vineyard Operator

Source: https://github.com/v6d-io/v6d/blob/main/k8s/README.md

This command installs the necessary Custom Resource Definitions (CRDs) and Kubernetes cluster roles required for the Vineyard Operator to function. It is a prerequisite for deploying the operator.

```bash
make predeploy
```

--------------------------------

### Schedule Workload YAML with PodAffinity and Socket Volume

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/cloud-native/vineyardctl.md

This example demonstrates how to schedule a Kubernetes Deployment to a vineyard cluster. It shows the original workload YAML and the resulting YAML after applying the `vineyardctl schedule workload` command, which adds pod affinity and a socket volume mount.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-client
  # Notice, you must set the namespace here
  namespace: vineyard-job
spec:
  selector:
    matchLabels:
      app: python
  template:
    metadata:
      labels:
        app: python
    spec:
      containers:
      - name: python
        image: python:3.10
        command: ["python", "-c", "import time; time.sleep(100000)"]
```

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  name: python-client
  namespace: vineyard-job
spec:
  selector:
    matchLabels:
      app: python
  strategy: {}
  template:
   metadata:
      creationTimestamp: null
      labels:
        app: python
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - vineyard-system-vineyardd-sample
            namespaces:
            - vineyard-system
            topologyKey: kubernetes.io/hostname

      containers:
      - command:
        - python
        - -c
        - import time; time.sleep(100000)
        env:
        - name: VINEYARD_IPC_SOCKET
          value: /var/run/vineyard.sock
        image: python:3.10
        name: python
        resources: {}
        volumeMounts:
        - mountPath: /var/run
          name: vineyard-socket
      volumes:
      - hostPath:
          path: /var/run/vineyard-kubernetes/vineyard-system/vineyardd-sample
        name: vineyard-socket
```

--------------------------------

### Install and Connect to Vineyard

Source: https://github.com/v6d-io/v6d/blob/main/docs/_templates/index.html

This snippet demonstrates how to install the vineyard Python client using pip and establish a connection to the vineyard service. It then shows how to put an object into vineyard and retrieve it.

```python
pip install vineyard
Successfully installed vineyard python client

import vineyard

client = vineyard.connect()
object_id = client.put('Hello, vineyard!')
client.get(object_id)
# Output: 'Hello, vineyard!'
```

--------------------------------

### Deploy vineyardd on Kubernetes using vineyardctl

Source: https://github.com/v6d-io/v6d/blob/main/k8s/cmd/README.md

Demonstrates how to deploy a vineyard cluster on Kubernetes using the vineyardctl CLI. Includes examples for default deployments and deployments with custom container images.

```shell
# deploy the default vineyard deployment on kubernetes
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config \
deploy vineyard-deployment

# deploy the vineyard deployment with customized image
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config \
deploy vineyard-deployment --image vineyardcloudnative/vineyardd:v0.12.2
```

--------------------------------

### Install v6d Dependencies on macOS

Source: https://github.com/v6d-io/v6d/blob/main/docs/notes/developers/build-from-source.rst

Installs the necessary dependencies for building vineyard on macOS using Homebrew. This includes Apache Arrow, boost, gflags, glog, grpc, protobuf, LLVM, MPICH, OpenSSL, and zlib.

```shell
brew install apache-arrow boost gflags glog grpc protobuf llvm mpich openssl zlib autoconf
```

--------------------------------

### Manage Hive Table Partitioning

Source: https://github.com/v6d-io/v6d/blob/main/java/hive/README.rst

Demonstrates how to create and manipulate Hive tables using both static and dynamic partitioning. These operations include inserting data into specific partitions and querying the resulting table structure.

```sql
create table hive_static_partition(src_id int, dst_id int) partitioned by (value int);
insert into table hive_static_partition partition(value=666) values (3, 4);
insert into table hive_static_partition partition(value=666) values (999, 2), (999, 2), (999, 2);
insert into table hive_static_partition partition(value=114514) values (1, 2);
select * from hive_static_partition;
select * from hive_static_partition where value=666;
select * from hive_static_partition where value=114514;
drop table hive_static_partition;

create table hive_dynamic_partition_data(src_id int, dst_id int, year int) stored as TEXTFILE location "file:///opt/hive/data/warehouse/hive_dynamic_partition_data";
insert into table hive_dynamic_partition_data values (1, 2, 2018),(3, 4, 2018),(1, 2, 2017);
create table hive_dynamic_partition_test(src_id int, dst_id int) partitioned by(mounth int, year int);
insert into table hive_dynamic_partition_test partition(mounth=1, year) select src_id,dst_id,year from hive_dynamic_partition_data;
select * from hive_dynamic_partition_test;
drop table hive_dynamic_partition_test;
drop table hive_dynamic_partition_data;
```