### Enable and Start Ceph MDS Service

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Enables the Ceph Metadata Server (MDS) service to start automatically on boot and immediately starts the service for the specified MDS ID. Assumes systemd is used.

```bash
sudo systemctl enable --now ceph-mds@$mdsid.service
```

--------------------------------

### Ceph Manager Setup and Activation

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This snippet details the steps to set up a Ceph manager daemon, including creating an access key, running the manager in debug mode, and enabling/starting the systemd service. It's crucial for offloading work from monitors and enabling scaling.

```bash
sudo -u ceph mkdir -p /var/lib/ceph/mgr/ceph-$mgrid
ceph auth get-or-create mgr.$mgrid mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-$mgrid/keyring

# test it:
sudo -u ceph ceph-mgr -i $mgrid -d

# actually activate it
sudo systemctl enable --now ceph-mgr@$mgrid.service
```

--------------------------------

### Ceph Monitor Configuration Example

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

A Ceph configuration setting to prevent operations if too many OSDs are out. This ensures data availability by enforcing a minimum ratio of operational OSDs.

```bash
# if too many osds are out, don't operate.
mon osd min in ratio = 100
```

--------------------------------

### Fstab Entry for RBD Mount

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

An example fstab entry for mounting an RBD device. It uses the `noauto` option, requiring manual mounting via `rbdmap.service` or a custom script. The `stripe=1024` option is recommended for ext4 filesystems.

```bash
/dev/rbd/$metadata_pool_name/$namespacename/$imagename /your/$mountpoint $filesystem defaults,noatime,noauto 0 0
/dev/yourvg/yourlv /your/$mountpoint $filesystem defaults,noatime,noauto 0 0
```

--------------------------------

### RBD: Automatic Mapping Configuration

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Example configuration line for `/etc/ceph/rbdmap` to automatically map an RBD image using a specific user's keyring. It specifies the image details and user credentials.

```bash
$poolname/$namespacename/$imagename name=client.$username,keyring=/etc/ceph/ceph.client.$username.keyring
```

--------------------------------

### Get CRUSH Tunables

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Retrieve the current CRUSH tunable profile of the cluster. This helps in understanding the current data distribution and balancing strategy.

```shell
ceph osd crush show-tunables
```

--------------------------------

### Define CRUSH Rule for Device Selection in Ceph

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Defines a CRUSH rule in Ceph that specifies how data is distributed across OSDs. This example rule targets 'ssd' class devices, selects hosts, and then emits the chosen OSDs for replicated storage. It includes steps for taking buckets, choosing leaves, and emitting.

```ceph
rule rulename {
    # unique id
    id 1
    type replicated
    # which pools can use this rule?
    min_size 1    # -> all pools
    max_size 10

    # now the device selection steps
    # start at the default bucket, but only for ssd devices
    step take default class ssd

    # chooseleaf firstn: recursively explore bucket to look for single devices
    # choose firstn: select bucket for next step
    # 0: choose as many buckets as needed for copies (-1: one less than needed, 3: exactly three)
    # host: bucket type to choose for the next step
    step chooseleaf firstn 0 type host

    # the set of osds was selected
    step emit
}
```

--------------------------------

### Get Information for a Specific CephFS Filesystem

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Retrieves detailed configuration and status information for a particular CephFS filesystem, identified by its name.

```shell
ceph fs get lolfs
```

--------------------------------

### Get OSD Metadata

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Retrieves metadata for a specific OSD, including information about its associated block devices.

```bash
# See which blockdevices the OSD uses & more
# see links in `/dev/mapper/` and `lsblk` to correlate ids with blockdevices etc
ceph osd metadata [$osdid]
```

--------------------------------

### Ceph Pool Configuration for Data Safety (Replica and EC)

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This section details recommended Ceph pool configurations for data safety, focusing on replica and erasure coding (EC) settings. It explains the importance of `min_size` to ensure data redundancy even during OSD failures, providing specific examples for replica and EC pools.

```bash
# View current pool details including size and min_size
ceph osd pool ls detail

# Example: Setting replica pool with size 3 and min_size 2
# ceph osd pool create <pool_name> <pg_num> replicated
# ceph osd pool set <pool_name> size 3
# ceph osd pool set <pool_name> min_size 2

# Example: Setting erasure coded pool (e.g., k=2, m=1 => total 3 chunks)
# ceph osd erasure-code-profile create <profile_name> k=2 m=1
# ceph osd pool create <pool_name> <pg_num> erasure <profile_name>
# ceph osd pool set <pool_name> min_size 2 # For k=2, m=1, min_size should be k+(m-i) where i=1, so 2+(1-1)=2
```

--------------------------------

### Control and Inspect Ceph Daemons

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Commands to interact with running Ceph daemons using their admin socket. This includes getting help, performance info, configuration differences, monitor status, and daemon performance.

```shell
ceph daemon $daemonid help

# performance info
ceph daemon $daemonid perf dump

# config differences
ceph daemon $daemonid config diff

# monitor status
ceph daemon mon.$id mon_status

# daemon performance watch
ceph daemonperf $daemontype.$mdsid

# mds sessions
ceph daemon mds.$id session ls

# view cache usage
ceph daemon mds.$id cache status
```

--------------------------------

### Get Ceph Auth Key Permissions

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Retrieves the effective permissions of a Ceph authorization key for a given client.

```bash
ceph auth get client.some.weird_name
```

--------------------------------

### Configure CephFS Client Capabilities

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets advanced capabilities for a CephFS client, including flags for managing quotas and layouts ('p') and snapshot permissions ('s'). Existing caps are overwritten, so fetching them first is recommended.

```bash
ceph auth caps client.lolroot mon 'allow r' mds 'allow rwp' osd 'allow rw tag cephfs data=cephfsname'
```

```bash
ceph auth caps client.blabla ... mds 'allow rw path=/some/dir, allow rws path=/some/dir/snapshots_allowed, ...' ...
```

--------------------------------

### Configure VirtIO-SCSI for VM Disks

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This tip recommends using `virtio-scsi` over `virtio-blockdevice` for virtual machine storage, enabling discard/unmapping. This provides better performance and allows for more efficient space management on the underlying Ceph storage by supporting TRIM operations.

```xml
<!-- Example QEMU/libvirt XML configuration for virtio-scsi -->
<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' discard='unmap'/>
  <source dev='/dev/rbd/pool_name/image_name'/>
  <target dev='sda' bus='scsi'/>
  <controller type='scsi' model='virtio-scsi'/>
  <address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
```

--------------------------------

### Create CephFS Snapshot

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Creates a snapshot in CephFS by creating a directory with a '.snap' prefix. This requires the client to have 's' permission for MDS.

```bash
dir/to/backup/.snap/snapshot_name
```

--------------------------------

### Create BlueStore OSD with LVM and Encryption

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This command creates a new BlueStore OSD using LVM for device management. It supports optional dm-crypt for HDD encryption and allows specifying a CRUSH device class. The data, WAL, and DB can be on separate devices.

```bash
sudo ceph-volume lvm create --dmcrypt --data /dev/partition
```

--------------------------------

### Initialize RBD Pool

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Prepares a Ceph pool for RBD usage, making it ready to store RBD images and metadata.

```bash
rbd pool init $pool_name
```

--------------------------------

### Create CephFS Volume

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Creates a new CephFS volume using the `ceph fs volume create` command. This is the modern approach to setting up CephFS, often simplifying the process compared to manually creating pools and mounting.

```bash
ceph fs volume create ...
```

--------------------------------

### FIO: Synced Writes with Caching

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This fio command tests cached writes that are synchronized to disk (`sync=1`). It uses the I/O scheduler (`direct=0`) but limits `iodepth` to 1 to ensure each write is synced individually, which is slow but guarantees data durability.

```bash
fio --filename=/mnt/rbd-fs/file --size=20G --direct=0 --sync=1 --iodepth=1 --runtime=15 --ioengine=libaio --time_based --rw=rw --bs=64K --numjobs=1 --group_reporting --name=test
```

--------------------------------

### FIO: Basic Read/Write Test

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This fio command performs a basic sequential read/write test on an RBD device. It tests without caching (`direct=0`) and with asynchronous writes (`sync=0`). The `iodepth` is set to 1024, allowing for significant I/O queue depth.

```bash
fio --filename=/mnt/rbd-fs/file --size=20G --direct=0 --sync=0 --iodepth=1024 --runtime=15 --ioengine=libaio --time_based --rw=rw --bs=64K --numjobs=1 --group_reporting --name=test
```

--------------------------------

### Create Metadata Server (MDS) Keyring and Directory

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets up the necessary directory structure and keyring for a Ceph Metadata Server (MDS). It generates a key for the MDS and adds it to the Ceph cluster's authentication system.

```bash
sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-$mdsid
sudo -u ceph ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-$mdsid/keyring --gen-key -n mds.$mdsid
sudo -u ceph ceph auth add mds.$mdsid osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-$mdsid/keyring
```

--------------------------------

### Activate All Ceph OSDs

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Enables Ceph to discover and activate all attached OSD disks automatically. This command also creates necessary systemd service files for OSD startup on subsequent boots, facilitating stateless OSD hosts.

```bash
sudo ceph-volume lvm activate --all
```

--------------------------------

### Benchmark Raw Device Write Speed

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Uses `fio` to benchmark the sequential write performance of a raw device, specifically for 4k blocks with sync enabled. Helps assess if a device meets performance expectations for Ceph.

```bash
# 4k sequential write with sync
fio --filename /dev/device --numjobs=1 --direct=1 --fdatasync=1 --ioengine=pvsync --iodepth=1 --runtime=20 --time_based --rw=write --bs=4k --group_reporting --name=ceph-iops
```

--------------------------------

### Set CRUSH Tunables to Optimal

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Apply the 'optimal' CRUSH tunables profile to the cluster. This aims to balance data distribution and performance according to Ceph's recommendations.

```shell
ceph osd crush tunables optimal
```

--------------------------------

### Export and Import PG Data with ceph-objectstore-tool

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Utilizes `ceph-objectstore-tool` to export PG data from an OSD for backup or analysis, and to import it back. Also mentions 'trim-pg-log' and compacting Bluestore.

```bash
# turn off the OSD so we can work on its store directly!

# export a pg from an OSD
# to delete it from OSD: --op export-remove ...
ceph-objectstore-tool --op export --data-path /var/lib/ceph/osd/ceph-$id --pgid $pgid --file $pgid-bup-osd$id

# import into an OSD:
ceph-objectstore-tool --op import --data-path /var/lib/ceph/osd/ceph-$id --file saved-pg-dump

# other useful ops: trim-pg-log

# compact bluestore:
ceph-kvstore-tool <rocksdb|bluestore-kv> /var/lib/ceph/osd/ceph-$id compact
```

--------------------------------

### View BlueFS Block Device Sizes

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Employs ceph-bluestore-tool to display the sizes of the block devices associated with BlueFS for a specified OSD path. This helps in understanding current storage allocation.

```bash
ceph-bluestore-tool --command bluefs-stats --path /var/lib/ceph/osd/ceph-$i
```

--------------------------------

### Migrate OSD Journal (WAL) to New Device

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Uses ceph-bluestore-tool to create a new WAL device for an OSD, migrating data from an all-in-one OSD to a new target partition. This command targets the block.wal component.

```bash
ceph-bluestore-tool --command bluefs-bdev-new-wal --dev-target /dev/system/osdwal$id --path /var/lib/ceph/osd/ceph-$id
```

--------------------------------

### Minimal Ceph Client Configuration

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

The essential configuration required for Ceph clients, such as mounting CephFS or mapping RBD devices. This typically includes the `fsid` and monitor host addresses.

```ini
[global]
fsid = your-fs-id
mon_host = mon1.rofl.lol, mon2.rofl.lol, mon3.rofl.lol, ...
```

--------------------------------

### Create Ceph Access Key for Pools

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Generates a Ceph authentication key with read-only access to a specific pool and read-write access to metadata and storage pools.

```bash
ceph auth get-or-create client.$name mon 'profile rbd' osd 'profile rbd pool=$metadata_pool_name, profile rbd pool=$storage_pool_name, profile rbd-read-only pool=$someotherpoolname'
```

--------------------------------

### Create Ceph Erasure Coding Profile

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Creates a new erasure coding profile for Ceph. The `k` parameter specifies the number of data chunks, `m` specifies the number of parity chunks, and `crush-failure-domain` ensures chunks are placed on different OSDs.

```bash
ceph osd erasure-code-profile set standard_8_2 k=8 m=2 crush-failure-domain=osd
```

--------------------------------

### Test CRUSH Rule Mappings in Ceph

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Tests the effectiveness of a CRUSH rule by simulating data placement. It takes the crushmap binary, a rule ID, and the number of replicas, then shows which OSDs would be selected for placement, including options to show bad mappings.

```bash
crushtool -i crushmap.bin --test --rule $ruleid --num-rep $reps --show-mappings
# other options: --show-bad-mappings
```

--------------------------------

### Configure LVM on RBD

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Add RBD device type to LVM configuration to allow creating LVM physical volumes directly on RBD devices. This requires modifying `/etc/lvm/lvm.conf`.

```shell
types=["rbd", 255]
```

--------------------------------

### List RBD Namespaces

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Displays all available namespaces within a specified RBD pool.

```bash
rbd --pool $metadata_pool_name namespace ls
```

--------------------------------

### Enable and Configure Ceph PG Autoscaler

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Enables the `pg_autoscaler` manager module in Ceph for automatic placement group management. It also shows how to view autoscaler status and set global or per-pool policies for PG autoscaling, recommending a 'warn' mode.

```bash
ceph mgr module enable pg_autoscaler

# view autoscale information and what the autoscaler would do
ceph osd pool autoscale-status

# policy for newly created pools
# I recommend setting warn, and _not_ on.
ceph config set global osd_pool_default_pg_autoscale_mode <mode>

# policy per-pool
# warn, on or off. recommended by me: warn.
ceph osd pool set $pool pg_autoscale_mode <mode>
```

--------------------------------

### Create RBD Image

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Creates a new RBD image with specified size, storage pool, namespace, and metadata pool.

```bash
rbd create --pool $metadata_pool_name --data-pool $storage_pool_name --namespace $namespacename --size 20G $imagename
```

--------------------------------

### RBD: Resizing and QoS Configuration

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Commands for resizing RBD images and configuring Quality of Service (QoS) limits at both the pool and image level. This includes setting IOPS limits and removing them.

```bash
rbd resize --size 9001T $pool/$img
rbd resize --size 20M $pool/$img --allow-shrink
rbd config pool set $pool rbd_qos_iops_limit 1000
rbd config pool rm $pool rbd_qos_iops_limit
rbd config image set $pool/$img rbd_qos_iops_limit 1000
rbd config image rm $pool/$img rbd_qos_iops_limit
```

--------------------------------

### Set OSD Minimum Client Compatibility

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Forces Ceph clients to meet a minimum compatibility level, specifically 'luminous' in this case, even if the client might report older capabilities. Use with caution.

```bash
ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it
```

--------------------------------

### Dump All CephFS Filesystem Information

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Outputs detailed information about all configured CephFS filesystems, useful for in-depth analysis and debugging.

```shell
ceph fs dump
```

--------------------------------

### Migrate OSD Database (DB) to New Device

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Utilizes ceph-bluestore-tool to create a new DB device for an OSD, migrating both the block.wal and hot RocksDB data to a new target partition. This command is for migrating the block.db.

```bash
ceph-bluestore-tool --command bluefs-bdev-new-db --dev-target /dev/system/osdwal$id --path /var/lib/ceph/osd/ceph-$id
```

--------------------------------

### Set Default PG Autoscaler Mode

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets the default PG autoscaler mode (`on`, `warn`, `off`) for newly created pools.

```bash
ceph config set global osd_pool_default_pg_autoscale_mode $mode
```

--------------------------------

### RBD: Basic Management Commands

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

A collection of essential rbd commands for managing block devices, including listing images, showing information, checking status, and managing pending deleted images.

```bash
rbd namespace ls $pool
rbd ls $pool/$namespace
rbd ls $pool
rbd info $pool/$namespace/$image
rbd status $pool/$namespace/$image
rbd trash ls [$pool]
```

--------------------------------

### FIO: Non-Cached Synced Writes

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This fio command tests non-cached writes (`direct=1`) with synchronization (`sync=1`). It allows a high `iodepth` of 512, potentially enabling parallel synchronized writes when bypassing the cache.

```bash
fio --filename=/mnt/rbd-fs/file --size=20G --direct=1 --sync=1 --iodepth=1 --runtime=15 --ioengine=libaio --time_based --rw=rw --bs=64K --numjobs=1 --group_reporting --name=test
```

--------------------------------

### Create Erasure Coded Data Pool and Replicated Metadata Pools

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets up a CephFS environment by creating an erasure-coded pool for data storage and replicated pools for metadata. It also configures `allow_ec_overwrites` for the data pool and sets `size` and `min_size` for the replicated pools.

```bash
# erasure coding pool (for data)
ceph osd pool create lol_data 32 32 erasure standard_8_2
ceph osd pool set lol_data allow_ec_overwrites true

# replicated pools (for metadata)
ceph osd pool create lol_root 32 replicated
ceph osd pool create lol_metadata 32 replicated

# min_size: minimal osd count (per PG) before a PG goes offline
ceph osd pool set lol_root size 3
ceph osd pool set lol_root min_size 2
ceph osd pool set lol_metadata size 3
ceph osd pool set lol_metadata min_size 2
```

--------------------------------

### Restart/Repeer PG and Check PG Status

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Commands to force a PG to re-establish connections between OSDs and to query the status and peers of a specific PG. Useful for PGs stuck in states like 'active+remapped'.

```bash
ceph pg repeer $pgid
ceph pg $pgid query
```

--------------------------------

### Map RBD Image to Device

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Maps a Ceph RBD image to a block device on a client system using monitor addresses from ceph.conf. Supports NBD mapping and namespace specification.

```bash
sudo rbd device map --name client.$name -k keyring [$metadata_pool_name/[$namespacename/]]$imagename[@$snapshotname]
```

```bash
# -t nbd to mount as nbd-device
sudo rbd device map --name client.$name -k keyring [$metadata_pool_name/[$namespacename/]]$imagename[@$snapshotname] -t nbd
```

```bash
# --namespace $namespacename to specify the rbd namespace
sudo rbd device map --name client.$name -k keyring [$metadata_pool_name/[$namespacename/]]$imagename[@$snapshotname] --namespace $namespacename
```

--------------------------------

### RBD: Performance Monitoring

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Command to display performance statistics for RBD pool images, providing insights into I/O operations.

```bash
rbd perf image iotop --pool $pool
```

--------------------------------

### List Ceph Pools with Detail and Filter with jq

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Lists all Ceph pools with detailed information in JSON format and pipes the output to `jq` for pretty-printing and potential filtering. This helps in understanding pool configurations, especially assigned erasure coding profiles.

```bash
ceph osd pool ls detail --format json | jq -C .
```

--------------------------------

### Separate OSD Database and Bulk Storage

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Demonstrates how to configure Ceph OSDs to store the metadata (block.db) and journal (block.wal) on faster devices (like SSDs) while keeping the main data on slower devices (like HDDs).

```bash
ceph-volume lvm create --data /dev/slowdevice --block.db /dev/fastdevice
```

```bash
ceph-volume lvm create --data /dev/slowdevice --block.wal /dev/fastdevice
```

--------------------------------

### Create Ceph Access Key for Namespace

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Creates a Ceph authentication key restricted to a specific RBD namespace on both storage and metadata pools. Essential for security to prevent cross-namespace access.

```bash
ceph auth get-or-create client.$name mon 'profile rbd' osd 'profile rbd pool=$metadata_pool_name namespace=$namespacename, profile rbd pool=$storage_pool_name namespace=$namespacename'
```

--------------------------------

### Assign Erasure Coded Pool to a Directory in CephFS

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Creates a new erasure coding profile (`backup_8_3`) and a corresponding pool (`lol_backup`). It then assigns this pool to a specific directory in CephFS using `setfattr`, ensuring new content within that directory is stored with the defined erasure coding scheme.

```bash
# for example, this 8+3 pool can be to store some directories 'more safe'
ceph osd erasure-code-profile set backup_8_3 k=8 m=3 crush-failure-domain=osd
ceph osd pool create lol_backup 64 64 erasure backup_8_3
ceph osd pool set lol_backup allow_ec_overwrites true

# in the cephfs, assign it to a directory and all its _new_ content:
setfattr -n ceph.dir.layout.pool -v lol_backup your-backup-directory-name
```

--------------------------------

### Configure Ceph RBD Read Ahead

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets the maximum number of bytes that can be pre-read for Ceph RBD devices. This is achieved by writing to the `read_ahead_kb` file in the block device's sysfs directory. The actual number of objects read is determined dynamically by the filesystem.

```bash
ACTION=="add", KERNEL=="rbd*", ATTR{bdi/read_ahead_kb}="32768"
```

```bash
echo 32768 > /sys/block/rbdxxx/queue/read_ahead_kb
```

--------------------------------

### Hint Ceph Autoscaler with Target Pool Size Ratio

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Provides a hint to the Ceph PG autoscaler about the expected size of a pool relative to the total cluster size by setting `target_size_ratio`. It also demonstrates how to set the autoscaler mode to 'warn' or 'on' for a specific pool.

```bash
# help the autoscaler by providing target_ratio:
# the fraction of total cluster size this pool is expected to consume.
ceph osd pool set foo target_size_ratio .2

# only warn that the pg-count is suboptimal
ceph osd pool set $poolname pg_autoscale_mode warn

# enable automatic pg adjustments on the given pool
ceph osd pool set $poolname pg_autoscale_mode on
```

--------------------------------

### Dump Ceph Configuration

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Dumps the current configuration of Ceph daemons. Useful for reviewing settings and troubleshooting.

```bash
# ceph central configs
ceph config dump
```

--------------------------------

### List Inconsistent Objects and Repair PG

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Commands to identify PGs with inconsistent objects due to data corruption (bitrot, drive issues, bugs) and initiate the repair process. Requires pool name and PG ID.

```bash
rados list-inconsistent-pg $poolname
rados list-inconsistent-obj $pgid
rados list-inconsistent-snapset $pgid
ceph health detail
ceph pg repair $pgid
```

--------------------------------

### Reformat and Reset an OSD

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Resets an OSD's on-disk data structures by wiping the block device and creating a new BlueStore filesystem. This process preserves LVM and encryption configurations while effectively re-initializing the OSD.

```bash
# observe and decide that OSD $osdid needs an reset

# go to the osd directory
cd /var/lib/ceph/osd/ceph-$osdid

# clear the bluestore header
dd if=/dev/zero of=./block count=1 bs=100MB

# create a new bluestore filesystem
sudo ceph-osd -f --id $osdid --setuser ceph --setgroup ceph --mkfs

# start the service again
sudo systemctl start ceph-osd@$osdid.service
```

--------------------------------

### Mount ext4 Filesystem with Specified Stripe Width

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Mounts an ext4 filesystem residing on a Ceph RBD device, specifying the `stripe` option to ensure optimal data distribution across RADOS objects.

```bash
mount -o defaults,noauto,stripe=1024 /dev/rbd/your/rbd /your/mountpoint
```

--------------------------------

### Monitor IO Scheduler Merges with iostat

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Uses `iostat` to display disk I/O statistics, including read and write merge requests (`rrqm`/`wrqm`). This helps in understanding how the IO scheduler is merging adjacent IO operations when not using `O_DIRECT`.

```bash
iostat -xm 2
```

--------------------------------

### Manual RBD Mapping

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Command to manually map an RBD device by writing its details to the sysfs interface. This requires the monitor IP, username, secret, pool, and image name.

```bash
echo $ceph_monitor_ip name=$username,secret=$secret,queue_depth=512 $pool $image > /sys/bus/rbd/add
```

--------------------------------

### Enable Ceph Crash Collection Module

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Enables the crash collection module within Ceph Manager and sets up the necessary authentication for crash reporting. This allows the `ceph-crash.service` to submit crash data to the manager.

```bash
ceph mgr module enable crash
ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash'

# or, restrict the crash reports to a specific subnet!
ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash network 1337.42.42.0/24'
```

--------------------------------

### Set OSD Memory Target in ceph.conf

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Demonstrates how to set the OSD memory target directly within a host's `ceph.conf` file. This is useful for testing configurations before applying them cluster-wide.

```ini
[osd]
osd_memory_target = 4294967296   # 4GiB
```

--------------------------------

### FIO: Cached Read Test

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This fio command tests cached reads (`direct=0`) using the I/O scheduler and cache. It submits 512 read requests, but performance might be limited due to cache interactions.

```bash
fio --filename=/mnt/rbd-fs/file --size=20G --direct=0 --sync=0 --iodepth=512 --runtime=15 --ioengine=libaio --time_based --rw=read --bs=64K --numjobs=1 --group_reporting --name=test
```

--------------------------------

### Inject OSD Memory Target Argument at Runtime

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Injects memory target arguments directly to OSD processes at runtime. This method allows for temporary adjustments or testing on specific OSDs without cluster-wide configuration changes.

```bash
ceph tell 'osd.*' injectargs '--osd_memory_target=2147483648'
```

--------------------------------

### Control Backfill and Recovery Speed

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Configuration commands to adjust the maximum number of concurrent backfills per OSD and the sleep time between recovery operations, useful for performance tuning during heavy data movement.

```bash
# set number of active pg-backfills for one osd
ceph tell 'osd.*' injectargs -- --osd_max_backfills=4

# forced sleeptime in ms between recovery operations
ceph tell 'osd.*' injectargs -- --osd_recovery_sleep_hdd=0
```

--------------------------------

### Optimize Ceph Placement with Custom Balancer

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Executes a custom Ceph balancer script to generate balancing movement instructions. It specifies the number of balancing movements to create and outputs them to a file for review.

```bash
# generate 100 balancing movements
./placementoptimizer.py -v balance -m 100 > /tmp/balance-instructions
# after you are happy with the results:
bash /tmp/balance-instructions
```

--------------------------------

### Adjust OSD Reweighting in Ceph

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This section warns against arbitrary use of `ceph osd reweight` and recommends using `ceph osd crush reweight` for artificially shrinking devices. Incorrect reweighting can lead to uneven PG distribution and performance issues, as it doesn't affect the host's bucket weight.

```bash
# Incorrect usage example (use with caution or avoid)
# ceph osd reweight <osd_id> <weight>

# Recommended for shrinking device capacity in CRUSH map
# ceph osd crush reweight <osd_id> <weight>
```

--------------------------------

### FIO: Non-Cached Read Test

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This fio command performs a non-cached read test (`direct=1`) with a high `iodepth` of 512. It bypasses the cache and I/O scheduler, allowing for a high degree of parallelism in read operations.

```bash
fio --filename=/mnt/rbd-fs/file --size=20G --direct=1 --sync=0 --iodepth=512 --runtime=15 --ioengine=libaio --time_based --rw=read --bs=64K --numjobs=1 --group_reporting --name=test
```

--------------------------------

### Configure Scrub Intervals and Behavior

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets various intervals and behaviors for PG scrubbing, including minimum and maximum intervals, deep scrub frequency, sleep time between scrub operations, and disabling scrubbing during recovery.

```ini
[global]
# every week if load is low enough
osd scrub min interval = 604800
# every two weeks even if the load is too high
osd scrub max interval = 2678400
# deep scrub once every month (60*60*24*31*1)
osd deep scrub interval = 2678400
# time to sleep for group of chunks
# to reduce client latency impact
osd scrub sleep = 0.05
# no scrub while there is recovery (performance)
osd scrub during recovery = false
```

--------------------------------

### Mark OSD Down and Repeer

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Forces an OSD to go down, allowing for maintenance or troubleshooting, and then repeers it. This is often used after adjusting PG limits or dealing with stuck PGs on new OSDs.

```bash
ceph osd down $(ceph osd ls-tree $yournewserver)
ceph pg repeer $pgid
```

--------------------------------

### View PG Status with jq

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Uses `jq` to pretty-print the JSON output of `ceph pg $pgid query`, making it easier to read and analyze the PG status.

```bash
ceph pg $pgid query | jq -C . | less
```

--------------------------------

### Mount CephFS with Specific Secret File

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Mounts a CephFS filesystem on a client machine. It specifies the filesystem name (`lolfs`), the metadata and root pool names, and uses a secret file for authentication. The mount command targets the Ceph monitor IP and port.

```bash
ceph fs new lolfs lol_metadata lol_root
mount -v -t ceph -o name=lolroot,secretfile=lolfs.root.secret 10.0.18.1:6789:/ mnt/
```

--------------------------------

### List RBD Images in Namespace

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Lists all RBD images residing within a specific namespace of an RBD pool.

```bash
rbd --pool $metadata_pool_name --namespace $namespacename ls
```

--------------------------------

### Inspect OSD Status and PG Limits

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Checks the status of a specific OSD, including the number of PGs it manages, and provides insight into soft and hard PG limits that might prevent new PGs from being accepted.

```bash
ceph daemon osd.$id status
ceph tell osd.$id status
```

--------------------------------

### Disable Bluestore Cache Autotuning (Host-specific)

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Disables the automatic tuning of the Bluestore cache for a specific host. This allows manual control over cache size using parameters like `bluestore_cache_size`.

```bash
ceph config set osd/host:yourspecialcheapservername bluestore_cache_autotune false
```

--------------------------------

### Configure CephFS Quotas

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets quotas for CephFS subdirectories using extended attributes to limit maximum bytes or files. Setting the value to '0' removes the quota.

```bash
setfattr -n ceph.quota.max_bytes -v 20971520 /a/directory   # 20 MiB
```

```bash
setfattr -n ceph.quota.max_files -v 5000 /another/dir       # 5000 files
```

--------------------------------

### Create ext4 Filesystem with Optimized Stripe Size for Ceph RBD

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Formats a Ceph RBD block device with an ext4 filesystem, optimizing for the underlying RADOS object size. The `stride` and `stripe_width` parameters are set to 1024 blocks (4MiB) to align with the typical 4MiB object size.

```bash
mkfs -E nodiscard,stride=1024,stripe_width=1024 /dev/rbd/your/rbd
```

--------------------------------

### Show Daemon Versions

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Displays the versions of all running Ceph daemons in the cluster. Helps identify outdated daemons.

```bash
# show daemon versions to find outdated ones ;)
ceph versions
```

--------------------------------

### Configure lvmcache for RBD Client Local Cache

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This configuration snippet demonstrates how to set up lvmcache to cache an RBD on local SSD storage. It emphasizes the benefits for reducing filesystem commit latency and improving read performance, especially for HDD-based Ceph pools. Ensure the cache pool is adequately sized for optimal results.

```bash
# Example command to configure lvmcache for an RBD (specific commands may vary based on setup)
# This is a conceptual example, actual implementation requires specific lvm and rbd commands.
# Refer to Ceph and LVM documentation for precise steps.
```

--------------------------------

### Enable Fast Pool Reads

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Enable the `fast_read` pool setting to improve read performance for small files, especially with CephFS. This makes the primary OSD query all shards for data and take the fastest reply, at the cost of increased network traffic and OSD load.

```shell
ceph osd pool set cephfs_data_pool fast_read 1
```

--------------------------------

### Find OSD Host

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Finds the host where a specific OSD is running.

```bash
# Find the host the OSD is in
ceph osd find $osdid
```

--------------------------------

### Benchmark Running OSD Object Writes

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Tests the object write performance of a running OSD by having it write objects of a specified size until a total amount is reached. Useful for checking OSD performance before bringing it 'in'.

```bash
# let the osd write objects of given size until amount is reached
ceph tell osd.$osdid bench $data_amount $object_size
```

--------------------------------

### Fio Test for Non-Cached Queued Writes

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Performs a read/write test using `fio` with `O_DIRECT` enabled, bypassing the Linux page cache and IO scheduler queue. This test aims to maximize direct IO performance and should show a high number of outstanding operations in `osdc`.

```bash
fio --filename=/mnt/rbd-fs/file --size=20G --direct=1 --sync=0 --iodepth=512 --runtime=15 --ioengine=libaio --time_based --rw=rw --bs=64K --numjobs=1 --group_reporting --name=test
```

--------------------------------

### Display CephFS Status Overview

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Provides a summary of the current status of the CephFS filesystems, including their health and configuration.

```shell
ceph fs status
```

--------------------------------

### Authorize CephFS User

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Authorizes a CephFS client with specified permissions to access a given path within the filesystem. The path restriction is for metadata only; data access restriction requires namespaces.

```bash
ceph fs authorize lolfs client.lolroot / rw
```

```bash
ceph fs authorize lolfs client.some.weird_name /lol/ho_ho_ho rw /stuff r
```

--------------------------------

### Ceph Monmap Manipulation

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Commands to extract, print, edit, and inject the Ceph monitor map (monmap). This is used for managing monitor names and IP addresses within the cluster. Requires `ceph-mon` and `monmaptool` utilities.

```bash
ceph-mon --extract-monmap --name mon.monname /tmp/monmap
monmaptool --print /tmp/monmap
monmaptool --help   # edit the monmap
ceph-mon --inject-monmap --name mon.monname /tmp/monmap
```

--------------------------------

### View Ceph Balancer Status and Control

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Provides a set of Ceph CLI commands to interact with the built-in balancer. This includes checking status, changing modes, evaluating scores, creating, showing, and executing balancing plans.

```bash
# balancer commands:
ceph balancer status
ceph balancer mode upmap         # upmap items as movement method, not reweighting.
ceph balancer eval               # evaluate current score
ceph balancer optimize myplan    # create a plan, don't run it yet
ceph balancer eval myplan        # evaluate score after myplan. optimal is 0
ceph balancer show myplan        # display what plan would do
ceph balancer execute myplan     # run plan, this misplaces the objects
ceph balancer rm myplan

# view auto-balancer status and durations
ceph tell 'mgr.$activemgrid' balancer status
```

--------------------------------

### Dump dm-crypt Configuration Keys

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Displays the configuration keys stored in Ceph MONs related to dm-crypt encryption. This command is used to inspect the secret HDD keys used for encrypted OSDs.

```bash
ceph config-key dump | grep dm-crypt
```

--------------------------------

### Set Ceph Pool Quotas for Storage and Object Count

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Configures storage quotas for Ceph pools. The commands set a maximum storage capacity in bytes (using shell arithmetic for 1 TiB) and a maximum number of objects allowed within a specified pool.

```bash
# set max storage bytes to 1TiB (uses shell-calculation)
ceph osd pool set-quota funny_pool_name max_bytes $((1 * 1024 ** 4))

# limit number of objects
ceph osd pool set-quota funny_pool_name max_objects 1000000
```

--------------------------------

### List Connected CephFS Clients

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Shows a list of clients currently connected to a specific MDS (Metadata Server), including their IP addresses. Replace `$mdsid` with the actual MDS identifier.

```shell
ceph tell mds.$mdsid client ls
```

--------------------------------

### Configure OSD Memory Target (Host-specific)

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Configures the OSD memory target for a specific host within the Ceph cluster. This allows for fine-grained control over memory allocation based on individual host capabilities or requirements.

```bash
ceph config set osd/host:yourspecialcheapservername osd_memory_target $size_in_byte
```

--------------------------------

### Configure CephFS Standby Replay

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Sets a CephFS filesystem property to control whether standby MDS servers can replay MDS state. This impacts failover performance and resource usage.

```bash
ceph fs set $fsname allow_standby_replay <true|false>
```

--------------------------------

### Calculate Ceph PG Shard Size

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Provides manual calculations for determining the appropriate shard size for Ceph pools based on their replication or erasure coding scheme. This helps in planning PG counts.

```bash
# shard size calculation
if pool is replica:
    shardsize = pool_size / pg_num
elif pool is erasurecoded(n+m):
    shardsize = pool_size / (pg_num * n)
```

--------------------------------

### Benchmark RBD Speed

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Benchmarks the read/write performance of a Ceph RBD image. Allows specifying I/O type, pattern, size, and threads.

```bash
rbd bench --io-type rw $poolname/$imagename
# other io types: read, write, rw
# --io-pattern seq rand
# --io-size $oneiosize (with B/K/M/G/T suffix)
# --io-total $totalbytecount (with suffix)
# --io-threads $threadcount
```

--------------------------------

### Map PG to OSDs

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Determines the 'last primary' OSD for a PG that is stuck in 'down' or 'unknown' state. This helps identify which OSD to investigate further.

```bash
ceph pg map $pgid
```

--------------------------------

### RBD Mount Script

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

A simple bash script to mount an RBD device to a specified mount point if it's not already mounted. This script is intended to be executed after the RBD has been mapped.

```bash
#!/bin/bash

mountpoint -q /your/mount || mount /your/mount
```

--------------------------------

### Ceph: View LVM tags of an encrypted LV

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

This command is used to retrieve the LVM tags associated with an encrypted logical volume, which are necessary for preparing a new device during OSD migration.

```bash
lvs -o lv_tags /dev/path-to-encrypted-lv
```

--------------------------------

### Assign Device Class to Ceph OSD

Source: https://github.com/thejj/ceph-cheatsheet/blob/master/README.md

Assigns a custom device class (e.g., 'huge', 'fast', 'ssd') to a specific Ceph OSD. This allows for more granular control over data placement by selecting OSDs based on their assigned classes within CRUSH rules.

```bash
ceph osd crush set-device-class $deviceclass $osdid
```