### kubectl get ps-backups Output Example

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Shows the output of the `kubectl get ps-backups` command, including the new TYPE column that distinguishes between full and incremental backups.

```bash
$ kubectl get ps-backups
NAME              STORAGE   TYPE          STATE       COMPLETED
weekly-full-1     s3-us     full          Succeeded   2026-03-15T00:00:00Z
daily-inc-mon     s3-us     incremental   Succeeded   2026-03-16T00:00:00Z
daily-inc-tue     s3-us     incremental   Succeeded   2026-03-17T00:00:00Z
daily-inc-wed     s3-us     incremental   Running     -
```

--------------------------------

### Build Operator Image with Specific Example

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Example of building the Operator image using 'myid' as the Docker ID and 'percona-server-mysql-operator' as the repository name, with tag 'k8sps-22'.

```bash
DOCKER_SQUASH=0 DOCKER_NOCACHE=0 make IMAGE=myid/percona-server-mysql-operator:k8sps-22
```

--------------------------------

### Example Existing CR YAML

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/TEMPLATE.md

Show that existing CRs remain compatible by including an example of an unchanged relevant CR section.

```yaml
# Show that existing CRs continue to work identically.
# Only include the relevant spec section.
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: cluster1
spec:
  # ... relevant section unchanged ...
```

--------------------------------

### Example New Workflow CR YAML

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/TEMPLATE.md

Illustrate new user-facing workflows with concrete YAML examples, including inline comments explaining operator behavior.

```yaml
# Show the new capability end-to-end.
# Add inline comments explaining operator behavior.
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: cluster1
spec:
  # ... new or changed fields with comments ...
```

--------------------------------

### Verify Operator Deployment with kubectl

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Use `kubectl get all` to check if the Operator deployment, pods, and related resources have been created successfully.

```bash
kubectl get all
```

--------------------------------

### Install CentOS Dependencies

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/e2e-tests/README.md

Installs necessary software packages on CentOS for building and testing the Operator. Includes Percona repositories, core utilities, Docker, and specific tools like yq and Google Cloud SDK.

```bash
sudo yum -y install epel-release https://repo.percona.com/yum/percona-release-latest.noarch.rpm
sudo yum -y install coreutils sed jq curl docker percona-xtrabackup-24
sudo curl -s -L https://github.com/mikefarah/yq/releases/download/4.35.1/yq_linux_amd64 -o /usr/bin/yq
sudo chmod a+x /usr/bin/yq
curl -s -L https://github.com/openshift/origin/releases/download/v3.11.0/openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz \
    | tar -C /usr/bin --strip-components 1 --wildcards -zxvpf - '*/oc' '*/kubectl'
curl -s https://get.helm.sh/helm-v3.12.3-linux-amd64.tar.gz \
    | tar -C /usr/bin --strip-components 1 -zxvpf - '*/helm'
curl https://sdk.cloud.google.com | bash
curl -fsSL https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz \
    | tar -xzf -
./krew-linux_amd64 install krew
export PATH="\${KREW_ROOT:-\$HOME/.krew}/bin:\$PATH"
kubectl krew install assert
kubectl krew install --manifest-url https://raw.githubusercontent.com/kubernetes-sigs/krew-index/a67f31ecb2e62f15149ca66d096357050f07b77d/plugins/kuttl.yaml
sudo tee /etc/yum.repos.d/google-cloud-sdk.repo << EOF
[google-cloud-cli]
name=Google Cloud CLI
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
sudo yum install -y google-cloud-cli google-cloud-cli-gke-gcloud-auth-plugin
```

--------------------------------

### Install Operator and Deploy to Kubernetes

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Install the Custom Resource Definitions (CRDs) and deploy the Operator to your Kubernetes cluster using the specified image.

```bash
make install deploy IMAGE=<your-docker-id>/percona-server-mysql-operator:k8sps-22
```

--------------------------------

### Example Design Decision Alternatives Table

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/TEMPLATE.md

Document design decisions by listing the chosen approach, reasoning, and alternatives considered with reasons for rejection.

```markdown
| Alternative | Why Rejected |
|------------|--------------|
| Alternative A | Reason — be specific about the trade-off |
| Alternative B | Reason |
```

--------------------------------

### Full Backup CR Example

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

This is a standard Kubernetes Custom Resource (CR) definition for a full backup. The 'type' defaults to 'full' if not specified.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLBackup
metadata:
  name: my-full-backup
spec:
  clusterName: my-cluster
  storageName: s3-us
```

--------------------------------

### `kubectl get ps-backups` Output

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

The `kubectl get ps-backups` command output includes a new `TYPE` column to distinguish between full and incremental backups.

```APIDOC
## User-Facing Behavior Changes

### `kubectl get ps-backups` Output

The `TYPE` column is added to display the backup type:

```
$ kubectl get ps-backups
NAME              STORAGE   TYPE          STATE       COMPLETED
weekly-full-1     s3-us     full          Succeeded   2026-03-15T00:00:00Z
daily-inc-mon     s3-us     incremental   Succeeded   2026-03-16T00:00:00Z
daily-inc-tue     s3-us     incremental   Succeeded   2026-03-17T00:00:00Z
daily-inc-wed     s3-us     incremental   Running     -
```
```

--------------------------------

### Verify CRDs with kubectl

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Use `kubectl get crds` to list all Custom Resource Definitions installed in the cluster, including those for Percona Server MySQL.

```bash
kubectl get crds
```

--------------------------------

### PerconaServerMySQL CR with ClusterSet Configuration

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Example of how ClusterSet configuration might be embedded within individual PerconaServerMySQL Custom Resources for primary and replica clusters. This approach was rejected due to splitting topology management across multiple CRs.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: primary-cluster
spec:
  # ...
  clusterSet:
    role: primary
---
# Replica cluster
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: replica-cluster
spec:
  # ...
  clusterSet:
    role: replica
    primaryEndpoints:
    - host: 10.0.1.13
      port: 3306
```

--------------------------------

### Incremental Backup using Explicit LSN

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

When streaming backups directly to cloud storage, a local copy of the previous backup is unavailable. In such cases, Percona XtraBackup supports specifying the starting LSN directly using the `--incremental-lsn` option, eliminating the need for `--incremental-basedir`.

```bash
xtrabackup --backup --incremental-lsn=<LSN>
```

--------------------------------

### Per-Site Bootstrap Mode with Manual Setting (After Change)

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Explains the behavior of Pod-0 bootstrap when `spec.mysql.bootstrap.mode` is set to `manual`, and how subsequent pods join the GR group.

```text
spec.mysql.bootstrap.mode = manual (new field; default auto)
  Pod-0 bootstrap
    → configureInstance()
    → connectToCluster() [fails: no peers]
      → peers.Len() == 1 + manual mode → short-circuit, return 0
    → exits 0; pod stays NotReady (no GR membership yet)
  Pod-1, Pod-2 do not start (OrderedReady blocks on Pod-0 Readiness)
  --- external actor runs dba.createClusterSet().createReplicaCluster() ---
  Pod-0 is now a single-member GR group → ReadinessProbe passes
  Pod-1 boots → connectToCluster succeeds via Pod-0 → addInstance()
  Pod-2 boots → likewise
```

--------------------------------

### Incremental Restore Prepare Process

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Restoring from an incremental backup chain involves a multi-step preparation process using Percona XtraBackup. All steps except the final one require the `--apply-log-only` flag. Each incremental directory is modified during this process and cannot be reused.

```bash
1. Prepare the base (full) backup with `--apply-log-only` (redo only, no rollback)
2. Apply each incremental in order, all with `--apply-log-only` except the last
3. Apply the final incremental without `--apply-log-only` (allows rollback)
4. Move back to datadir as with any full backup
```

--------------------------------

### Run Unit and Integration Tests

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Execute the 'make test' command to run all unit and integration tests for the operator.

```bash
make test
```

--------------------------------

### MySQL Cluster Bootstrap Flow (Before Change)

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Illustrates the sequence of events when a MySQL cluster boots and forms a Group Replication (GR) group in the existing architecture.

```text
Pod-0 boots
  → /opt/percona/bootstrap (cmd/bootstrap → cmd/bootstrap/gr)
    → configureInstance()                                  (mysqlsh, local exec)
    → connectToCluster() [fails: no peers]
      → peers.Len() == 1 → dba.createCluster()             (mysqlsh, local exec)
    → cluster.status() / addInstance() / rescan()
    → exits 0
  → Readiness/Liveness probes run
Pod-1, Pod-2... boot in OrderedReady sequence
  → connectToCluster() succeeds via Pod-0
  → addInstance() joins them to the existing GR group
```

--------------------------------

### Run Bundle Generation Commands

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/installers/olm/README.md

Execute these make commands to generate the OLM bundles. You can generate all bundles or specify a particular bundle type.

```bash
# Generate all bundles (community and redhat):
make bundles
```

```bash
# Generate only specific bundle:
make bundles/community
```

--------------------------------

### Before Change: Full Backup Flow

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Illustrates the sequence of events for creating a full backup before the incremental backup feature was introduced. This involves user creation of a CR, controller reconciliation, and the execution of a Kubernetes Job that uses a sidecar HTTP API.

```text
User creates PerconaServerMySQLBackup CR
  → Backup Controller reconciles
    → Validates cluster, storage, resolves source pod
    → Creates K8s Job (run-backup.sh)
      → Job POSTs BackupConfig JSON to sidecar HTTP API (port 6450)
        → Sidecar runs: xtrabackup --backup --stream=xbstream | xbcloud put
    → Controller monitors Job, updates CR status
```

--------------------------------

### Example CRD Spec Field Documentation

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/TEMPLATE.md

Document new or changed fields in the CRD spec, including their purpose, hierarchy, default values, and validation rules.

```markdown
- **`spec.path.to.newField`** *(optional, default: `"value"`)*:
  Description of what this field controls. Must be one of `[a, b]`.
  When omitted, the system behaves as [describe]. Only applies when
  [describe conditions, e.g., clusterType is group-replication].
```

--------------------------------

### Storage Directory Convention for Incremental Backups

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Illustrates the naming convention for full and incremental backups in storage, emphasizing the `.incr/` suffix for linking incrementals to their base backup.

```bash
<storage-prefix>/
  cluster-2026-03-15-full/                                ← base (full) backup data (unchanged)
  cluster-2026-03-15-full.incr/                           ← incremental backups directory for given base
    cluster-2026-03-16T000000-incr/                       ← first incremental
    cluster-2026-03-17T000000-incr/                       ← second incremental
    cluster-2026-03-18T000000-incr/                       ← third incremental
```

--------------------------------

### After Change: Unified Restore Flow (Incremental)

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Explains the unified restore process, specifically for incremental backups. This involves resolving the incremental chain, pausing the cluster, and executing a job that applies incremental changes sequentially.

```text
User creates PerconaServerMySQLRestore CR
  → Restore Controller reconciles
    → Resolves backup destination:
      - backupName: looks up backup CR → gets status.destination + storage config
      - backupSource: reads destination + storage config from spec
    → Checks if backup type is incremental (via CR type or destination path)
    → For incremental restore:
      → resolveIncrementalChain():
        → Parses destination to extract base path (IncrementalBaseDestination())
          and incrementals dir (IncrementalsDir())
        → Connects to storage, lists all objects under ".incr/" prefix
        → Extracts timestamp-based names, sorts lexicographically
        → Filters to include only entries ≤ target backup timestamp
        → Returns DestinationInfo{Base, Incrementals}
      → Pauses cluster
      → Creates K8s Job with env vars:
        - BACKUP_DEST = base backup path
        - BACKUP_INCREMENTALS_DEST = comma-separated incremental paths
        → Job downloads and extracts base backup
        → Prepares base with --apply-log-only
        → For each incremental: download, extract, apply with --apply-log-only
          (except the last one, which omits --apply-log-only)
        → Move back to datadir
      → Unpauses cluster
    → For full restore: existing logic unchanged
```

--------------------------------

### Configure and Run Operator Locally

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Set up environment variables, modify the CR file, and use make commands to run a local development version of the operator. This is useful for faster debugging cycles.

```bash
export WATCH_NAMESPACE=<desired_namespace>
export OPERATOR_NAMESPACE=<desired_namespace>
# Uncomment the initImage key in your deploy/cr.yaml
make install
make run
```

--------------------------------

### Stage, Commit, and Push Changes

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Stage all changes, commit them with a message referencing the Jira issue, and push the new branch to your origin.

```bash
git add .
git commit -m "K8SPS-22 fixed by ......
git push -u origin K8SPS-22-fix-feature X
```

--------------------------------

### Creating a PerconaServerMySQLClusterSet

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Defines a ClusterSet with a primary and replica cluster. The controller will automatically create the ClusterSet and configure the replica cluster.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLClusterSet
metadata:
  name: my-clusterset
spec:
  primaryCluster: cluster1
  sslMode: AUTO
  credentialsSecret:
    name: cluster1-secret
    key: clusterset
  mysqlshellRunner:
    image: percona/percona-server:8.0.36
  clusters:
    - innodbClusterName: cluster1
      endpoints:
        - host: cluster1-mysql.example.com
          port: 3306
    - innodbClusterName: cluster2
      endpoints:
        - host: cluster2-mysql.example.com
          port: 3306

```

--------------------------------

### Build Operator Image with Custom Settings

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Build the Docker image for the Operator, disabling Docker cache and squash features. Specify your Docker Hub ID, repository name, and a custom tag.

```bash
DOCKER_SQUASH=0 DOCKER_NOCACHE=0 make IMAGE=<your-docker-id>/<custom-repository-name>:<custom-tag>
```

--------------------------------

### Create and Checkout Git Branch

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Use this command to create a new branch for your changes, incorporating a Jira issue number if applicable.

```bash
git checkout -b K8SPS-22-fix-feature-X
```

--------------------------------

### Deploy Operator using kubectl

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/README.md

Use this command to deploy the Percona Operator for MySQL from the bundle.yaml file.

```sh
kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mysql-operator/main/deploy/bundle.yaml
```

--------------------------------

### After Change: Incremental Backup Flow

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Details the enhanced backup flow for creating incremental backups. This includes resolving the base backup, fetching the last LSN, computing the destination, and executing the backup job with specific environment variables and API calls.

```text
User creates PerconaServerMySQLBackup CR with type: incremental
  → Backup Controller reconciles
    → setIncrementalBaseAnnotations():
      → Resolves base backup (explicit via spec.incrementalBaseBackupName,
        or auto-resolves latest succeeded full backup for same clusterName + storageName)
      → Validates base backup is succeeded and uses same storage
      → Sets percona.com/base-backup-name annotation with base backup's destination name
    → getLastBackupLSN():
      → Finds latest succeeded backup (full or incremental) for same cluster
      → Calls sidecar's /backup/checkpoint-info endpoint
        → Sidecar fetches xtrabackup_checkpoints from storage via xbcloud get
        → Parses and returns to_lsn
    → Computes destination: <base-name>.incr/<cluster>-<timestamp>-incr
    → Creates K8s Job (run-backup.sh)
      → Sets INCREMENTAL_LSN env var on the job container
      → Job POSTs BackupConfig (includes incrementalLsn) to sidecar HTTP API
        → Sidecar runs: xtrabackup --backup --stream=xbstream
            --incremental-lsn=<LSN> | xbcloud put <computed-destination>
    → Updates CR status with type and destination
```

--------------------------------

### Deploy Database Cluster using kubectl

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/README.md

Apply the cr.yaml file to deploy the database cluster managed by the operator.

```sh
kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mysql-operator/main/deploy/cr.yaml
```

--------------------------------

### Undeploy Operator and Uninstall Resources

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Execute make commands to clean up the Operator deployment, CRDs, and CR objects. Remember to also clean up PVCs separately for a complete removal.

```bash
make undeploy uninstall
```

--------------------------------

### Build Operator Image Without Pushing

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Build the Operator image without automatically pushing it to a registry. Use `DOCKER_PUSH=0` and push manually later if needed.

```bash
DOCKER_PUSH=0 make IMAGE=<your-docker-id>/<custom-repository-name>:<custom-tag>
```

--------------------------------

### Manually Push Docker Image

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Manually push a previously built Docker image to your registry after setting `DOCKER_PUSH=0` during the build process.

```bash
docker push <your-docker-id>/<custom-repository-name>:<custom-tag>
```

--------------------------------

### Set Environment Variables for Bundle Generation

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/installers/olm/README.md

Set these environment variables before running the bundle generation commands. Ensure you update IMAGE_TAG_OWNER if you use a different repository for image tags.

```bash
# operator version
export VERSION=1.0.0
# By default we use perconalab for tag owner. Please update this variable to use another repo
export IMAGE_TAG_OWNER=percona
# Min k8s version
export MIN_KUBE_VERSION=1.27.0
# Openshift versions:
export OPENSHIFT_VERSIONS="v4.16-v4.20"
```

--------------------------------

### xtrabackup_checkpoints File Format

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

This file contains critical information about a Percona XtraBackup backup, including its type, the Log Sequence Numbers (LSNs) it covers, and the LSN at the time of completion. The `from_lsn` of an incremental backup must match the `to_lsn` of the preceding backup in the chain.

```text
backup_type = full-backuped | incremental
from_lsn = <starting LSN>
to_lsn = <checkpoint LSN at backup end>
last_lsn = <final copied LSN>
```

--------------------------------

### Streaming Incremental Backups

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Incremental backups are streamed to cloud storage using the same method as full backups, typically involving `xtrabackup --backup --stream=xbstream | xbcloud put`. Instead of full `.ibd` files, `.delta` files representing changed pages are streamed. Each backup is stored as an independent object in cloud storage.

```bash
xtrabackup --backup --stream=xbstream | xbcloud put
```

--------------------------------

### Build Operator Image

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/e2e-tests/README.md

Executes the build script for the Operator's Docker image. Ensure the `IMAGE` environment variable is set if using a custom repository.

```bash
./e2e-tests/build
```

--------------------------------

### Seeding Replica Cluster from Backup

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Specifies a PerconaServerMySQLRestore CR to seed a replica cluster from an existing backup. This is recommended for large datasets over the default 'clone' method.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLRestore
metadata:
  name: clusterset-seed
spec:
  clusterName: cluster2
  backupSource:
    destination: s3://my-bucket/backups/cluster1-full
    storage:
      type: s3
      s3:
        bucket: my-bucket
        credentialsSecret: s3-credentials
        region: us-east-1

```

--------------------------------

### Check Cluster Resources with kubectl

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Use kubectl commands to inspect the status of Percona Server MySQL custom resources and related Kubernetes objects. This helps in understanding the current state of the deployment.

```bash
kubectl get ps
NAME       REPLICATION   ENDPOINT                         STATE   MYSQL   ORCHESTRATOR   ROUTER   AGE
cluster1   async         cluster1-mysql-primary.default   ready   3       3                       14m

kubectl get all
NAME                                                 READY   STATUS    RESTARTS      AGE
pod/cluster1-mysql-0                                 2/2     Running   0             16m
pod/cluster1-mysql-1                                 2/2     Running   1 (15m ago)   15m
pod/cluster1-mysql-2                                 2/2     Running   1 (14m ago)   14m
pod/cluster1-orc-0                                   2/2     Running   0             10m
pod/cluster1-orc-1                                   2/2     Running   0             11m
pod/cluster1-orc-2                                   2/2     Running   0             12m
pod/percona-server-mysql-operator-6b65cdc6fc-nkzlk   1/1     Running   0             26m

NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
service/cluster1-mysql           ClusterIP   None             <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   16m
service/cluster1-mysql-primary   ClusterIP   10.110.253.17    <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   16m
service/cluster1-mysql-unready   ClusterIP   None             <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   16m
service/cluster1-orc             ClusterIP   None             <none>        3000/TCP,10008/TCP                      16m
service/cluster1-orc-0           ClusterIP   10.101.142.185   <none>        3000/TCP,10008/TCP                      16m
service/cluster1-orc-1           ClusterIP   10.109.216.54    <none>        3000/TCP,10008/TCP                      16m
service/cluster1-orc-2           ClusterIP   10.97.8.119      <none>        3000/TCP,10008/TCP                      16m

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/percona-server-mysql-operator   1/1     1            1           26m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/percona-server-mysql-operator-6b65cdc6fc   1         1         1       26m

NAME                              READY   AGE
statefulset.apps/cluster1-mysql   3/3     16m
statefulset.apps/cluster1-orc     3/3     16m
```

--------------------------------

### Apply Custom Resource for Percona Cluster

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Create a custom resource (CR) for a Percona Server MySQL cluster using the provided YAML configuration file.

```bash
kubectl apply -f config/samples/ps_v2_perconaserverformysql.yaml
perconaservermysql.ps.percona.com/cluster1 created
```

--------------------------------

### Storage Directory Convention

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Defines the directory structure for full and incremental backups to maintain chain linkage.

```APIDOC
## Storage Path Conventions

### Full Backups
- Format: `<cluster>-<timestamp>-full`

### Incremental Backups
- Format: `<base-backup-name>.incr/<cluster>-<timestamp>-incr`

**Example Structure:**
```
<storage-prefix>/
  cluster-2026-03-15-full/                                ← base (full) backup data
  cluster-2026-03-15-full.incr/                           ← incremental backups directory for given base
    cluster-2026-03-16T000000-incr/                       ← first incremental
    cluster-2026-03-17T000000-incr/                       ← second incremental
    cluster-2026-03-18T000000-incr/                       ← third incremental
```
```

--------------------------------

### Run Specific kuttl Test

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/e2e-tests/README.md

Executes a specific kuttl test using the `kubectl kuttl test` command. The `--config` flag points to the kuttl configuration file, and `--test` filters tests by name using a regular expression.

```bash
kubectl kuttl test --config e2e-tests/kuttl.yaml --test "^test-name
$"
```

--------------------------------

### Before Change: Full Restore Flow

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Describes the process for restoring data from a full backup prior to the implementation of incremental backups. This flow includes pausing the cluster, creating a restore Job, and then unpausing the cluster.

```text
User creates PerconaServerMySQLRestore CR
  → Restore Controller reconciles
    → Pauses cluster (Spec.Pause=true, waits for pods=0)
    → Creates K8s Job (run-restore.sh)
      → Job runs: xbcloud get | xbstream -x → xtrabackup --prepare → --move-back
    → Unpauses cluster
```

--------------------------------

### Check Operator Logs

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Retrieve the logs for the running Percona Server MySQL Operator pod to verify its status and troubleshoot issues.

```bash
kubectl logs percona-server-mysql-operator-<pod-hash>
```

--------------------------------

### Run a Single End-to-End Test with kuttl

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Use kubectl kuttl test with a specific configuration and test name to run a single end-to-end test. The '--skip-delete' flag can be used to prevent the test namespace from being deleted for troubleshooting.

```bash
kubectl kuttl test --config ./e2e-tests/kuttl.yaml --test ^<test-name>/$
```

```bash
kubectl kuttl test --config ./e2e-tests/kuttl.yaml --test ^<test-name>/$ --skip-delete
```

--------------------------------

### Operator Reconciliation Flow for ClusterSet (After Change)

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Describes the process when a user applies a PerconaServerMySQLClusterSet CR, detailing the operator's reconciliation steps and execution methods.

```text
User applies PerconaServerMySQLClusterSet CR
  → New controller in operator pod reconciles
    → Probe each clusters[].endpoints
    → Decide action: bootstrap | switchover | force-failover |
                     add-replica | remove | rejoin | refresh-status | dissolve
    → Execute:
       - Fast verbs via kubectl exec into the mysqlshell-runner Pod
       - createReplicaCluster as a Kubernetes Job using the runner image (initial cloning may take hours)
    → Update CR status from observed topology
```

--------------------------------

### Existing PerconaServerMySQL CR (Unchanged)

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

This is the default configuration for a PerconaServerMySQL CR, identical to previous versions. It defaults to 'auto' bootstrap mode for Group Replication.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: cluster1
spec:
  mysql:
    clusterType: group-replication
    size: 3
  # ... unchanged ...

```

--------------------------------

### Set Docker Platform

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/e2e-tests/README.md

Sets the `DOCKER_DEFAULT_PLATFORM` environment variable to specify the target platform for Docker images. Defaults to `linux/amd64` if not set.

```bash
export DOCKER_DEFAULT_PLATFORM=linux/amd64
```

--------------------------------

### Standalone Replica Cluster Configuration

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Configure a PerconaServerMySQL CR to act as a standalone replica cluster, a precursor to ClusterSet membership. It uses manual bootstrap mode so Pod-0 does not auto-form a Group Replication group.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: cluster2
spec:
  mysql:
    clusterType: group-replication
    size: 3
    bootstrap:
      mode: manual                  # NEW: Pod-0 boots standalone, no dba.createCluster()
  secretsName: cluster2-creds
  # ... rest unchanged ...

```

--------------------------------

### Configuring Incremental Recovery for Replica Cluster

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/cross-cluster-replication/cross-cluster-replication.md

Adds an option to the ClusterSet configuration to use incremental recovery when seeding a replica cluster, assuming the replica has already been restored from a backup.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLClusterSet
metadata:
  name: my-clusterset
spec:
  # ...
  createReplicaClusterOptions:
    recoveryMethod: incremental

```

--------------------------------

### Build Operator Image with Base Tag Override

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/CONTRIBUTING.md

Build the Operator image, overriding the default image tag detection from the branch. This allows specifying a base image name and registry path.

```bash
DOCKER_SQUASH=0 DOCKER_NOCACHE=0 make IMAGE_TAG_BASE=<your-docker-id>/percona-server-mysql-operator
```

--------------------------------

### LSN Retrieval (controller → sidecar → storage)

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

The sidecar exposes a /backup/checkpoint-info endpoint to retrieve checkpoint information for incremental backups.

```APIDOC
## POST /backup/checkpoint-info

### Description
Retrieves checkpoint information from a backup's storage location.

### Method
POST

### Endpoint
/backup/checkpoint-info

### Parameters
#### Request Body
- **BackupConfig** (object, required) - The BackupConfig of the backup to retrieve checkpoint information from, including storage credentials and destination.

### Response
#### Success Response (200)
- **CheckpointInfo** (object) - Contains checkpoint details.
  - **backup_type** (string) - The type of backup (full or incremental).
  - **from_lsn** (string) - The starting LSN of the backup.
  - **to_lsn** (string) - The ending LSN of the backup.
  - **last_lsn** (string) - The last LSN flushed.
  - **flushed_lsn** (string) - The last LSN flushed.
  - **redo_memory** (string) - Redo memory information.
  - **redo_frames** (string) - Redo frames information.
```

--------------------------------

### Restore from Incremental Backup (In-Place)

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/docs/enhancements/incremental-backups/incremental-backups.md

Kubernetes CR for restoring a Percona Server MySQL cluster from an incremental backup. The operator automatically detects the backup type, resolves the incremental chain by parsing the backup destination and listing '.incr/' directories, and applies the full backup followed by all dependent incremental backups.

```yaml
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLRestore
metadata:
  name: restore-to-inc2
spec:
  clusterName: my-cluster
  backupName: my-inc-backup-2
  # Operator checks backup type is incremental, calls resolveIncrementalChain():
  #   parses destination (e.g., "prefix/cluster-ts-full.incr/cluster-ts-incr"),
  #   lists the ".incr/" prefix to discover the chain,
  #   and restores: full → inc1 → inc2
```

--------------------------------

### Set Custom Docker Image Repository

Source: https://github.com/percona/percona-server-mysql-operator/blob/main/e2e-tests/README.md

Configures the environment variable `IMAGE` to specify a custom repository for the Operator's Docker images during the build process. This is useful when pushing to a private or personal repository.

```bash
export IMAGE=bob/my_repository_for_test_images:K8SPS-129-fix-feature-X
```