### Dodo Installation

Source: https://github.com/thearas/dodo/blob/master/llms.txt

Installs the Dodo CLI tool using a provided shell script from a GitHub repository. This is the primary method for users to get started with Dodo.

```shell
curl -sSL https://raw.githubusercontent.com/Thearas/dodo/master/install.sh | bash
```

--------------------------------

### Install Dodo CLI

Source: https://github.com/thearas/dodo/blob/master/README.md

Installs the Dodo command-line interface using a curl script. This is the primary method for setting up the tool on your system.

```sh
curl -sSL https://raw.githubusercontent.com/Thearas/dodo/master/install.sh | bash
```

--------------------------------

### Bash: Installing Command-Line Auto-Completion

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Instructions on how to enable command-line auto-completion for the Dodo tool, typically provided after installation or via a specific command.

```bash
dodo completion --help
```

--------------------------------

### AI-Assisted Bug Reproduction with Gemini

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Guides on using Dodo with Gemini CLI to automatically reproduce user bugs. It outlines the required directory structure, setup steps, and the command to initiate the process.

```shell
# Directory structure for user case:
# ├── ddl
# │   ├── example.ob.table.sql
# │   └── example.rb.table.sql
# ├── sql
# │   └── q0.sql
# └── prompt.txt

# Setup:
# 1. Install dodo, Gemini CLI, and mysql command.
# 2. Clone the dodo repository.
# 3. Create a dodo.yaml configuration file in the repository root:
#    host: 127.0.0.1
#    port: 9030
#    http-port: 8030
#    user: root
#    dbs: [example]
#    llm: deepseek-chat      # or o3-mini, etc.
#    llm-api-key: sk-xxxx    # LLM API key
#    anonymize: true         # anonymize SQL before sending to LLM

# Run Gemini:
gemini -iyp 'Your task: @example/usercase/prompt.txt, do not ask any questions, just proceed'
```

--------------------------------

### Dodo Create Schemas

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Demonstrates how to use the 'dodo create' command to create tables and views in a Doris instance after dumping them locally. It shows examples for creating all tables/views for specified databases, specific tables, or running SQL files.

```sh
# Create all dump tables and views for db1 and db2
dodo create --dbs db1,db2

# Create dump table1 and table2
dodo create --dbs db1 --tables table1,table2

# Run any create table/view SQL in db1
dodo create --ddl 'dir/*.sql' --db db1
```

--------------------------------

### Dodo Configuration File

Source: https://github.com/thearas/dodo/blob/master/introduction.md

An example `dodo.yaml` configuration file used for setting up the Dodo tool, especially for AI-driven tasks. It specifies connection details for a database, the LLM to be used, and options for anonymizing SQL before sending it to the LLM.

```yaml
host: 127.0.0.1
port: 9030
http-port: 8030
user: root
dbs: [example]

llm: deepseek-chat      # or o3-mini, etc.
llm-api-key: sk-xxxx    # LLM API key
anonymize: true         # anonymize SQL before sending to LLM
```

--------------------------------

### Generate Data Example

Source: https://github.com/thearas/dodo/blob/master/README.md

An example demonstrating how to generate fake data using a create table SQL statement and specifying the number of rows. It shows the input SQL and the generated output format.

```bash
echo 'create table t1 (
    a varchar(2),
    b struct<foo:tinyint>,
    c date
)' > t1.sql

dodo gendata --ddl t1.sql --rows 5

cat output/gendata/t1/*
```

--------------------------------

### Bash: AI Data Generation with Query and LLM

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Example of using the 'dodo gendata' command with AI capabilities, specifying LLM model, API key, a SQL query to guide generation, and anonymization.

```bash
dodo gendata --dbs db1 --tables t1,t2 \
    --llm 'deepseek-chat' --llm-api-key 'sk-xxx' \
    --query 'select * from t1 join t2 on t1.a = t2.b where t1.c IN ("a", "b", "c") and t2.d = 1' \
    --anonymize
```

--------------------------------

### Doris Dump Statistics Configuration Example

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Example of a `stats.yaml` file showing column statistics, including the `method` field. A `SAMPLE` method indicates potential deviation from actual values.

```yaml
columns:
  - name: col_int
    ndv: 10
    null_count: 4969
    data_size: 800000
    avg_size_byte: 8
    min: "2022"
    max: "2030"
    method: SAMPLE # <-- here
```

--------------------------------

### Multi-FE Replay Example

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Demonstrates how to export and replay audit logs from multiple Doris Frontend (FE) instances concurrently. It shows separate dump commands for each FE and parallel replay commands.

```shell
# Export audit logs separately for fe1 and fe2
dodo dump --dump-query --audit-logs fe1.audlt.log -O fe1
dodo dump --dump-query --audit-logs fe2.audlt.log -O fe2

# Replay audit logs from fe1 and fe2 simultaneously
nohup dodo replay -H <fe1.ip> -f fe1/sql/q0.sql -O fe1 &
nohup dodo replay -H <fe2.ip> -f fe2/sql/q0.sql -O fe2 &

```

--------------------------------

### Dodo Multi-FE Replay Example

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Demonstrates how to dump and replay audit logs from a multi-Frontend (FE) cluster. It emphasizes dumping and replaying each FE's logs separately but simultaneously for accurate results.

```APIDOC
Multi-FE Replay:

# Dump audit logs for fe1 and fe2 separately
dodo dump --dump-query --audit-logs fe1.audlt.log -O fe1
dodo dump --dump-query --audit-logs fe2.audlt.log -O fe2

# Replay audit logs for fe1 and fe2 simultaneously
nohup dodo replay -H <fe1.ip> -f fe1/sql/q0.sql -O fe1 &
nohup dodo replay -H <fe2.ip> -f fe2/sql/q0.sql -O fe2 &

# Note: Replace <fe1.ip> and <fe2.ip> with actual FE IP addresses.
```

--------------------------------

### Generated Data Output Example

Source: https://github.com/thearas/dodo/blob/master/README.md

Shows the content of the generated data file after running the 'dodo gendata' command with a specific table schema.

```text
sO☆{"foo":-66}☆2020-07-23
lg☆{"foo":-121}☆2021-06-15
4☆{"foo":-117}☆2015-06-17
8h☆{"foo":-83}☆2024-09-06
KW☆{"foo":7}☆2019-02-02
```

--------------------------------

### Dodo Generate and Import Data

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Provides examples for using 'dodo gendata' to generate data for tables and 'dodo import' to import generated data or CSV files into Doris tables. It covers generating data from dumped tables or SQL files, and importing data via StreamLoad.

```sh
# Generate data for all dump tables in db1 and db2
dodo gendata --dbs db1,db2

# Generate data for dump table1
dodo gendata --tables db1.table1 # or --dbs db1 --tables table1

# Data can also be generated for any create table SQL without prior dump
# P.S. It might not necessarily be Doris; other databases like Hive also work
dodo gendata --ddl 'ddl/*.sql'

# Generate data with config
dodo gendata ... --genconf gendata.yaml


# Import data for all tables with generated data in db1 and db2
dodo import --dbs db1,db2

# Import data for table1 with generated data
dodo import --tables db1.table1 # or --dbs db1 --tables table1

# Import any CSV data file(s) into table1
dodo import --tables db1.table1 --data 'my_table/*.csv'
```

--------------------------------

### Auto-Increment Generator

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Configures an auto-increment generator with custom step and start values.

```yaml
columns:
  - name: t_string
    format: "string-inc-{{%d}}"
    # `length` won't work, override by `gen`
    # length: 10
    gen:
      inc: 2      # Step is 2 (default 1)
      start: 100  # Starts from 100 (default 1)
```

--------------------------------

### Find Slow SQL Queries with Ripgrep

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Utilizes `ripgrep` (rg) to efficiently search replay result files for SQL queries that exceeded a specified duration. This example shows how to find queries with execution times greater than 1 second or 6 seconds by pattern matching on the `durationMs` field.

```sh
# Find SQLs with execution time exceeding 1s
rg "\"durationMs\":\d{4}\" output/replay

# Find SQLs with execution time exceeding 6s
rg -e "\"durationMs\":[6-9]\d{3}\" -e "\"durationMs\":\d{5}\" output/replay
```

--------------------------------

### Dodo Building from Source

Source: https://github.com/thearas/dodo/blob/master/llms.txt

Instructions for building the Dodo project from its source code using Make. It also covers building with optional performance enhancements like hyperscan.

```shell
make
make build-hyper
```

--------------------------------

### Format String Placeholders

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Demonstrates the use of Go's fmt.Sprintf-like placeholders and built-in tags within a format string.

```go
format: 'substr length 1-5: {{%s}} and a build-in tags: {{preposition_simple}}'
```

--------------------------------

### Dodo Configuration and Parameter Priority

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Explains how Dodo CLI parameters can be set via command-line arguments, environment variables (prefixed with DORIS_xxx), or configuration files. It outlines the priority order for these settings.

```APIDOC
dodo --help

Parameter Priority (High to Low):
1. Command-line arguments
2. Environment variables (e.g., `DORIS_HOST=xxx` is equivalent to `--host xxx`)
3. Configuration file specified by `--config`
4. Default configuration file `~/.dodo.yaml`
```

--------------------------------

### Bash: AI Data Generation from DDL and Query

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Demonstrates generating data using AI by providing DDL files and a SQL query, executed from a specific directory.

```bash
dodo gendata -C example.dodo.yaml --ddl 'ddl/*.sql' --query "$(cat sql/*)"
```

--------------------------------

### Dodo Configuration and Environment Variables

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Explains how to configure the Dodo tool using environment variables or configuration files. It details the priority order for parameters: command-line arguments, environment variables, `--config` file, and default `~/.dodo.yaml`.

```APIDOC
Configuration and Parameter Priority:

1. Command-line arguments
2. Environment variables (prefixed with `DORIS_`)
3. Configuration file specified via `--config`
4. Default configuration file `~/.dodo.yaml`

Environment Variables:
- `DORIS_HOST=xxx`: Equivalent to `--host xxx`.
- `DORIS_YES=1` or `DORIS_YES=0`: Automates confirmation prompts (yes/no).

Configuration File (`.yaml`):
- Supports parameters like `host`, `port`, `http-port`, `user`, `dbs`, `llm`, `llm-api-key`, `anonymize`.
```

--------------------------------

### Format String Configuration

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Applies a format string after column data generation, allowing Go's fmt.Sprintf-like placeholders and built-in tags.

```yaml
columns:
  - name: t_str
    format: 'substr length 1-5: {{%s}} and a build-in tags: {{preposition_simple}}'
    length:
      min: 1
      max: 5
```

--------------------------------

### Dodo Import Debugging Tip

Source: https://github.com/thearas/dodo/blob/master/introduction.md

A tip for debugging the 'dodo import' process by showing the specific curl command used for StreamLoad, which is helpful for reproduction and troubleshooting.

```APIDOC
> [!TIP]
> 
> - Specifying -Ldebug during import shows the specific curl command, which is helpful for reproducing and troubleshooting issues.
```

--------------------------------

### Dodo Workflows

Source: https://github.com/thearas/dodo/blob/master/llms.txt

Illustrates typical usage patterns and workflows for the Dodo CLI tool, covering scenarios with and without data generation requirements.

```APIDOC
Typical Dodo Workflows:

1. No data generation needed:
   `Dump -> Replay -> Diff Replay Results`

2. Data generation needed:
   `Dump -> Create Schemas (Optional) -> Generate and Import Data -> Replay -> Diff Replay Results`
```

--------------------------------

### Dodo CLI: AI-Assisted Data Generation

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Demonstrates using Dodo's `gendata` command with AI models (OpenAI/Deepseek) for data generation. It shows how to specify models, API keys, queries, and anonymization.

```bash
# Generate data from exported t1, t2 tables
dodo gendata --dbs db1 --tables t1,t2 \
    --llm 'deepseek-chat' --llm-api-key 'sk-xxx' \
    --query 'select * from t1 join t2 on t1.a = t2.b where t1.c IN ("a", "b", "c") and t2.d = 1' \
    --anonymize # anonymize sql before sending to LLM
```

```bash
# Generate data from any create-table and query
cd example/usercase
dodo gendata -C example.dodo.yaml --ddl 'ddl/*.sql' --query "$(cat sql/*)"
```

```bash
# Use --prompt to add additional hints
dodo gendata ... --prompt 'Generate 1000 rows for each table'
```

--------------------------------

### Batch Replay with Dodo CLI

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Demonstrates large-scale batch replay of SQL logs by processing data in hourly batches. It uses `dodo dump` to extract data within specific time ranges and `dodo replay` to execute the SQL, with `dodo diff` for result comparison. The script iterates through days and hours to manage large datasets efficiently.

```sh
YEAR_MONTH="2025-03" # <-- Change this line
dump DORIS_YES=1

for day in {1..31} ; do
  day=$(printf "%02d" $day)

  for hour in {0..23} ; do
      hour=$(printf "%02d" $hour)
      output=output/$day/$hour
      sql=$output/q0.sql

      echo "dumping and replaying at $day-$hour"

      # Dump
      dodo dump --dump-query --from "$YEAR_MONTH-$day $hour:00:00" --to "$YEAR_MONTH-$day $hour:59:59" --audit-log-table __internal_schema.audit_log --output "$output"

      # Replay, clear previous replay results, 50 clients concurrently, run continuously
      dodo replay -f "$sql" --result-dir result --clean --client-count 50 --speed 999999

      # View replay results
      dodo diff --min-duration-diff 1s --original-sqls $sql result -Ldebug 2>&1 | tee -a "result-$day.txt"
  done
done
```

--------------------------------

### Dodo CLI Commands Overview

Source: https://github.com/thearas/dodo/blob/master/llms.txt

Lists the main commands available in the Dodo CLI tool, detailing their primary functions for database operations and workflow management.

```APIDOC
Dodo CLI Commands:

- dodo dump: Dumps database schemas and/or queries from audit logs.
- dodo create: Creates tables and views in a database, often using schemas dumped by `dodo dump`.
- dodo gendata: Generates CSV data for tables based on SQL schema files.
- dodo import: Imports data from CSV files (often generated by `dodo gendata`) into database tables.
- dodo replay: Replays SQL queries from files (typically generated by `dodo dump`).
- dodo diff: Compares the results of replay sessions.
- dodo anonymize: Anonymizes identifiers in SQL statements.
- dodo completion: Generates shell completion scripts.
```

--------------------------------

### Dodo Replay Command

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Provides details on the `dodo replay` command for replaying SQL queries against a Doris cluster. It includes options for specifying the host, SQL files, output directories, and performance tuning.

```APIDOC
dodo replay [options]

- `-H <host>`: Specifies the Doris FE host to connect to.
- `-f <file>`: Path to the SQL file to replay.
- `-O <dir>`: Output directory for replay results.
- `--result-dir <dir>`: Directory to store replay results.
- `--clean`: Cleans previous replay results before starting.
- `--client-count <num>`: Number of concurrent clients for replay.
- `--speed <num>`: Controls replay speed (e.g., `999999` for near real-time).
- `--log-level <level>`: Sets the logging level (`debug`, `trace`). `trace` provides detailed execution information.
```

--------------------------------

### Batch Replay for Large Datasets

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Provides a script for replaying large volumes of SQL logs, such as a month's worth of data, by processing them in hourly batches. It includes steps for exporting and replaying data hour by hour.

```shell
export YEAR_MONTH="2025-03" # <-- Change this line
export DORIS_YES=1

for day in {1..31} ; do
  day=$(printf "%02d" $day)

  for hour in {0..23} ; do
      hour=$(printf "%02d" $hour)
      output=output/$day/$hour
      sql=$output/q0.sql

      echo "dumping and replaying at $day-$hour"

      # Export data in hourly batches
      dodo dump --dump-query --from "$YEAR_MONTH-$day $hour:00:00" --to "$YEAR_MONTH-$day $hour:59:59" --audit-log-table __internal_schema.audit_log --output "$output"

      # Replay data, clean previous results, use 50 concurrent clients, run continuously
      dodo replay -f "$sql" --result-dir result --clean --client-count 50 --speed 999999

      # Compare replay results
      dodo diff --min-duration-diff 1s --original-sqls $sql result -Ldebug 2>&1 | tee -a "result-$day.txt"
  done
done

```

--------------------------------

### Dodo Create Commands

Source: https://github.com/thearas/dodo/blob/master/README.md

Commands for creating database schemas, tables, and views. Can create objects based on dumped schemas or execute specific SQL DDL files.

```APIDOC
dodo create --help
  Displays help information for the create command.

dodo create --dbs db1,db2 --host <host> --port <port> --user root --password '***'
  Creates all tables and views for databases db1 and db2, automatically locating dumped schemas in the 'output/' directory.

dodo create --ddl 'dir/*.sql' --db db1
  Executes SQL DDL statements from files matching '*.sql' in the 'dir/' directory to create objects in database db1.
```

--------------------------------

### Dodo CLI: Replay Audit Logs

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Provides instructions for using Dodo's `replay` command to re-execute SQL queries captured in audit logs. It covers dumping queries and replaying them, noting that replay results overwrite previous files.

```APIDOC
Dodo Replay Commands:

Dump Queries:
  dodo dump --dump-query --audit-logs fe.audit.log
    - Dumps SQL queries from specified audit log files.
    - Options:
      --dump-query: Enables query dumping.
      --audit-logs: Path to the audit log file(s).

Replay Queries:
  dodo replay -f output/q0.sql
    - Replays SQL queries from a specified file.
    - Results are saved in the 'output/replay' directory by default.
    - Each file in the replay directory represents a client's query results.
    - Parameters:
      -f, --file: Path to the SQL file to replay.
    - Note: Each replay operation will overwrite existing result files for the same client.

Replay Speed and Concurrency:
  - SQL from different clients runs concurrently.
  - SQL from the same client runs serially.
  - Intervals between SQLs from the same client are calculated based on:
    Interval duration = sql2 start time - sql1 start time - sql1 execution duration
```

--------------------------------

### Dodo Tool Deployment and Impact

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Addresses concerns about distributing the Dodo tool to clients and its impact on production environments. It clarifies that the Linux binary is self-contained and typically performs read-only operations.

```APIDOC
Client Deployment and Production Impact:

- **Distribution**: Provide clients with the latest pre-compiled binary release. The Linux version is self-contained and requires no external dependencies.
- **Production Impact**: By default, Dodo performs read-only operations on the cluster. It does not modify data or cluster state unless explicitly instructed (e.g., via replay).
- **Resource Consumption**: During export, setting `--parallel=1` can minimize resource usage. Memory consumption is typically low (tens of MB), and execution time is usually in seconds.
- **Replay**: When replaying, ensure sufficient resources and consider batch execution.
```

--------------------------------

### Bash: Replaying SQL from Exported File

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Executes a replay of SQL statements from an exported file. The results are stored in the default 'output/replay' directory, with each file representing a client's execution.

```bash
dodo replay -f output/q0.sql
```

--------------------------------

### Dodo Replay Commands

Source: https://github.com/thearas/dodo/blob/master/README.md

Commands for replaying SQL queries from dump files. Allows filtering by time range, users, and databases, controlling replay speed, and specifying output directories for results.

```APIDOC
dodo replay --help
  Displays help information for the replay command.

dodo replay --host <host> --port <port> --user root --password '***' -f output/sql/q0.sql
  Replays SQL queries from the 'output/sql/q0.sql' file, connecting to the specified host and port.

dodo replay -f output/sql/q0.sql \
    --from '2024-09-20 08:00:00' --to '2024-09-20 09:00:00' \
    --users 'readonly,root' --dbs 'db1,db2' \
    --speed 0.5 \
    --result-dir output/replay \
    --clean
  Replays queries from 'output/sql/q0.sql', filtering by time, users, and databases. It sets the replay speed to 0.5, saves results to 'output/replay', and cleans the directory beforehand.
```

--------------------------------

### Dodo Command-line Autocompletion

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Provides instructions on how to enable shell autocompletion for the Dodo CLI. This enhances user experience by offering command and argument suggestions.

```APIDOC
dodo completion --help

Usage:
  dodo completion

Description:
  Prints shell completion scripts for the Dodo CLI. Follow the instructions provided by the command to enable autocompletion for your shell.
```

--------------------------------

### Dodo Monitoring Log Levels

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Details the use of log levels for monitoring the dump and replay process. 'debug' provides a brief overview, while 'trace' offers detailed insights into SQL execution times and process steps.

```APIDOC
Monitoring Parameters:

--log-level <level>
  Sets the logging verbosity.
  - 'debug': Outputs a brief process summary.
  - 'trace': Shows detailed process information, including SQL execution time and steps during replay.
```

--------------------------------

### Bash: Exporting Audit Logs for Replay

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Command to export audit logs, specifically enabling the `--dump-query` option to prepare SQL statements for replaying the recorded operations.

```bash
dodo dump --dump-query --audit-logs fe.audit.log
```

--------------------------------

### Bash: AI Data Generation with Custom Prompt

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Shows how to add a custom prompt to the 'dodo gendata' command to influence the AI-generated data, such as specifying the number of rows per table.

```bash
dodo gendata ... --prompt '每张表生成 1000 行数据'
```

--------------------------------

### Finding Slow SQL Queries

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Shows how to use `rg` (ripgrep) or `grep` to find SQL queries that exceeded a specific execution duration (e.g., 1s or 6s) within replay output files.

```shell
# Find queries with execution time over 1 second (durationMs >= 1000)
rg "\"durationMs\":\d{4}" output/replay

# Find queries with execution time over 6 seconds (durationMs >= 6000)
rg -e "\"durationMs\":[6-9]\d{3}" -e "\"durationMs\":\d{5}" output/replay

```

--------------------------------

### Dodo Dump Parameters

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Details various command-line parameters for the 'dodo dump' operation, affecting table analysis, concurrency, statistics dumping, SQL selection, time range filtering, query duration, query states, syntax validation, audit log encoding, and data anonymization.

```APIDOC
--analyze: Automatically runs ANALYZE TABLE <table> WITH SYNC before dumping a table to make statistics more accurate. Default is off.
--parallel: Controls the dump concurrency. Increasing it speeds up the dump; decreasing it uses fewer resources. Default is min(machine_cores-2, 10).
--dump-stats: Also dumps table statistics when dumping tables. Statistics are dump to output/ddl/db.stats.yaml. Default is on.
--only-select: Whether to dump only SELECT statements. Default is on.
--from and --to: Dump SQL within a specified time range.
--query-min-duration: Minimum execution duration for dump SQL.
--query-states: States of the SQL to be dump, can be ok, eof, and err.
-s, --strict: Validates SQL syntax correctness when dumping from audit logs.
--audit-log-encoding: Audit log file encoding. Default is auto-detect.
--anonymize: Anonymizes data during dump, e.g., select * from table1 becomes select * from a.
--anonymize-xxx: Other anonymization parameters, see [Anonymization](#anonymization).
```

--------------------------------

### Bash: Comparing Replay Results

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Demonstrates comparing the results of two different replay operations or comparing an exported SQL file against its replay output, with options to filter by duration differences.

```bash
# Compare two replay directories
dodo diff output/replay1 output/replay2

# Compare original SQLs with replay results, showing differences over 2s
dodo diff --min-duration-diff 2s --original-sqls 'output/sql/*.sql' output/replay
```

--------------------------------

### Dodo Import Commands

Source: https://github.com/thearas/dodo/blob/master/README.md

Commands for importing generated data into databases. Supports importing data for specified tables or from CSV files.

```APIDOC
dodo import --help
  Displays help information for the import command.

dodo import --dbs db1,db2 --host <host> --http-port <http-port> --user root --password '***'
  Imports generated data for databases db1 and db2, connecting to the specified host and HTTP port.

dodo import --dbs db1 --table t1,t2
  Imports generated data specifically for tables t1 and t2 in database db1.

dodo import --tables db1.t1 --data data.csv
  Imports data from the 'data.csv' file into table t1 of database db1.
```

--------------------------------

### Bash: Customizing Replay Speed and Concurrency

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Illustrates controlling replay speed and concurrency using command-line arguments. `--speed` adjusts the time intervals between SQLs from the same client, while `--client-count` reallocates SQLs across a specified number of clients.

```bash
# Example: Slow down replay by half, use 50 clients with no interval
dodo replay --speed 0.5 --client-count 50

# Example: Fast forward replay by double, use 50 clients with no interval
dodo replay --speed 2 --client-count 50

# Example: Max speed and 50 clients for independent SQLs
dodo replay --speed 999999 --client-count 50
```

--------------------------------

### Precision and Scale Configuration

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Specifies the precision and scale for DECIMAL types, along with optional min/max values.

```yaml
columns:
  - name: t_decimal
    precision: 10
    scale: 3
    min: 100
    max: 102  # Actual maximum value is 102.999
```

--------------------------------

### Bash: Exporting Table Data to Storage

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Command to export table data to various storage locations like S3, HDFS, or local filesystem. It supports placeholders for database and table names in the target URL and allows specifying storage-specific parameters.

```bash
# Export t1, t2 from db1 to S3 with specific endpoint and credentials
dodo export --dbs db1 --tables t1,t2 --target s3 --url 's3://bucket/export/{db}/{table}_' -p timeout=60 -w s3.endpoint=xxx -w s3.access_key=xxx -w s3.secret_key=xxx
```

--------------------------------

### Dodo Dump Commands

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Commands for dumping database schemas, tables, views, and queries from audit logs. Supports dumping schema definitions, table statistics, and SQL queries from specified time ranges or log files. Output is organized into 'ddl' and 'sql' directories.

```APIDOC
dodo dump --help
  Displays help information for the dump command.

dodo dump --dump-schema
  Dumps CREATE statements for tables and views from a Doris database.
  By default, it also dumps table statistics.
  --analyze: Recommended if statistics differ significantly from actual data.
  --host, --port, --user, --password: Connection details for the database.
  --dbs: Comma-separated list of databases to dump.
  Default output: output/ddl/

dodo dump --dump-query
  Dumps queries from an audit log table or file.
  --only-select=false: Include statements other than SELECT (defaults to true).
  --audit-log-table <db.table>: Specify the audit log table name.
  --from 'YYYY-MM-DD HH:MM:SS': Start timestamp for query dumping.
  --to 'YYYY-MM-DD HH:MM:SS': End timestamp for query dumping.
  --audit-logs 'file1,file2*': Specify audit log files (wildcards supported).
  Default output: output/sql/

Notes:
- When dumping from log files, each file corresponds to a separate output SQL file (e.g., q0.sql, q1.sql).
- When dumping from a log table, all queries are written to a single file (e.g., q0.sql).
- Each dump operation overwrites previous dump SQL files.
```

--------------------------------

### Dodo Other Replay Parameters

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Lists additional parameters for controlling the replay process, including cluster selection, result directory, user filtering, time range specification, and hash result row limits for consistency checks.

```APIDOC
Other Replay Parameters:

-c, --cluster <cluster_name>
  Specifies the cluster for replay, applicable only in Cloud mode.

--result-dir <directory>
  Sets the directory for replay results. Defaults to 'output/replay'.

--users <user_list>
  Filters SQLs to replay only those initiated by specified users. Defaults to all users.

--from <timestamp>, --to <timestamp>
  Replays SQLs within a specified time range.

--max-hash-rows <number>
  Maximum number of hash result rows to record for consistency checks. Defaults to no hashing.

--max-conn-idle-time <duration>
  Maximum idle time for a client connection before recycling. Defaults to '5s'. If the interval between consecutive SQLs exceeds this, the connection is recycled.
```

--------------------------------

### Dodo Anonymize Command Parameters

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Documentation for the `dodo anonymize` command, detailing its parameters for SQL data anonymization. It covers options for input files, reserving specific IDs, setting minimum ID length for anonymization, and choosing between 'hash' or 'minihash' methods.

```APIDOC
dodo anonymize --help

Parameters:
  -f, --file: Read SQL from a file. If '-', read from standard input.
  --anonymize-reserve-ids: Reserve ID fields, do not anonymize them.
  --anonymize-id-min-length: ID fields with a length less than this value will not be anonymized. Default is `3`.
  --anonymize-method: Hash method, `hash` or `minihash`. The latter generates a concise dictionary based on the former, making anonymized IDs shorter. Default is `minihash`.
  --anonymize-minihash-dict: When the hash method is `minihash`, specify the concise dictionary file. Default is `./dodo_hashdict.yaml`.

Notes:
- Anonymization uses the Go version of Doris Antlr4 Parser, which is currently case-insensitive. For example, `table1` and `TABLE1` will produce the same result.
```

--------------------------------

### YAML: Integer Generator for Varchar

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Demonstrates using an integer generator for a varchar column in YAML configuration, specifying a format and a range for the generated integers.

```yaml
columns:
  - name: t_varchar2
    format: "year: {{%d}}, month: {{month}}"
    gen:
      type: int
      min: 1997
      max: 2097
```

--------------------------------

### Table-Level Generation Rules

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Illustrates how to define table-specific configurations, overriding global defaults for row count, column NULL frequency, and value ranges.

```yaml
tables:
  - name: employees
    row_count: 100  # Optional, default is 1000 (can also be specified by --rows)
    columns:
      - name: department_id
        null_frequency: 0.1  # 10% NULL
        min: 1
        max: 10
```

--------------------------------

### Dodo Dump Commands

Source: https://github.com/thearas/dodo/blob/master/README.md

Commands for dumping database schemas and queries. Supports dumping from databases or audit log files/tables, with options to specify databases, hosts, ports, users, passwords, and audit log sources.

```APIDOC
dodo dump --help
  Displays help information for the dump command.

dodo dump --dump-schema --dbs db1,db2 --host <host> --port <port> --user root --password '***'
  Dumps schemas for databases db1 and db2, connecting to the specified host and port with root user credentials.

dodo dump --dump-schema --dump-query --dbs db1,db2 --audit-logs 'fe.audit.log,fe.audit.log.20240802-1'
  Dumps schemas and queries from audit logs (files) for databases db1 and db2.

dodo dump --dump-query --audit-log-table <db.table> --from '2024-11-14 18:45:25' --to '2024-11-14 18:45:26'
  Dumps queries from a specified audit log table within a given time range. Requires audit plugin to be enabled.

--only-select=false
  Option to dump all SQL statements, not just SELECT statements (default is true).
```

--------------------------------

### Parts Generator with Literals and Generators

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Demonstrates the 'parts' generator combining literal strings with generated values. The parts are sequentially inserted into the format string's placeholders.

```yaml
columns:
  - name: t_null_char # char(10)
    format: "{{%s}}--{{%02d}}" # parts must be used with format
    gen:
      parts:
        - "prefix"
        - gen:
            enum: [2, 4, 6, 8, 10]
```

--------------------------------

### YAML: Struct Generator for Varchar

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Shows how to generate data for a varchar column using a struct type in YAML configuration, defining the structure of the struct.

```yaml
columns:
  - name: t_varchar2
    gen:
      type: struct<foo:int, bar:text>
```

--------------------------------

### Dodo Generate Data Commands

Source: https://github.com/thearas/dodo/blob/master/README.md

Commands for generating fake data for tables. Supports generating data from create-table SQL statements (MySQL, Hive, etc.) or directly from queries. Includes AI-powered generation using Deepseek LLM.

```APIDOC
dodo gendata --help
  Displays help information for the gendata command.

dodo gendata --ddl table.sql
  Generates data based on the provided create-table SQL statement in 'table.sql'.

dodo gendata --dbs db1,db2 --host <host> --port <port> --user root --password '***'
  Generates data for databases db1 and db2, automatically finding dumped schemas in the 'output/' directory.

dodo gendata --dbs db1 --genconf example/gendata.yaml
  Generates data for db1 using custom configurations specified in 'example/gendata.yaml'.

dodo gendata -l 'deepseek-chat' -k '<deepseek-api-key>' --ddl table.sql --query 'select xxx'
  Generates data using the Deepseek LLM, providing the API key, a DDL statement, and a sample query.
```

--------------------------------

### Doris Statistics Export and Analysis

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Explains how exported statistics might differ from actual values if sampling is used. Recommends using `--analyze` during export or running `ANALYZE` commands beforehand.

```APIDOC
Statistics Discrepancy:

If the `method` field in `stats.yaml` shows `SAMPLE`, the exported statistics (like NDV) may deviate significantly from actual values.

Recommendation:
- Use the `--analyze` flag during `dodo dump`.
- Alternatively, execute `ANALYZE DATABASE WITH SYNC` or `ANALYZE TABLE WITH SYNC` on the Doris cluster before exporting statistics.
```

--------------------------------

### Anonymization Consistency

Source: https://github.com/thearas/dodo/blob/master/README.md

Note on maintaining consistent anonymization results by keeping the './dodo_hashdict.yaml' file, which stores hashing dictionaries.

```text
> [!NOTE]
> Keep "./dodo_hashdict.yaml" if you want the result to be consistent (put it at current directory, or specify by "--anonymize-minihash-dict").
```

--------------------------------

### Custom Generator using Go Code

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Enables custom data generation logic by embedding Go code directly within the configuration. This allows for complex, stateful, or logic-heavy generation using the Go standard library. Use with caution due to readability concerns.

```go
columns:
  - name: t_varchar
    gen:
      golang: |
        import "fmt"

        var i int
        func gen() any {
            i++
            return fmt.Sprintf("Is odd: %v.", i%2 == 1)
        }
```

--------------------------------

### Dodo Export Commands

Source: https://github.com/thearas/dodo/blob/master/README.md

Commands for exporting table data. This section only shows the help command, indicating the functionality exists.

```APIDOC
dodo export --help
  Displays help information for the export command.
```

--------------------------------

### Dodo Custom Generation Rules Configuration

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Explains how to use custom generation rules via a YAML configuration file specified with 'dodo gendata --genconf'. It also mentions the possibility of concatenating multiple YAML files for sequential rule application.

```yaml
# Dataset 1
null_frequency: 0
type:
  ...
tables:
  ...
---

```

--------------------------------

### Dodo Anonymize Usage

Source: https://github.com/thearas/dodo/blob/master/README.md

Demonstrates two methods for anonymizing SQL queries: piping SQL to 'dodo anonymize' or using the '--anonymize' flag during the 'dodo dump' process.

```bash
echo "select * from table1" | dodo anonymize -f -

dodo dump ... --anonymize
```

--------------------------------

### YAML with Go: Custom String Generation

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Illustrates generating custom string data using an embedded Go function within a YAML configuration. The Go code defines a stateful generator.

```yaml
columns:
  - name: t_varchar
    gen:
      golang: |
        import "fmt"
        
        var i int
        func gen() any {
            i++
            return fmt.Sprintf("Is odd: %v.", i%2 == 1)
        }
```

--------------------------------

### Dodo Diff Command

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Explains the `dodo diff` command used to compare replay results. It allows filtering differences based on duration and specifying original SQLs.

```APIDOC
dodo diff [options]

- `--min-duration-diff <duration>`: Minimum duration difference to report (e.g., `1s`).
- `--original-sqls <file>`: Path to the original SQL file for comparison.
- `<result_dir>`: Directory containing replay results.
- `-L<level>`: Sets the log level (e.g., `debug`).
```

--------------------------------

### Troubleshooting Exported SQL Syntax Errors

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Addresses potential syntax errors in exported SQL. It suggests using the `-s, --strict` flag during export and checking for issues like reversed escape characters or corrupted audit logs.

```APIDOC
SQL Syntax Errors in Exported Files:

1.  **Use Strict Mode**: Add the `-s` or `--strict` flag to `dodo dump` to validate SQL syntax during export.
2.  **Check Escape Characters**: The tool might unescape `\r`, `\n`, `\t`. If the original SQL contained these characters, it could lead to syntax errors.
3.  **Audit Log Integrity**: Verify the integrity of the audit logs themselves, as corruption can cause issues.

If errors persist, manual correction of the exported SQL file might be necessary.
```

--------------------------------

### Length Configuration for String/Array/Map

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Specifies the length range for bitmap, string, array, or map types.

```yaml
columns:
  - name: t_str
    # or just `length: <int>` if min and max are the same, like `length: 5`
    length:
      min: 1
      max: 5
```

--------------------------------

### YAML: HLL Type Generation from Other Column

Source: https://github.com/thearas/dodo/blob/master/introduction-zh.md

Shows how to generate HLL type data by referencing another column in YAML configuration, using its value for hashing.

```yaml
columns:
  - name: t_hll # t_hll 的值将为 `hll_hash(t_str)`
    from: t_str
```

--------------------------------

### Dodo Diff Commands

Source: https://github.com/thearas/dodo/blob/master/README.md

Commands for comparing replay results. Allows diffing based on minimum duration differences or comparing two different replay result directories.

```APIDOC
dodo diff --help
  Displays help information for the diff command.

dodo diff --min-duration-diff 200ms --original-sqls 'output/sql/*.sql' output/replay
  Compares replay results in 'output/replay' against original SQLs in 'output/sql/*.sql', flagging queries that are slower by more than 200ms.

dodo diff replay1/ replay2/
  Compares the replay results found in the 'replay1/' directory against those in the 'replay2/' directory.
```

--------------------------------

### Parts Generator with Format String

Source: https://github.com/thearas/dodo/blob/master/introduction.md

Utilizes the 'parts' generator in conjunction with a 'format' string to construct a final value by combining multiple generated parts. The parts are filled into placeholders like {{%s}} or {{%02d}} in the format string.

```yaml
columns:
  - name: date1 # date
    format: "{{year}}-{{%02d}}-{{%02d}}"
    gen:
      parts:
        - gen: # month
            type: int
            min: 1
            max: 12
        - gen: # day
            ref: table1.column1
```