YB Voyager

https://github.com/yugabyte/yb-voyager
Admin
YugabyteDB Voyager is an open-source data migration engine that accelerates cloud-native adoption by...
Tokens:19,629
Snippets:101
Trust Score:9.7
Update:1 year ago
Show doc for...
Tokens:
Raw
### Apache License Boilerplate

Source: https://github.com/yugabyte/yb-voyager/blob/main/licenses/APACHE-LICENSE-2.0.txt

A template for applying the Apache License to your project, including copyright and license notices. This block specifies the standard text to be included in files.

```APIDOC
Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```

--------------------------------

### yb-voyager Installer Script Version Flag

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Introduces the `--version` flag for the voyager installer script, allowing users to specify the exact version of yb-voyager to install.

```Shell
yb-voyager-installer.sh --version <version_number>
```

--------------------------------

### Create Index on Low-Cardinality Column (Example)

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

Demonstrates the SQL syntax for creating a B-tree index on a column. Indexes on low-cardinality columns can lead to poor data distribution across tablets in distributed databases.

```sql
CREATE INDEX idx_test_low_card_employed ON schema1.test_hotspot_timestamp USING btree (employed);
```

--------------------------------

### yb-voyager General Commands & Fixes

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Covers general yb-voyager commands like 'get data-migration-report' and 'end migration', along with bug fixes related to transaction retries and datatype mapping.

```APIDOC
yb-voyager get data-migration-report
  - Description: Retrieves a report for data migration.
  - Bug Fixes:
    - Did not work in dockerised version if SSL certificates were previously passed.

yb-voyager end migration
  - Description: Ends an ongoing migration process.
  - Bug Fixes:
    - Did not work in dockerised version if SSL certificates were previously passed.

General Bug Fixes:
  - Workaround for YugabyteDB bug where batched queries in a transaction were partially retried without respecting transaction atomicity.
  - Increased maximum retry count of event batch to 50 for import data streaming.
  - Changed Oracle DATE:date datatype mapping to DATE:timestamp to prevent time data loss.
```

--------------------------------

### YB-Voyager Known Issue: Import State Incompatibility

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Highlights a known issue where YB-Voyager v1.6 cannot continue data import operations started with v1.5 or earlier due to an incompatible import data state structure.

```APIDOC
YB-Voyager Known Issue: Import State Incompatibility

- Voyager v1.6 uses a different and incompatible structure for representing the import data state compared to earlier releases.
- As a result, Voyager v1.6 cannot 'continue' a data import operation that was started using Voyager v1.5 or earlier.
```

--------------------------------

### Unsupported Query Construct: System Column xmin

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

Identifies the use of the system column 'xmin' in queries as unsupported in YugabyteDB. The provided SQL example includes 'xmin' alongside other system columns, indicating a query construct that needs modification.

```sql
System Column - xmin
SELECT ctid, tableoid, xmin, xmax, cmin, cmax FROM schema1.employees2
Level 2

Category Description
Source database queries not supported in YugabyteDB, identified by scanning system tables.

SQL Statement
SELECT ctid, tableoid, xmin, xmax, cmin, cmax
FROM schema1.employees2
```

--------------------------------

### Unsupported System Columns in YugabyteDB Queries

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

YugabyteDB does not support certain PostgreSQL system columns like 'xmin', 'xmax', 'cmin', and 'cmax' in queries. This snippet shows an example SQL statement that would fail due to these unsupported columns.

```SQL
SELECT ctid, tableoid, xmin, xmax, cmin, cmax
FROM schema1.employees2
```

--------------------------------

### yb-voyager Live Migration Commands

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Documentation for yb-voyager's live migration capabilities, including fall-back options and support for partitioned tables and case-sensitive names from PostgreSQL sources.

```APIDOC
yb-voyager live migration
  - Description: Supports live migration from source databases.
  - Features:
    - Supports fall-back options for PostgreSQL source databases.
    - Supports migration of partitioned tables and multiple schemas from PostgreSQL.
    - Supports migration of case-sensitive table/column names from PostgreSQL.
  - Enhancements:
    - Detects and skips unsupported data types before starting live migration from PostgreSQL (with user confirmation).
  - Known Issues:
    - In dockerised versions, 'get data-migration-report' and 'end migration' may not work if SSL certificates were previously passed in export/import data commands.
```

--------------------------------

### yb-voyager CLI Command Signatures and Parameters

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

This section details common yb-voyager CLI commands, their purpose, and key parameters. It covers export, import, and schema analysis functionalities.

```APIDOC
yb-voyager export data
  --source-db-type=<db_type> (e.g., mysql, oracle, postgres)
  --source-db-host=<host>
  --source-db-port=<port>
  --source-db-user=<user>
  --source-db-password=<password> (or use env var SOURCE_DB_PASSWORD)
  --export-dir=<directory>
  --accelerated (optional, for faster export)
  --source-db-ssl=<true|false> (optional, for SSL connection)
  --pg-dump-args-file=<path> (optional, path to pg_dump-args.ini)

yb-voyager export schema
  --source-db-type=<db_type>
  --source-db-host=<host>
  --source-db-port=<port>
  --source-db-user=<user>
  --source-db-password=<password> (or use env var SOURCE_DB_PASSWORD)
  --export-dir=<directory>

yb-voyager import data file
  --source-db-type=<db_type>
  --source-db-host=<host>
  --source-db-port=<port>
  --source-db-user=<user>
  --source-db-password=<password> (or use env var SOURCE_DB_PASSWORD)
  --file-table-map=<table_name>=<file_path_or_glob> (e.g., "users=data/users.csv", "orders=data/orders/*.csv")
  --data-dir=<directory> (optional, can be S3 path)
  --escape-char=<char> (optional, default is \" for CSV)
  --quote-char=<char> (optional, default is \" for CSV)
  --null-string=<string> (optional, specifies string representation of NULL)
  --file-format=<format> (e.g., csv, text; default is csv)
  --delimiter=<char> (optional, default is , for CSV, \t for TEXT)

yb-voyager analyze-schema
  --source-db-type=<db_type>
  --source-db-host=<host>
  --source-db-port=<port>
  --source-db-user=<user>
  --source-db-password=<password> (or use env var SOURCE_DB_PASSWORD)
  --export-dir=<directory>

Environment Variables:
  SOURCE_DB_PASSWORD: Password for the source database.
  TARGET_DB_PASSWORD: Password for the target database.
  BETA_FAST_DATA_EXPORT: Set to 1 to enable faster data export for MySQL/Oracle.

Configuration File:
  /etc/yb-voyager/pg_dump-args.ini: Custom arguments for pg_dump command used in export.

Notes:
- yb-voyager explicitly closes database connections upon exit.
- Import status reporting for CSV files with empty lines has been fixed.
- Version incompatibility exists for continuing imports started with older yb-voyager versions (pre-v1.4).
```

--------------------------------

### New YB-Voyager Commands for Live Migration

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Introduces new commands for managing live migration processes, including reporting and finalization.

```APIDOC
New YB-Voyager Commands:

- `yb-voyager get data-migration-report`
  - Description: Displays table-wise statistics during and post live migration.

- `yb-voyager end migration`
  - Description: Completes migration by cleaning up metadata on all involved databases and backing up migration artifacts (reports, schema, data, logs).
```

--------------------------------

### yb-voyager Schema Migration Commands

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Documentation for yb-voyager commands related to schema migration, including assessing migration readiness, exporting schema, and analyzing schema. Details new features and bug fixes.

```APIDOC
yb-voyager assess migration (TECH PREVIEW)
  - Description: Analyzes a source database (PostgreSQL supported) to optimize migration to YugabyteDB. Generates an assessment report in HTML/JSON formats.
  - Usage:
    - Can be run via bash/psql scripts for offline analysis.
    - Automatically modifies CREATE TABLE DDLs with recommendations when 'export schema' is run.

yb-voyager export schema
  - Description: Exports the database schema.
  - Enhancements:
    - Automatically modifies CREATE TABLE DDLs to incorporate recommendations from the 'assess migration' feature.

yb-voyager analyze schema
  - Description: Analyzes the database schema.
  - Bug Fixes:
    - Fixed a bug reporting false-positive unsupported cases for 'FETCH CURSOR'.
    - Fixed a bug where schema analysis report had an incorrect value for invalid count of objects in summary.
```

--------------------------------

### Unsupported Query Construct: System Column ctid

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

Detects the use of the system column 'ctid' in queries, which is not supported in YugabyteDB. The example SQL demonstrates a SELECT statement referencing 'ctid', 'tableoid', and other system columns.

```sql
System Column - ctid
SELECT ctid, tableoid, xmin, xmax, cmin, cmax FROM schema1.employees2
Level 2

Category Description
Source database queries not supported in YugabyteDB, identified by scanning system tables.

SQL Statement
SELECT ctid, tableoid, xmin, xmax, cmin, cmax
FROM schema1.employees2
Description
System column 'ctid' is not yet supported in YugabyteDB.
Refer to the [docs](https://docs.yugabyte.com/preview/yugabyte-voyager/known-issues/postgresql/#system-columns-is-not-yet-supported "https://docs.yugabyte.com/preview/yugabyte-voyager/known-issues/postgresql/#system-columns-is-not-yet-supported") for more details.
```

--------------------------------

### yb-voyager Export/Import Data Commands

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Documentation for yb-voyager commands related to exporting and importing data, including handling of partitioned tables and case sensitivity. Covers bug fixes and enhancements for these operations.

```APIDOC
yb-voyager export data
  - Description: Exports data from a source database.
  - Bug Fixes:
    - Fixed issue where partitioned tables were ignored if only the root table name was specified in --table-list.
    - Handles case-sensitive table/column names from PostgreSQL, allowing table names in any form (case sensitive/insensitive/quoted/unquoted) in --table-list.

yb-voyager import data
  - Description: Imports data into a YugabyteDB database.
  - Enhancements:
    - Supports importing data via the root table name for partitioned tables in PostgreSQL, allowing changes to leaf table names or partitioning logic.
  - Bug Fixes:
    - Fixed issues related to SSL certificates when using dockerised yb-voyager with commands like 'get data-migration-report' and 'end migration'.
    - Fixed a bug where schema analysis report had an incorrect value for invalid count of objects in summary.

yb-voyager import data to source replica
  - Description: Imports data to a source replica.
  - Bug Fixes:
    - Fixed issues related to SSL certificates when using dockerised yb-voyager with commands like 'get data-migration-report' and 'end migration'.
```

--------------------------------

### Unsupported Feature: Stored Generated Columns

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

Identifies stored generated columns as unsupported in YugabyteDB, specifically for versions prior to 2.25.0.0. The example shows a column 'full_name' generated from other columns, suggesting a workaround using triggers.

```sql
Stored Generated Columns
schema1.employees2
Level 1

Category Description
Features of the source database that are not supported on the target YugabyteDB.

Object Type
TABLE
Object Name
schema1.employees2
SQL Statement
CREATE TABLE schema1.employees2 (
    id integer NOT NULL,
    first_name character varying(50) NOT NULL,
    last_name character varying(50) NOT NULL,
    full_name character varying(101) GENERATED ALWAYS AS ((((first_name)::text || ' '::text) || (last_name)::text)) STORED,
    department character varying(50)
);
Supported In (versions)
>=2.25.0.0 (2.25 series)
Description
Stored generated columns are not supported in YugabyteDB. Detected columns are (full_name). Using Triggers to update the generated columns is one way to work around this issue.
Refer to the [docs](https://docs.yugabyte.com/preview/yugabyte-voyager/known-issues/postgresql/#generated-always-as-stored-type-column-is-not-supported "https://docs.yugabyte.com/preview/yugabyte-voyager/known-issues/postgresql/#generated-always-as-stored-type-column-is-not-supported") for more details.
```

--------------------------------

### Sharding and Sizing Recommendations

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

This section outlines recommendations for sharding and sizing for the target YugabyteDB environment based on the migration assessment. It lists colocated and sharded tables, along with suggested instance specifications for vCPU, memory, and connection pooling.

```text
Sharding Recommendations
------------------------

Colocated Tables

Sharded Tables

*   public.mixed_data_types_table1
*   public.child_table
*   public.parent_table
*   schema1.test_hotspot_timestamp

*   schema1.employees2

Sizing Recommendations
----------------------

Parameter

Recommendation

Num of Nodes

3

vCPU per instance

4

Memory per instance(GiB)

16

Optimal select connections per node

8

Optimal insert connections per node

12
```

--------------------------------

### Process Management

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Information on how yb-voyager manages child processes.

```APIDOC
Process Management
  - Behavior: yb-voyager now gracefully shuts down all child processes on exit.
  - Purpose: Prevents orphan processes.
```

--------------------------------

### Skip Unsupported Query Constructs Detection

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Unsupported Query Constructs detection is now skipped if `pg_stat_statements` is not loaded via `shared_preloaded_libraries` in the source database.

```APIDOC
assess-migration

Bug Fix:
  Skips detection of unsupported query constructs when `pg_stat_statements` is not loaded.
```

--------------------------------

### yb-voyager CLI Commands and Flags

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Documentation for key yb-voyager commands and flags related to data migration, schema analysis, and migration management. This includes changes to command names, flag behavior, and default values.

```APIDOC
yb-voyager get data-migration-report
  - Description: Retrieves a report on the data migration status.
  - Enhancements: Improved UX with pagination for large sets of tables.

yb-voyager initiate cutover to target
  - Description: Initiates the cutover process to the target YugabyteDB.
  - Bug Fixes: Resolved an issue where fallback intent was stored even if the process was aborted.

yb-voyager end migration
  - Description: Completes and cleans up the migration process.
  - Bug Fixes: Fixed an issue where the source database was not cleaned up if '--save-migration-reports' was set to false.

--post-import-data flag
  - Renamed to: --post-snapshot-import
  - Purpose: Incorporates live migration workflows, allowing index creation concurrently with CDC phase.

--parallel-jobs flag
  - Description: Controls the number of parallel jobs for importing data to YugabyteDB.
  - Changes: Default value reduced to 0.25 of total cores (from 0.5) to improve target YugabyteDB stability.

--table-list argument
  - Description: Specifies a list of tables for export data operations.
  - Bug Fixes: Fixed failure when a single case-sensitive table name was provided.

--save-migration-reports flag
  - Description: Controls whether migration reports are saved.
  - Bug Fixes: Affects cleanup in 'yb-voyager end migration' command.
```

--------------------------------

### Assess Migration Unsupported Index Methods

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

The 'assess-migration' command reports unsupported BRIN and SPGIST index methods from PostgreSQL sources, providing insights into potential compatibility issues.

```cli
yb-voyager assess-migration --source-db ... --target-db ...
```

--------------------------------

### YB Voyager Uncategorized SQL Files

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Object types that YB Voyager cannot categorize during schema analysis are now placed in a separate `uncategorized.sql` file. Information about this file is provided as a note in the `analyze-schema` report.

```APIDOC
analyze-schema <schema_file>

Output:
  - Uncategorized Objects: Placed in a file named `uncategorized.sql`.
  - Report Notes: The report generated by `analyze-schema` includes a note referencing the `uncategorized.sql` file.

Purpose:
  Provides clarity on objects that YB Voyager could not automatically classify, aiding manual review.
```

--------------------------------

### YB Voyager Resumption Script Usage

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/resumption/README.md

Shows the command-line syntax for executing the YB Voyager resumption test script. It takes the test name, an optional environment script, and a configuration file as arguments.

```bash
TEST_NAME [env.sh] [config_file]
```

--------------------------------

### Schema Analysis Report Structure

Source: https://github.com/yugabyte/yb-voyager/blob/main/yb-voyager/cmd/templates/schema_analysis_report.html

This snippet represents the overall structure of the Schema Analysis Report, using Go template syntax to dynamically populate migration details, schema summaries, and issue tracking. It includes placeholders for database versions, object counts, and specific migration issues.

```go-template
Schema Analysis Report
======================

### Migration Information

{{if .SchemaSummary.SchemaNames }} {{end}} 

Voyager Version

{{ .VoyagerVersion }}

Target DB Version

{{ .TargetDBVersion }}

Database Name

{{ .SchemaSummary.DBName }}

Schema Name

{{ join .SchemaSummary.SchemaNames ", " }}

DB Version

{{ .SchemaSummary.DBVersion }}

### Schema Summary

{{ range .SchemaSummary.DBObjects }} {{ if .TotalCount }} {{ end }} {{ end }} 

Object

Total Objects

Objects Without Issues

Objects With Issues

Object Names

Details

{{ .ObjectType }}

{{ .TotalCount }}

{{ sub .TotalCount .InvalidCount }}

{{ .InvalidCount }}

{{range split .ObjectNames ","}} {{.}}  
{{end}}

{{ .Details }}

{{ sumDbObjects .SchemaSummary.DBObjects "TotalCount" }}

{{ sumDbObjects .SchemaSummary.DBObjects "ValidCount" }}

{{ sumDbObjects .SchemaSummary.DBObjects "InvalidCount" }}

### Issues

{{ range $index, $issue := .Issues }}1.  #### Issue in Object {{ $issue.ObjectType }}
    
    *   **Issue Type:** {{ $issue.IssueType }}
    *   **Object Name:** {{ $issue.ObjectName }}
    *   **Reason:** {{ $issue.Reason }}
    *   **SQL Statement:**
        
        {{ $issue.SqlStatement }}
        
    *   **File Path:** {{ $issue.FilePath }} [["Preview"]]({{ $issue.FilePath }})
    {{ $supporterVerStr := getSupportedVersionString $issue.MinimumVersionsFixedIn }} {{ if $supporterVerStr }}*   **Fixed in Versions:** {{ $supporterVerStr }}
    {{ end }} {{ if $issue.Suggestion }}*   **Suggestion:** {{ $issue.Suggestion }}
    {{ end }} {{ if $issue.GH }}*   **Github Issue Link:** [{{ $issue.GH }}]({{ $issue.GH }})
    {{ end }} {{ if $issue.DocsLink }}*   **Docs Link:** [{{ $issue.DocsLink }}]({{ $issue.DocsLink }})
    {{ end }}
{{ end }}

{{ if .SchemaSummary.Notes }}

### Notes

{{ range .SchemaSummary.Notes }}*   {{ . }}
{{ end }}

{{ end }}
```

--------------------------------

### Assess Migration Unsupported Constructs

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

The 'assess-migration' command identifies and reports SQL queries with unsupported features like advisory locks, system columns, and XML functions. This aids in early detection of potential migration issues.

```cli
yb-voyager assess-migration --source-db ... --target-db ...
```

--------------------------------

### Live Migration Features

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Details on the live migration capabilities of yb-voyager, including fall-forward options and concurrent index/trigger import.

```APIDOC
Live Migration
  - Description: Supports migrating data from PostgreSQL to YugabyteDB in real-time.
  - Features:
    - Fall-forward option: Allows switching to a source-replica PostgreSQL database during migration.
    - Concurrent Index/Trigger Import: Indexes and triggers can be created during the CDC phase, before cutover, reducing downtime.
  - Bug Fixes:
    - Resolved transaction conflict errors or bad data in the CDC phase of live migration (including fall-back/fall-forward).
    - Fixed silent conversion of '\r\n' to '\n' in text data, affecting snapshot and offline migration with BETA_FAST_DATA_EXPORT.
```

--------------------------------

### YB-Voyager Analyze Schema Report Enhancements

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Describes additional details provided by `yb-voyager analyze-schema` regarding unsupported YugabyteDB index types.

```APIDOC
YB-Voyager Analyze Schema Report Enhancements:

- `yb-voyager analyze-schema` now includes details in the report about indices that are not exported, such as reverse indexes, which are unsupported in YugabyteDB.
```

--------------------------------

### YB-Voyager Known Issues: Dockerized Version

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Lists known issues specific to the dockerized version of YB-Voyager, including limitations with Oracle SSL exports and the `end migration` command.

```APIDOC
YB-Voyager Known Issues (Dockerized Version):

- Export schema and export data from Oracle database with SSL (via --oracle-tns-alias) fails. Workaround: Use a non-docker version of yb-voyager.
- The `end migration` command fails. This issue will be addressed in an upcoming release.
```

--------------------------------

### Migration Assessment Report Summary

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

This section provides a summary of a migration assessment report from PostgreSQL to YugabyteDB. It details the Voyager version, source database information, target YB version, migration complexity, and a breakdown of database objects like schemas, extensions, sequences, tables, indexes, functions, and policies.

```text
Migration Assessment Report
===========================

**Voyager Version:** main

**Database Name:** pg_basic_assessment_report

**Database Version:** 17.2 (Debian 17.2-1.pgdg120+1)

**Target YB Version:** 2024.2.3.1

**Migration Complexity:** MEDIUM

Database Objects
----------------

Objects that will be created on the target YugabyteDB.

Object Type

Total Objects

Object Names

SCHEMA

2

*   public
*   schema1

EXTENSION

1

*   pg_stat_statements

SEQUENCE

2

*   public.parent_table_id_seq
*   schema1.employees2_id_seq

TABLE

5

*   public.parent_table
*   public.child_table
*   public.mixed_data_types_table1
*   schema1.employees2
*   schema1.test_hotspot_timestamp

INDEX

5

*   idx_employees2_first_name ON schema1.employees2
*   idx_employees2_full_name ON schema1.employees2
*   idx_employees2_last_name ON schema1.employees2
*   idx_test_hotspot_timestamp ON schema1.test_hotspot_timestamp
*   idx_test_low_card_employed ON schema1.test_hotspot_timestamp

FUNCTION

1

*   public.manage_large_object

POLICY

1

*   policy_test_report ON public.parent_table
```

--------------------------------

### Dockerized yb-voyager

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Information regarding issues with the dockerized version of yb-voyager.

```APIDOC
Dockerized yb-voyager
  - Bug Fixes: Fixed an issue where 'end migration' fails when using the dockerized installation.
```

--------------------------------

### Improve Permission Grant Script

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

The `yb-voyager-pg-grant-migration-permissions.sql` script has been improved to internally detect table owners, removing the need for the `original_owner_of_tables` flag.

```APIDOC
yb-voyager-pg-grant-migration-permissions.sql

Enhancement:
  Internally detects table owners, eliminating the need for the `original_owner_of_tables` flag.
```

--------------------------------

### yb-voyager analyze-schema Command

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

The `analyze-schema` command has been updated to report unsupported data types encountered during schema analysis. Deprecated `--file-opts` argument is replaced by `--escape-char` and `--quote-char`.

```bash
yb-voyager analyze-schema

# Replaced by:
yb-voyager import data file --escape-char=\" --quote-char=\"
```

--------------------------------

### Database Migration Report Template

Source: https://github.com/yugabyte/yb-voyager/blob/main/yb-voyager/cmd/templates/schema_analysis_report.txt

This template is used by Yugabyte Voyager to generate a detailed report of a database migration process. It displays migration version, database details, schema summaries, identified issues with SQL statements and file paths, and any notes or suggestions. The template utilizes Go's text/template syntax for dynamic content rendering.

```text/template
+-------------------------------------------------+
|               Database Migration Report         |
+-------------------------------------------------+

Migration Information
---------------------

Voyager Version               : {{ .VoyagerVersion }}
Database Name                 : {{ .SchemaSummary.DBName }}
Schema Name(s)                : {{ join .SchemaSummary.SchemaNames ", " }}
DB Version                    : {{ .SchemaSummary.DBVersion }}
Target DB Version             : {{ .TargetDBVersion }}


Schema Summary
---------------
{{ range .SchemaSummary.DBObjects }}
Object Type                    : {{ .ObjectType }}
  - Total Objects              : {{ .TotalCount }}
  - Objects Without Issues     : {{ sub .TotalCount .InvalidCount }}
  - Objects With Issues        : {{ .InvalidCount }}
  - Object Names               : {{ .ObjectNames }}{{ if .Details }}  
  - Details                    : {{ .Details }}
{{ end }}
{{ end }}

Issues
-------
{{ if .Issues }} {{ range $index, $issue := .Issues }}
{{ add $index 1 }}. Issue in Object     : {{ .ObjectType }}
  - Object Name     : {{ .ObjectName }}
  - Reason          : {{ .Reason }}
  - SQL Statement   : {{ .SqlStatement }}
  - File Path       : {{ .FilePath }}{{ if .Suggestion }}
  - Suggestion      : {{ .Suggestion }} {{ end }}{{ if .GH }}
  - Github Issue    : {{ .GH }}{{ end }}{{ if .DocsLink }}
  - Docs Link       : {{ .DocsLink }}{{ end }}
{{ $supporterVerStr := getSupportedVersionString .MinimumVersionsFixedIn }}
{{ if $supporterVerStr }}
  - Fixed in Versions: {{ $supporterVerStr }}
{{ end }}
{{ end }}{{ else }}
No issues found.{{ end }}

{{ if .SchemaSummary.Notes }}
Notes
------
{{ range $index, $note := .SchemaSummary.Notes }}
{{ add $index 1 }}. {{ $note }}
{{ end }}
{{ end }}

```

--------------------------------

### Import Data with Table Truncation Option

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

The `import data file` and `import data to source replica` commands now support a `truncate-tables` flag. When used with `start-clean true`, this option truncates all tables in the target or source-replica database before importing data.

```APIDOC
import data file [options] --truncate-tables
import data to source replica [options] --truncate-tables

Options:
  start-clean true: Enables truncation of tables before import.
  --truncate-tables: Truncates all tables in the target/source-replica database before data import.
```

--------------------------------

### YB Voyager Resumption File Import Configuration

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/resumption/README.md

A sample YAML configuration for YB Voyager's file import resumption testing. It specifies file mapping, additional import flags, expected row counts, and detailed resumption parameters.

```yaml
file_table_map: "table1_data.sql:table1"

additional_flags:
  --delimiter: "\t"
  --format: "text"

row_count:
  table1: 35000000

resumption:
  max_restarts: 20
  min_interrupt_seconds: 18
  max_interrupt_seconds: 48
  min_restart_wait_seconds: 3
  max_restart_wait_seconds: 6

varying_configs:
  --parallel-jobs:
    type: range
    value: [1, 10]
```

--------------------------------

### YB Voyager Oracle Migration Enhancements

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Several improvements for Oracle migrations, including the default availability of the Orafce extension, relaxed user creation permissions, and accurate migration of numeric data type precision and scale.

```APIDOC
Oracle Migration Features:

1. Orafce Extension:
   - Availability: Now created by default on the target database.
   - Purpose: Enables Oracle compatibility functions in PostgreSQL.

2. User Creation Permissions:
   - Requirement Change: No longer requires EXECUTE permissions on PROCEDURE, FUNCTION, PACKAGE, and PACKAGE BODY objects.
   - Benefit: Simplifies user setup and reduces permission management overhead.

3. Numeric Data Type Migration:
   - Precision and Scale: Accurately migrated from Oracle to the target database.
   - Benefit: Ensures data integrity for numeric values.
```

--------------------------------

### YugabyteDB Migration and Configuration Notes

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/oracle/assessment-report-test/expectedAssessmentReport.html

Key considerations for migrating data and configuring YugabyteDB, including unsupported datatypes, colocation strategies, indexing, and feature previews.

```APIDOC
YugabyteDB Migration and Configuration:

Unsupported Datatypes:
  - CLOB: Source CLOB datatypes are not supported on target YugabyteDB.
  - NCLOB: Source NCLOB datatypes are not supported on target YugabyteDB.
  - TEXT: Source TEXT datatypes are not supported on target YugabyteDB.

Feature Support and Preview Flags:
  - Some features may be supported under a preview flag in specific YugabyteDB versions. Consult official release notes for details.

Colocation and Hotspots:
  - Avoid colocating tables that experience disproportionately high load to prevent tablet hotspots.
  - Refer to YugabyteDB documentation for detailed considerations on colocated tables.

Indexing Strategies:
  - Range Sharding for Range Queries: For columns frequently used in range-based queries (e.g., timestamps), explicitly configure indexes with range sharding.
  - Default Sharding: YugabyteDB defaults to hash sharding for indexes, which is not optimal for range predicates.
  - PostgreSQL Compatibility Mode: Range sharding is enabled by default only in PostgreSQL compatibility mode.

Partitioned Tables:
  - Default Creation: All partitions of a partitioned table are created as colocated by default during the export schema phase.
  - Manual Schema Modification: Refer to specific GitHub issues (e.g., #1581) for manual schema modification guidance.
  - Sizing Recommendations: Reference and System Partitioned tables are treated as normal tables and excluded from target cluster sizing recommendations.

Index Type Conversion:
  - BITMAP to GIN: BITMAP indexes are converted to GIN indexes.
  - GIN Index Support: GIN indexes have partial support in YugabyteDB. Review related issues (e.g., #7850) for compatibility and potential modifications.
```

--------------------------------

### Unsupported PostgreSQL Features and Constructs

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Reporting of unsupported features in yb-voyager's `analyze-schema` and `assess-migration` commands. This includes unsupported indexes on specific data types, primary/unique key constraints, and the REFERENCING clause for triggers.

```APIDOC
Unsupported Features:
  - Indexes on data types: daterange, int4range, int8range, tsrange, tstzrange, numrange, interval
  - Primary and Unique Key Constraints on various data types
  - REFERENCING clause for triggers
  - BEFORE ROW triggers on partitioned tables
  - Deferrable and Exclusion Constraints

Unsupported Query Constructs:
  - XMLTABLE() function

Related Commands:
  - analyze-schema
  - assess-migration
```

--------------------------------

### Process List Items to Highlight Links

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

This JavaScript code iterates through list items, identifies URLs within the text, and converts them into clickable anchor tags (`<a>`) with a specific class ('highlight-link') for styling and external linking.

```javascript
const notesListItems = document.querySelectorAll("#notes-list li");
notesListItems.forEach((li) => {
    const originalText = li.textContent.trim();
    const words = originalText.split(' ');
    const hasUrl = words.some(word => word.startsWith('http://') || word.startsWith('https://'));
    if (!hasUrl) return;
    li.textContent = '';
    words.forEach((word, index) => {
        let url = word;
        let charThatWillBeRemoved = '';
        const isUrl = url.startsWith('http://') || url.startsWith('https://');
        if (isUrl) {
            const lastChar = url[url.length - 1];
            const urlCantEndWithTheseChars = [',', '.', '!', '?', ';', ':'];
            if (urlCantEndWithTheseChars.includes(lastChar)) {
                charThatWillBeRemoved = lastChar;
                url = url.slice(0, -1);
            }
            const aTag = document.createElement("a");
            aTag.href = url;
            aTag.textContent = url;
            aTag.classList.add("highlight-link");
            aTag.target = "_blank";
            li.appendChild(aTag);
            if (charThatWillBeRemoved) {
                li.appendChild(document.createTextNode(charThatWillBeRemoved));
            }
        } else {
            li.appendChild(document.createTextNode(word));
        }
        if (index < words.length - 1) {
            li.appendChild(document.createTextNode(' '));
        }
    });
});
```

--------------------------------

### Analyze Schema Unsupported Datatypes

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

The 'analyze-schema' command reports unsupported datatypes for live migration, including user-defined types and array of enums. It also includes PostGIS datatypes like BOX2D, BOX3D, and TOPOGEOMETRY.

```cli
yb-voyager analyze-schema --source-db ... --target-db ...
```

--------------------------------

### YB Voyager Bug Fixes

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Details on resolved issues, including function parsing errors and table migration problems with reserved keywords.

```APIDOC
Bug Fixes:

1. Function Parsing:
   - Issue: Fixed function parsing when the entire function body was on a single line.
   - Reference: [[765]](https://github.com/yugabyte/yb-voyager/issues/765)

2. Reserved Keywords as Table Names:
   - Issue: Fixed the problem of migrating tables whose names were reserved keywords in the target database.
   - Reference: [[757]](https://github.com/yugabyte/yb-voyager/issues/757)
```

--------------------------------

### Schema Analysis and Data Export

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Information on schema analysis enhancements and data export functionalities within yb-voyager.

```APIDOC
analyze schema
  - Description: Analyzes the source schema for compatibility with YugabyteDB.
  - Enhancements: Reports unsupported extensions on YugabyteDB.

export data status
  - Description: Reports the status of data export operations.
  - Enhancements: Reports the number of rows exported for each table in offline migration scenarios.
```

--------------------------------

### Import Data Adaptive Parallelism

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Dynamically adjusts the number of parallel jobs for the 'import data' command based on YugabyteDB cluster CPU and memory usage. This optimizes import speeds and enhances efficiency without manual intervention.

```cli
yb-voyager import data --target-db ... --source-db ...
```

--------------------------------

### Handle NOT VALID Constraints During Import

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Fixes an issue where NOT VALID constraints in the schema could cause constraint violation errors during data imports. These constraints are now correctly created during the post-snapshot import phase.

```APIDOC
import data

Fix:
  NOT VALID constraints are now handled to prevent import errors, created during the post-snapshot import phase.
```

--------------------------------

### yb-voyager Password Handling via Environment Variables

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

yb-voyager commands requiring passwords now support using environment variables SOURCE_DB_PASSWORD and TARGET_DB_PASSWORD. This enhances security by preventing passwords from appearing in process lists or log files.

```bash
export SOURCE_DB_PASSWORD="your_source_password"
export TARGET_DB_PASSWORD="your_target_password"

yb-voyager import data file --source-db-password $SOURCE_DB_PASSWORD --target-db-password $TARGET_DB_PASSWORD
```

--------------------------------

### YB Voyager PostgreSQL pg_dump Password Handling

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

For PostgreSQL migrations, YB Voyager now securely handles the password for the background `pg_dump` command by setting it as an environment variable internally, rather than including it directly in the command line arguments.

```APIDOC
PostgreSQL Migration Security:

pg_dump Password Handling:
  - Method: Password is now set as an internal environment variable for `pg_dump`.
  - Security Improvement: Avoids exposing the password in command-line history or process lists.

Notes:
  This change enhances the security posture during PostgreSQL data dumps.
```

--------------------------------

### yb-voyager Import Data File Enhancements

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

The `import data file` command supports importing multiple files into the same table using glob expressions in `--file-table-map`. It also allows direct importing from GCS (gs://) and Azure Blob Storage (https://) in addition to AWS S3.

```bash
# Import multiple files using glob expressions
yb-voyager import data file --file-table-map "table_name=path/to/files/*.csv"

# Import from GCS
yb-voyager import data file --file-table-map "table_name=gs://your-bucket/data/*.csv"

# Import from Azure Blob Storage
yb-voyager import data file --file-table-map "table_name=https://your-storage-account.blob.core.windows.net/your-container/data/*.csv"
```

--------------------------------

### Prevent Panics with Empty Export Directory

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Prevents Voyager from panicking or erroring out during `analyze-schema` and `import data` operations when the `export-dir` is empty.

```APIDOC
analyze-schema
import data

Bug Fix:
  Handles cases where `export-dir` is empty to prevent application crashes.
```

--------------------------------

### YB-Voyager CLI Argument Standardization

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Details changes in how boolean arguments are handled in YB-Voyager commands for improved CLI consistency. Boolean values can now be specified using various formats.

```APIDOC
YB-Voyager CLI Argument Standardization:

- Boolean arguments are now standardized as string arguments.
- No need to use `=` for setting boolean flags to false; whitespace is sufficient.
- Boolean flag names cannot be used without a value (e.g., use `--send-diagnostics true` instead of `--send-diagnostics`).
- Supported boolean values: `true/false`, `yes/no`, `1/0`.
```

--------------------------------

### yb-voyager Guardrail Arguments for Table Filtering

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Arguments `--table-list` and `--exclude-table-list` are used in guardrails to specify which PostgreSQL tables require permission checks during export.

```Shell
yb-voyager assess-migration --table-list "table1,table2"
```

```Shell
yb-voyager assess-migration --exclude-table-list "table3,table4"
```

--------------------------------

### yb-voyager Accelerated Data Export SSL Support

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

When using accelerated data export for MySQL and Oracle, yb-voyager can now connect to source databases using SSL for secure communication.

```bash
# Example command structure (SSL details would be part of connection string or config)
yb-voyager export data --accelerated --source-db-ssl=true
```

--------------------------------

### yb-voyager Accelerated Data Export Performance

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Accelerated data export for MySQL and Oracle is now 2-4x faster when the BETA_FAST_DATA_EXPORT=1 environment variable is set. This mode supports features like migrating partitioned tables and sequences.

```bash
export BETA_FAST_DATA_EXPORT=1
yb-voyager export data --source-db-type=mysql --source-db-host=localhost
```

--------------------------------

### YB Voyager import data file CSV Options

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Enhancements to the `import data file` command for CSV files, allowing flexible specification of escape and quote characters using the `--file-opts` flag. This provides greater compatibility with diverse CSV formats.

```APIDOC
import data file <file_path>

--file-opts <options>
  Specifies options for file processing. For CSV files, supports:
  - quote_char: Any character can be used as the quote character (e.g., ' or ").
  - escape_char: Any character can be used as the escape character (e.g., \ or ').

Example:
  import data file data.csv --file-opts "quote_char=' escape_char=\""

Notes:
  Previously, only double quotes (") were supported for both quote and escape characters.
```

--------------------------------

### Enable YB-Voyager Guardrails Checks

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Enables comprehensive checks for YugabyteDB Voyager commands, including source/target database permissions, binary dependencies, and PostgreSQL version compatibility. This feature is disabled by default and can be activated using a specific flag.

```cli
--run-guardrails-checks
```

--------------------------------

### YB-Voyager Export/Import Table List Enhancements

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Enhancements to the `--table-list` and `exclude-table-list` arguments for YB-Voyager export/import data commands, including file path support and wildcard characters.

```APIDOC
YB-Voyager Export/Import Table List Enhancements:

- `--table-list` and `exclude-table-list` arguments can now be provided via file paths using `--table-list-file-path` or `exclude-table-list-file-path`.
- Table list arguments now support glob wildcard characters: `?` (matches one character) and `*` (matches zero or more characters).
- `table-list` and `exclude-table-list` can be used together in a command.
- Table names in `yb-voyager import-data` are now case-insensitive by default. Enclose names in double quotes for case-sensitive matching.
```

--------------------------------

### Enhanced Guardrail Checks for YugabyteDB Aeon

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Improves guardrail checks within the `import-schema` command specifically for YugabyteDB Aeon, ensuring better validation during schema import.

```APIDOC
import-schema --target-db-version aeon

Enhancement:
  Strengthened guardrail checks for YugabyteDB Aeon compatibility during schema import.
```

--------------------------------

### Text, Alignment, and Code Block Styles (CSS)

Source: https://github.com/yugabyte/yb-voyager/blob/main/migtests/tests/pg/basic-assessment-report-test/expectedAssessmentReport.html

Provides styles for text alignment, specific element formatting like paragraph references, and general rules for pre/code blocks to ensure consistent display of code snippets within the report.

```css
.centered-cell {
  text-align: center;
  vertical-align: middle;
}

#refer-p {
  font-size: 0.8em;
  color: inherit;
}

#refer-p a {
  color: inherit;
}

.list_item {
  margin-bottom: 15px;
}

pre, code {
  width: 100%;
  margin: 0;
  padding: 8px;
  background: none;
  border: none;
  white-space: pre-wrap;
  word-break: break-word;
  overflow-wrap: break-word;
  box-sizing: border-box;
}
```

--------------------------------

### Upgrade Compatibility

Source: https://github.com/yugabyte/yb-voyager/blob/main/RELEASE_NOTES.md

Allows yb-voyager upgrades during migration from recent breaking releases (like v1.8.5) to later versions, ensuring a smoother upgrade path.

```APIDOC
yb-voyager upgrade

Enhancement:
  Supports upgrading yb-voyager from older versions to newer ones, including transitions from breaking releases.
```
Tokens:
Raw