### FastTransfer Logging Configuration Example Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/index.md This JSON snippet shows a sample configuration for FastTransfer's logging settings. It specifies connection strings for logging databases, enabled logging sinks, and the minimum logging level. ```json { "ConnectionStrings": { "MS_FastTransferLogs": "Server=localhost;Database=FastTransferLogs;Integrated Security=SSPI;Encrypt=True;TrustServerCertificate=True" }, "Serilog": { "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Sinks.MSSqlServer", "Serilog.Enrichers.Environment", "Serilog.Enrichers.Thread", "Serilog.Enrichers.Process", "Serilog.Enrichers.Context" ], "MinimumLevel": "Debug", ``` -------------------------------- ### MSSQL to PostgreSQL Data Transfer Example Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/index.md This example demonstrates how to transfer data from a SQL Server database to a PostgreSQL database using FastTransfer. It specifies source and target connection details, load mode, and degree of parallelism. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "mssql" ` --sourceserver "localhost" ` --sourceuser "fastuser" ` --sourcepassword "fastpassword" ` --sourcedatabase "AdventureWorks2017" ` --sourceschema "Person" ` --sourcetable "Person" ` --targetconnectiontype "pgcopy" ` --targetserver "localhost" ` --targetuser "fastuser" ` --targetpassword "fastpassword" ` --targetdatabase "fastdb" ` --targetschema "public" ` --targettable "Person" ` --method "RangeId" ` --distributeKeyColumn "PersonID" ` --loadmode "Truncate" ` --degree -2 ` #Automatically adapt the degree of parallelism to 1/2 of cpu available --runid "mssql-to-pgcopy-123456" ``` -------------------------------- ### PostgreSQL to SQL Server Transfer: Ctid and DataDriven Methods Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This section shows PostgreSQL to SQL Server transfers. The first example uses the Ctid method for PostgreSQL sources. The second example demonstrates using a query with DataDriven partitioning, specifying a distribution key column and degree of parallelism. ```bash # PostgreSQL to SQL Server with Ctid method (recommended for PostgreSQL sources) ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --sourcedatabase "tpch" \ --sourceschema "tpch_10" \ --sourcetable "lineitem" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targettrusted \ --targetdatabase "tpch_test" \ --targetschema "dbo" \ --targettable "lineitem" \ --method "Ctid" \ --degree 10 \ --loadmode "Truncate" ``` ```bash # Using query instead of table with DataDriven partitioning ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --sourcedatabase "tpch" \ --query "select * from tpch_10.orders where o_orderdate >= '1995-01-01'" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targettrusted \ --targetdatabase "tpch_test" \ --targetschema "dbo" \ --targettable "orders" \ --method "DataDriven" \ --distributeKeyColumn "FLOOR(o_orderkey/10000000)" \ --degree 6 \ --loadmode "Append" ``` -------------------------------- ### PostgreSQL to SQL Server Data Transfer Example Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/index.md This example shows how to transfer data from a PostgreSQL database to a SQL Server database using FastTransfer. It includes options for specifying connection details, load mode, run ID, and column mapping method. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:15432" ` --sourceuser "fastuser" ` --sourcepassword "fastpassword" ` --sourcedatabase "fastdb" ` --sourceschema "Public" ` --sourcetable "Person" ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetuser "fastuser" ` --targetpassword "fastpassword" ` --targetdatabase "AdventureWorks2017" ` --targetschema "Person" ` --targettable "Person" ` --method "Ctid" ` # Ctid is for pgsql and pgcopy source only --loadmode "Truncate" ` --degree -2 ` #Automatically adapt the degree of parallelism to 1/2 of cpu available --runid "pgsql-to-msbulk-123456" ` --mapmethod "Name" ``` -------------------------------- ### SQL Server to PostgreSQL Transfer: Random and DataDriven Methods Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This section illustrates SQL Server to PostgreSQL transfers. The first example uses the Random method with a specified distribution key column. The second example shows the DataDriven method for a partitioned SQL Server table, utilizing a partition-specific distribution key. ```bash # SQL Server to PostgreSQL with Random method ./FastTransfer \ --sourceconnectiontype "mssql" \ --sourceserver "localhost" \ --sourcedatabase "tpch_test" \ --sourcetrusted \ --targetconnectiontype "pgcopy" \ --targetserver "localhost:5432" \ --targetdatabase "tpch_import" \ --targetuser "fastuser" \ --targetpassword "fastpassword" \ --query "select * from dbo.orders" \ --targetschema "tpch_10" \ --targettable "orders" \ --method "Random" \ --distributeKeyColumn "o_orderkey" \ --degree -3 \ --loadmode "Truncate" ``` ```bash # SQL Server partitioned table with DataDriven method ./FastTransfer \ --sourceconnectiontype "mssql" \ --sourceserver "localhost" \ --sourcedatabase "tpch_test" \ --sourcetrusted \ --targetconnectiontype "pgcopy" \ --targetserver "localhost:5432" \ --targetuser "fastuser" \ --targetpassword "fastpassword" \ --targetdatabase "tpch_import" \ --query "select * from dbo.orders_part" \ --targetschema "tpch_10" \ --targettable "orders" \ --method "DataDriven" \ --distributeKeyColumn "$PARTITION.PF_DATE(o_orderdate)" \ --degree 10 \ --loadmode "Truncate" ``` -------------------------------- ### FastTransfer: Configure license file or URL Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This example shows how to specify a license for FastTransfer using either a local file path or a URL. The `--license` argument accepts a string representing the path to the license file or the HTTP(S) URL where the license can be retrieved. This is essential for enabling the tool's functionality. ```bash # Use license file from custom path ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --sourcedatabase "tpch" \ --sourcetable "orders" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targettrusted \ --targetdatabase "tpch_test" \ --targettable "orders" \ --method "None" \ --license "/licenses/FastTransfer.lic" ``` ```bash # Use license from HTTP URL ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --sourcedatabase "tpch" \ --sourcetable "orders" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targettrusted \ --targetdatabase "tpch_test" \ --targettable "orders" \ --method "None" \ --license "https://license-server.company.com/FastTransfer.lic" ``` -------------------------------- ### PostgreSQL to PostgreSQL: pgsql Driver with INSERT UNNEST Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This example shows a PostgreSQL to PostgreSQL transfer using the pgsql driver and the INSERT UNNEST method. It includes source and target connection information, schema and table details, and specific parameters for batching. ```bash ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourcedatabase "tpch" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --targetconnectiontype "pgsql" \ --targetserver "localhost:5432" \ --targetdatabase "tpch_import" \ --sourceschema "tpch_10" \ --sourcetable "orders" \ --targetuser "fastuser" \ --targetpassword "fastpassword" \ --targetschema "tpch_10" \ --targettable "orders" \ --method "Ctid" \ --degree 10 \ --loadmode "Truncate" \ --batchsize 10000 ``` -------------------------------- ### MSSQL Partitioning Source to MSSQL Target (DataDriven Method) Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example shows how to transfer data from a source MSSQL database to a target MSSQL database using the 'DataDriven' method. This method is faster for targets with a clustered columnstore index. It includes partitioning details and load mode. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "mssql" ` --sourceserver "localhost" ` --sourcedatabase "tpch10_collation_bin2" ` --sourcetrusted ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "tpch10_collation_bin2_copy" ` --query "select * from orders_part" ` --targettrusted ` --targetschema "dbo" ` --targettable "orders" ` --method "DataDriven" ` --distributeKeyColumn "`$PARTITION.PF_DATE(o_orderdate)" ` --degree 10 ` --loadmode "Truncate" ``` -------------------------------- ### Netezza to MSSQL Transfer using DataDriven Method Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example demonstrates transferring data from a Netezza source to an MSSQL target using the 'DataDriven' method, with distribution based on DATASLICEID. It includes source and target connection details, schema, table, and degree of parallelism. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "nzsql" ` --sourceserver "NZSOURCESYS" ` --sourcedatabase "TPCH" ` --sourceuser "TPCH" ` --sourcepassword "TPCH" ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "tpch_test" ` --sourceschema "tpch_10" ` --sourcetable "orders" ` --targettrusted ` --targetschema "dbo" ` --targettable "orders" ` --method "DataDriven" ` --distributeKeyColumn "DATASLICEID" ` --degree 6 ``` -------------------------------- ### SQL Server to MySQL: Random Method Transfer Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This example demonstrates transferring data from SQL Server to MySQL using the Random method. It includes connection details for both source and target, a query for data selection, and parameters for target schema, table, and distribution key. ```bash ./FastTransfer \ --sourceconnectiontype "mssql" \ --sourceserver "localhost" \ --sourcedatabase "tpch_test" \ --sourcetrusted \ --targetconnectiontype "mysqlbulk" \ --targetserver "mysql-server:3306" \ --targetdatabase "tpch" \ --query "select * from dbo.orders" \ --targetuser "fastuser" \ --targetpassword "fastpassword" \ --targetschema "tpch" \ --targettable "orders" \ --method "Random" \ --distributeKeyColumn "o_orderkey" \ --degree 6 \ --loadmode "Truncate" ``` -------------------------------- ### Linux: Verify FastTransfer binary integrity and authenticity Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This guide details the steps to verify the integrity and authenticity of the FastTransfer binary on a Linux system. It includes commands for checking the SHA256 checksum, downloading and verifying the GPG signing key, and verifying the signature of the checksum file. Finally, it shows how to make the binary executable and run it. ```bash # Verify SHA256 checksum cd /path/to/FastTransfer-linux-x64 sha256sum -c FastTransfer.sha256sum.txt # Expected output: FastTransfer: OK # Download GPG signing key gpg --keyserver keys.openpgp.org --recv-keys 37007D091BF7DCC74D208A1F869511C7130465C8 # Verify key fingerprint gpg --list-key --with-fingerprint 130465C8 # Expected output should contain: 3700 7D09 1BF7 DCC7 4D20 8A1F 8695 11C7 1304 65C8 # Verify signature authenticity gpg --verify FastTransfer.sha256sum.txt.asc FastTransfer.sha256sum.txt # Expected output: gpg: Good signature from "Francois Pacull " # Make executable (if needed) chmod +x FastTransfer # Run FastTransfer ./FastTransfer --help ``` -------------------------------- ### FastTransfer: Use SQL file for query Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This example demonstrates how to configure FastTransfer to execute a data transfer operation using an external SQL file for the source query. This approach is useful for complex or reusable queries. It specifies source and target database details, along with the path to the SQL file. ```bash # Use SQL file instead of inline query ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --sourcedatabase "tpch" \ --fileinput "/queries/select_orders.sql" \ --sourceschema "tpch_10" \ --sourcetable "orders" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targetdatabase "tpch_test" \ --targettrusted \ --targetschema "dbo" \ --targettable "orders_filtered" \ --method "Ctid" \ --degree 6 \ --loadmode "Truncate" ``` -------------------------------- ### MSSQL to MSSQL Transfer without Specific Method Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example demonstrates transferring data from a source MSSQL database to a target MSSQL database using the 'None' method. It specifies connection details, the source query, and target table information. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "mssql" ` --sourceserver "localhost" ` --sourcedatabase "FASTExportData" ` --sourcetrusted ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "FASTExportData_copy" ` --query "select * from TEST_71_1M_12m" ` --targettrusted ` --targetschema "dbo" ` --targettable "TEST_71_1M_12m" ` --method "None" ``` -------------------------------- ### Transfer PostgreSQL to HANA Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md Migrates data from PostgreSQL to HANA using the 'hanabulk' method. This example specifies connection details for both databases, a query to select data, and table mappings. It also includes batch size and schema information. ```powershell . FastTransfer.exe \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:15432" \ --sourcedatabase "tpch" \ --sourceuser "FastUser" \ --sourcepassword "FastPassword" \ --sourceschema "tpch_10" \ --sourcetable "orders" \ --targetconnectiontype "hanabulk" \ --targetserver "hxehost:39015" \ --targetdatabase "" \ --targetuser "SYSTEM" \ --targetpassword "AetP202440" \ --targetschema "TPCH" \ --targettable "ORDERS" \ --method "None" \ --loadmode "Truncate" \ --batchsize 5000 \ --query "select * from tpch_10.orders limit 100000" ``` -------------------------------- ### FastTransfer: Configure custom logging settings Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This section shows how to configure FastTransfer's logging behavior using a custom JSON settings file. The example JSON defines connection strings for logs and Serilog settings, including console, file, and MSSqlServer sinks with specific configurations for each. The bash command then applies this custom settings file to the FastTransfer execution. ```json { "ConnectionStrings": { "MS_FastTransferLogs": "Server=log-server;Database=FastTransferLogs;Integrated Security=SSPI;Encrypt=True;TrustServerCertificate=True" }, "Serilog": { "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Sinks.MSSqlServer" ], "MinimumLevel": "Debug", "WriteTo": [ { "Name": "Console", "Args": { "outputTemplate": "{Timestamp:yyyy-MM-ddTHH:mm:ss.fff zzz} -|- {Application} -|- {runid} -|- {Level:u12} -|- {fulltargetname} -|- {Message}{NewLine}{Exception}" } }, { "Name": "File", "Args": { "fileSizeLimitBytes": 4194304, "path": "Logs/FastTransfer_.json", "rollingInterval": "Day", "rollOnFileSizeLimit": true } }, { "Name": "MSSqlServer", "Args": { "connectionString": "MS_FastTransferLogs", "sinkOptionsSection": { "TableName": "EventLogs", "SchemaName": "dbo", "AutoCreateSqlTable": true, "BatchPostingLimit": 50 } } } ] } } ``` ```bash # Use custom settings file ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --sourcedatabase "tpch" \ --sourceschema "tpch_10" \ --sourcetable "orders" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targettrusted \ --targetdatabase "tpch_test" \ --targetschema "dbo" \ --targettable "orders" \ --method "Ctid" \ --degree 10 \ --settingsfile "/config/custom_settings.json" \ --runid "custom-logging-transfer" ``` -------------------------------- ### PostgreSQL to PostgreSQL Transfer using Insert Unnest with PowerShell Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example demonstrates transferring data from PostgreSQL to PostgreSQL using the 'Insert Unnest' method. It configures source and target connections, specifies `--degree` and `--batchsize`, and uses `Truncate` for the load mode. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:5432" ` --sourcedatabase "tpch" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --targetconnectiontype "pgsql" ` --targetserver "localhost" ` --targetdatabase "tpch_import" ` --sourceschema "tpch_10" ` --sourcetable "orders" ` --targetuser "FastUser" ` --targetpassword "FastPassword" ` --targetschema "tpch_10" ` --targettable "orders" ` --method "Ctid" ` --degree 10 ` --loadmode "Truncate" ` --batchsize 10000 ``` -------------------------------- ### Dynamic Degree using Negative Degree with PowerShell Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example demonstrates using FastTransfer with a negative degree for dynamic adjustment. It specifies source and target connection details for PostgreSQL and MS Bulk respectively, including server, user, password, database, schema, and table information. The load mode is set to 'Truncate'. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:15432" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --sourcedatabase "tpch" ` --sourceschema "tpch_10" ` --sourcetable "orders" ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "tpch_test" ` --targettrusted ` --targetschema "dbo" ` --targettable "orders" ` --method "Ctid" ` --degree -4 ` --loadmode "Truncate" ``` -------------------------------- ### Core CLI Command: Transfer PostgreSQL to SQL Server Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This snippet demonstrates a basic data transfer from PostgreSQL to SQL Server using the Ctid parallelization method. It requires specifying source and target connection details, table names, and load mode. The expected output includes information about the transfer's start, row count, and completion status. ```bash # Transfer data from PostgreSQL to SQL Server using parallel Ctid method ./FastTransfer \ --sourceconnectiontype "pgsql" \ --sourceserver "localhost:5432" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --sourcedatabase "tpch" \ --sourceschema "tpch_10" \ --sourcetable "orders" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targetuser "fastuser" \ --targetpassword "fastpassword" \ --targetdatabase "tpch_test" \ --targetschema "dbo" \ --targettable "orders" \ --method "Ctid" \ --degree -2 \ --loadmode "Truncate" \ --mapmethod "Name" \ --runid "pgsql-to-mssql-orders" # Expected output: # [Timestamp] -|- FastTransfer -|- pgsql-to-mssql-orders -|- INFORMATION -|- dbo.orders -|- Transfer started # [Timestamp] -|- FastTransfer -|- pgsql-to-mssql-orders -|- INFORMATION -|- dbo.orders -|- Loaded 15000000 rows in 12345ms # [Timestamp] -|- FastTransfer -|- pgsql-to-mssql-orders -|- INFORMATION -|- dbo.orders -|- Transfer completed successfully ``` -------------------------------- ### PostgreSQL to PostgreSQL Transfer using Ctid Method with PowerShell Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example shows a data transfer between PostgreSQL instances using the 'Ctid' method. It specifies source and target connection details using `pgcopy` and includes parameters like `--degree` and `--loadmode 'Truncate'`. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:5432" ` --sourcedatabase "tpch" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --targetconnectiontype "pgcopy" ` --targetserver "localhost" ` --targetdatabase "tpch_import" ` --sourceschema "tpch_10" ` --sourcetable "orders" ` --targetuser "FastUser" ` --targetpassword "FastPassword" ` --targetschema "tpch_10" ` --targettable "orders" ` --method "Ctid" ` --degree 6 ` --loadmode "Truncate" ``` -------------------------------- ### PostgreSQL to PostgreSQL Transfer using Pgcopy Binary (Method Ctid) with PowerShell Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example demonstrates a PostgreSQL to PostgreSQL data transfer using `pgcopy` connection type and the 'Ctid' method. It specifies source and target connection parameters, including server addresses, database names, schemas, tables, degree, and load mode. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgcopy" ` --sourceserver "localhost:5432" ` --sourcedatabase "tpch" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --targetconnectiontype "pgcopy" ` --targetserver "localhost:15432" ` --targetdatabase "tpch_import" ` --sourceschema "tpch_10" ` --sourcetable "orders" ` --targetuser "FastUser" ` --targetpassword "FastPassword" ` --targetschema "tpch_10" ` --targettable "orders" ` --method "Ctid" ` --degree 10 ` --loadmode "Truncate" ``` -------------------------------- ### PostgreSQL to MSSQL: Parallel Transfer using DataDriven method Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example showcases a parallel data transfer from PostgreSQL to MSSQL using the 'DataDriven' method. It employs `--distributeKeyColumn` to define how data is partitioned and `--degree` to specify the number of parallel threads. This method is suitable for large datasets where performance is critical. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:15432" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --sourcedatabase "tpch" ` --query "select * from tpch_10.orders" ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "tpch_test" ` --targettrusted ` --targetschema "dbo" ` --targettable "orders" ` --method "DataDriven" ` --distributeKeyColumn "FLOOR(o_orderkey/10000000)" ` --degree 6 ``` -------------------------------- ### PostgreSQL to PostgreSQL Transfer via Query with PowerShell Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example demonstrates transferring data from one PostgreSQL instance to another using a custom SQL query. It utilizes the `pgcopy` connection type for both source and target. The `--query` parameter specifies the SELECT statement to retrieve data. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:5432" ` --sourcedatabase "tpch" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --targetconnectiontype "pgcopy" ` --targetserver "localhost" ` --targetdatabase "tpch_import" ` --query "select * from \"tpch_10\".orders" ` --targetuser "FastUser" ` --targetpassword "FastPassword" ` --targetschema "tpch_10" ` --targettable "orders" ` --method "None" ``` -------------------------------- ### PostgreSQL to MSSQL: Transfer specified schema and table Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example shows how to transfer a specific table from a PostgreSQL schema to an MSSQL table. It utilizes `--sourceschema`, `--sourcetable`, `--targetschema`, and `--targettable` to precisely define the data source and destination. The `--method 'None'` indicates a standard, non-parallel transfer. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:5432" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --sourcedatabase "tpch" ` --sourceschema "tpch_10" ` --sourcetable "orders" ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "tpch_test" ` --targettrusted ` --targetschema "dbo" ` --targettable "orders" ` --method "None" ` --loadmode "Truncate" ``` -------------------------------- ### Transfer ClickHouse to PostgreSQL - Query Based Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md Transfers data from a ClickHouse source to a PostgreSQL target using a SQL query. This example uses the `pgcopy` target connection type for efficient PostgreSQL loading. It defines source and target connection parameters, the query to select data from ClickHouse, and load modes. ```powershell . FastTransfer.exe \ --targetconnectiontype "pgcopy" \ --targetserver "localhost:15432" \ --targetdatabase "tpch" \ --targetuser "FastUser" \ --targetpassword "FastPassword" \ --targetschema "tpch_10" \ --targettable "orders_empty" \ --sourceconnectiontype "clickhouse" \ --sourceserver "localhost:8123" \ --sourcedatabase "default" \ --sourceuser "default" \ --sourcepassword "jKpHQEfXC91VCWrXoMb3" \ --sourceschema "default" \ --query "select * from default.orders limit 100000" \ --method "None" \ --degree 15 \ --loadmode "Truncate" \ --batchsize 10000 ``` -------------------------------- ### DuckDB File to PostgreSQL Transfer with Vector Data Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt Transfers data from a DuckDB file to PostgreSQL, including vector data, using the Ntile method. This example specifies a DuckDB file as the source and converts vector embeddings to text for transfer. It requires defining source and target connection parameters, along with the specific query to select and transform data. ```bash ./FastTransfer \ --sourceconnectiontype "duckdbstream" \ --sourceserver "/data/dbpedia_14.duckdb" \ --sourceschema "main" \ --sourcetable "dbpedia_14" \ --query "SELECT label, title, content, split, content_word_count, chunk, content_n_chunks, chunk_index, chunk_word_count, chunk_embedding::text FROM dbpedia_14" \ --targetconnectiontype "pgcopy" \ --targetserver "localhost:5432" \ --targetuser "postgres" \ --targetpassword "postgres_password" \ --targetdatabase "wikipedia" \ --targetschema "public" \ --targettable "dbpedia_14" \ --method "Ntile" \ --distributeKeyColumn "chunk_index" \ --degree 14 \ --loadmode "Truncate" \ --runid "DuckDB_To_pgcopy_14Thread" ``` -------------------------------- ### PostgreSQL to MSSQL: Transfer using SQL Query for another table Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md Similar to the first example, this snippet transfers data from PostgreSQL to MSSQL using a SQL query, but targets a different table ('nation'). It highlights the flexibility of the `--query` parameter for selecting specific datasets and mapping them to different target tables. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:5432" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --sourcedatabase "tpch" ` --query "select * from tpch_10.nation" ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "tpch_test" ` --targettrusted ` --targetschema "dbo" ` --targettable "nation" ` --method "None" ` --loadmode "Truncate" ``` -------------------------------- ### PostgreSQL to MSSQL: Parallel Transfer using Random method Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example utilizes the 'Random' method for parallel data transfer from PostgreSQL to MSSQL. It includes connection details, a SQL query for data selection, and specifies the target table. The `--distributeKeyColumn` is used to define the column for random distribution, and `--degree` controls the level of parallelism. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "pgsql" ` --sourceserver "localhost:15432" ` --sourceuser "FastUser" ` --sourcepassword "FastPassword" ` --sourcedatabase "tpch" ` --query "select * from tpch_10.orders" ` --targetconnectiontype "msbulk" ` --targetserver "localhost" ` --targetdatabase "tpch_test" ` --targettrusted ` --targetschema "dbo" ` --targettable "orders" ` --method "Random" ` --distributeKeyColumn "o_orderkey" ` --degree 6 ``` -------------------------------- ### MSSQL to MSSQL with Random Distribution using Physloc Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/samples/samples.md This example shows migrating data between two MSSQL instances. It utilizes the 'Random' distribution method with explicit column mapping using '%%physloc%%' to define the distribution key. The source query selects all data from a specified table, and the target uses MSBulk load with Truncate mode. This method is suitable for distributing data across multiple nodes or partitions. ```powershell .\FastTransfer.exe ` --sourceconnectiontype "mssql" ` --sourceserver "localhost" ` --sourcetrusted ` --sourcedatabase "tpch_test" ` --sourceschema "dbo" ` --query "select * from dbo.orders_15M" ` --targetconnectiontype "msbulk" ` --targetserver "localhost,11433" ` --targettrusted ` --targetdatabase "tpch_test" ` --targetschema "dbo" ` --targettable "orders_ccsi" ` --loadmode "Truncate" ` --batchsize 120000 ` --method "Random" ` --distributekeycolumn "ABS(CAST(CAST(%%physloc%% AS BINARY(1)) as bigint))" ` --degree 10 ` --mapmethod "Name" ` --runid "test_MSSQL22_to_MSSQL25_P10_Random_Physloc_to_ccsi" ``` -------------------------------- ### FastTransfer Command-Line Options Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/index.md This section outlines the available command-line options for configuring source and target connections, data manipulation, and parallelism for the FastTransfer tool. ```APIDOC ## FastTransfer Command-Line Options **WARNING : WE STRONG ADVISE TO USE LONG PARAMETERS** ### General Options - `-?`, `--help`: Show help information. ### Source Connection Options - `-c`, `--sourceconnectiontype `: Source Connection Type. Allowed Values: * `clickhouse` for ClickHouse * `duckdb` for duckdb * `duckdbstream` for duckdb using streaming (more memory efficient) * `hana` for SAP HANA (.net driver) * `msoledbsql` SQL Server OleDB * `mssql` SQL Server Native Client (.Net) * `mysql` MySQL * `nzcopy` Netezza copy (WIP) * `nzoledb` Netzza OleDB * `nzsql` Netezza Native (.net driver) * `odbc` ODBC datasource (DSN must be configured) * `oledb` Generic OleDB datasource * `oraodp` Oracle ODP.Net * `pgcopy` PostgreSQL Copy * `pgsql` PostgreSQL Native (.Net) * `teradata` Teradata (.Net) - `-g`, `--sourceconnectstring `: Source Connection String. (override all other source connection parameters). - `-n`, `--sourcedsn `: ODBC DSN. (only if odbc source type is used). Drivers must be already installed on the machine and DSN must be configured. - `-p`, `--sourceprovider `: OleDB provider (e.g., `MSOLEDBSQL` for MSSQL or `NZOLEDB` for Netezza...). Only if oledb source type is used. OleDB provider must be already installed on the machine. - `-i`, `--sourceserver `: * Source SQL instance (Server or Server\instance or Server:port) * for DuckDB the .duckdb file or :memory: - `-u`, `--sourceuser `: Source user. - `-x`, `--sourcepassword `: Source user's password. - `-a`, `--sourcetrusted`: Switch to use trusted authentication on source. - `-d`, `--sourcedatabase `: Source database. - `-s`, `--sourceschema `: Source schema. (must be set if pgsql Ctid method is used) - `-t`, `--sourcetable `: Source table. (must be set if pgsql Ctid method is used) - `-q`, `--query `: Plain text SQL query. Will be used instead of source table if provided. - `-f`, `--fileinput `: Input file storing SQL query. The file must exist. ### Target Connection Options - `-C`, `--targetconnectiontype `: Target Connection Type. Allowed Values: * `clickhousebulk` for ClickHouse BulkCopy * `duckdb` for DuckDB * `hanabulk` for SAP HANA * `msbulk` for SQL Server BulkCopy * `mysqlbulk` for MySQL BulkCopy * `nzbulk` for Netezza BulkCopy * `orabulk` for Oracle BulkCopy * `oradirect` for Oracle Direct * `pgcopy` for postgresql using copy (Binary format for postgresql sources and Text for others) * `pgsql` for PostgreSQL Native (.Net) * `teradata` for Teradata (.Net) - `-I`, `--targetserver `: * Target SQL instance : Server or Server\instance or Server:port. * .duckdb file for DuckDB - `-U`, `--targetuser `: Target user. - `-X`, `--targetpassword `: Target user's password. - `-A`, `--targettrusted`: Switch to use trusted authentication on target. - `-D`, `--targetdatabase `: Target database. - `-S`, `--targetschema `: Target schema. - `-T`, `--targettable
`: Target table. ### Parallelism and Data Distribution Options - `-M`, `--method `: Method for parallelism (if needed). Allowed Values: - `DataDriven`: Use the distinct value of the distributeKeyColumn (which can be a column or an expression) to distribute data - `Ctid`: Recommanded and exclusive for postgreSQL and postgreSQL compatible. Use the Ctid pseudo column (for pgsql and pgcopy source only) - `Random`: Use a modulo on the distributeKeyColumn to distribute data - `Rowid`: For Oracle sources only : use rowid slices - `RangeId`: Use a numeric range to distribute data (useful with an identity column or sequence without gaps. The column must be numerical) - `Ntile`: Use the ntile function to distribute data evenly data can be numerical, date, datetime or string - `NZDataSlice`: Netezza source only. Use the data slices to distribute data retrieval - `None`: No parallelism Default Value: `None`. If you want to use a parallel export/import use other than None. Try to use Parallelism only if you have a large amount of data to transfer (more than 1M cells). - `-K`, `--distributeKeyColumn `: Column to be used to distribute data Not needed if Ctid method is used. - `-P`, `--degree `: Degree of Parallelism * `0` for Auto * `0 > n < 1024` for fixed degree. * `n < 0` negative values will be used to adapt the degree of parallelism to the number of available CPUs. eg : -2 will use half the cpus on the machine where FastTransfer is launched Nota : whatever the degree, if the method is `None`, the extraction will remain serial Default Value: `-2`. - `Q`, `--datadrivenquery`: (Description not provided in source text) ``` -------------------------------- ### MySQL to SQL Server: DataDriven Transfer with Partitioning Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This snippet illustrates transferring data from MySQL to SQL Server using the DataDriven method, specifically partitioning by year. It specifies connection details for both databases, a custom query, and target schema/table information. ```bash ./FastTransfer \ --sourceconnectiontype "mysql" \ --sourceserver "localhost" \ --sourcedatabase "tpch" \ --sourceuser "fastuser" \ --sourcepassword "fastpassword" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targetdatabase "tpch_test" \ --query "select * from tpch.orders" \ --targettrusted \ --targetschema "dbo" \ --targettable "orders" \ --method "DataDriven" \ --distributeKeyColumn "year(o_orderdate)" \ --degree 10 ``` -------------------------------- ### Build and Display Transfer Command (JavaScript) Source: https://github.com/aetperf/fasttransfer-documentation/blob/main/FastTransfer_Wizard.html Constructs the command-line arguments for FastTransfer based on user input from HTML elements. It dynamically builds a command string and displays it in a designated area. Dependencies include HTML elements with specific IDs like 'license', 'result', and 'baseCmd'. ```javascript function updateCommand() { const se = document.getElementById('license').value.trim(); if (license) { args.push(` --license "${license}"`); } // Build final command let cmd = baseCmd; if (args.length > 0) { cmd += " " + args.join(" "); } // Display document.getElementById('result').textContent = cmd; } ``` -------------------------------- ### Oracle to SQL Server: Rowid Method Transfer Source: https://context7.com/aetperf/fasttransfer-documentation/llms.txt This example transfers data from Oracle to SQL Server using the Rowid method, leveraging Oracle-specific parallelism. It includes source and target connection details, schema and table specifications, and load mode. ```bash ./FastTransfer \ --sourceconnectiontype "oraodp" \ --sourceserver "ora-server:1521/ORCLPDB" \ --sourcedatabase "ORCLPDB" \ --sourceuser "TPCH_IN" \ --sourcepassword "TPCH_IN" \ --sourceschema "TPCH_IN" \ --sourcetable "ORDERS_FLAT" \ --targetconnectiontype "msbulk" \ --targetserver "localhost" \ --targetdatabase "tpch_test" \ --targettrusted \ --targetschema "dbo" \ --targettable "orders" \ --method "Rowid" \ --loadmode "Truncate" \ --degree 12 ```