### Install Specific Python3 Library Example Source: https://docs.matillion.com/metl/docs/8366145 Example of installing a specific version of the boto3 Python3 library. ```bash sudo pip3 install boto3=1.17.45 ``` -------------------------------- ### Install Specific Python2 Library Example Source: https://docs.matillion.com/metl/docs/8366145 Example of installing a specific version of the boto3 Python2 library. ```bash sudo pip install boto3==1.14.53 ``` -------------------------------- ### Install Boto3 Module with Pip Source: https://docs.matillion.com/metl/docs/2234735 Example command to install the Boto3 module, a popular AWS SDK for Python, using pip. ```bash sudo pip install boto3 ``` -------------------------------- ### API Key as Query Parameter Example Source: https://docs.matillion.com/metl/docs/authentication-with-api-profiles Demonstrates how to include an API key as a query parameter in a GET request. ```http GET /https://api.exchangeratesapi.io/latest?api_key=abcdef12345 ``` -------------------------------- ### Resource Metadata Example (GET Response) Source: https://docs.matillion.com/metl/docs/2916124 This JSON response shows the metadata for a project resource, including available child endpoints like 'getSchedule'. ```json { "endpoints": [ { "httpMethod": "PATH", "name": "ProjectInstanceService", "children": [ { "httpMethod": "PATH", "name": "getSchedule", "description": "Allows accessing Schedule APIs", "path": "/schedule", "children": [ { "httpMethod": "GET", "name": "export", "description": "Exports all of the schedules in the project", "path": "/export" }, { "httpMethod": "PATH", "name": "getBySomething", "description": "Get a Schedule API by query parameters", "path": "/instance" }, ...... ``` -------------------------------- ### Matillion ETL API v1 Get Specific Group Example Source: https://docs.matillion.com/metl/docs/2916124 Example of a GET request to access a specific group by its name, using the group/name/ endpoint. ```http GET http:///rest/v1/group/name/Example-group ``` -------------------------------- ### Execute Installation Script Source: https://docs.matillion.com/metl/docs/2478877 Download and run the orchestration installation script using the provided variables file. ```bash curl https://bitbucket.org/matillion/matillion-etl-system-configuration/raw/production/orchestrate-install.sh --output /tmp/orchestrate-install.sh sudo bash /tmp/orchestrate-install.sh /tmp/installation.vars ``` -------------------------------- ### Matillion ETL API v1 Get All Groups Example Source: https://docs.matillion.com/metl/docs/2916124 Example of a GET request to retrieve a list of all available groups defined within the Matillion ETL instance. ```http GET https:///rest/v1/group ``` -------------------------------- ### Install MkDocs and Plugins Source: https://docs.matillion.com/data-productivity-cloud/private-docs/contribute-to-docs/contribute-prerequisites Run this command to install MkDocs, the Material theme, and several essential plugins for documentation management. If 'pip' is not found, try 'pip3'. ```bash pip install mkdocs mkdocs-material mkdocs-include-markdown-plugin mkdocs-redirects mkdocs-glightbox mkdocs-open-in-new-tab mkdocs-macros-plugin ``` -------------------------------- ### Check Installed Matillion ETL Version Source: https://docs.matillion.com/metl/docs/updating-to-specific-releases Verify the currently installed version of Matillion ETL by listing installed packages starting with 'matillion*'. ```bash yum list installed matillion* ``` -------------------------------- ### Configuration File Example (JSON) Source: https://docs.matillion.com/assets/snippets/modular/configure A sample JSON file structure for storing configuration parameters. This format is widely used for its readability and ease of parsing. ```json { "database_url": "postgresql://user:password@host:port/database", "api_key": "your_api_key" } ``` -------------------------------- ### Python Pushdown: Key-Based Lookup Example Source: https://docs.matillion.com/data-productivity-cloud/designer/docs/grid-variables Demonstrates using a key-based lookup function with the `LoadOptions` grid variable in Python Pushdown to retrieve specific configuration values. ```python rows = LoadOptions # Returns a 2D Python list def get_value_by_key(key): for row in rows: if row[0] == key: return row[1] return None # Look up values by key and assign them to Python variables file_format = get_value_by_key('File Format') # will return 'CSV' compression = get_value_by_key('Compression') # will return 'GZIP' ``` -------------------------------- ### GET /permission/user Source: https://docs.matillion.com/metl/docs/2156378 To get the list of users for the specified permission. This example using GET method REST API call to export the permission users. ```APIDOC ## GET /permission/user ### Description To get the list of users for the specified permission. ### Method GET ### Endpoint http://[InstanceAddress]/rest/v1/permission/user ### Parameters #### Query Parameters - **[InstanceAddress]** (string) - Required - The server IP address or domain name. ### Response #### Success Response (200) - **user-1** (string) - Description of user-1 - **user-2** (string) - Description of user-2 - **user-3** (string) - Description of user-3 ### Response Example { "user-1": "", "user-2": "", "user-3": "" } ``` -------------------------------- ### Get Shared Job Packages Response Example Source: https://docs.matillion.com/metl/docs/2956236 Example JSON response when retrieving shared job packages. ```json [ "TestPackage-1", "TestPackage-2", "TestPackage-3" ] ``` -------------------------------- ### Download default installation.vars file Source: https://docs.matillion.com/metl/docs/2478877 Use this command to download the default `installation.vars` file for configuring the Matillion ETL installation. This file specifies service versions and other configuration options. ```bash curl https://bitbucket.org/matillion/matillion-etl-system-configuration/raw/production/example-config/installation.vars --output /tmp/installation.vars ``` -------------------------------- ### Example Key Pattern for Define Tables Source: https://docs.matillion.com/metl/docs/2965324 Illustrates how to group keys under a single table using a key pattern in the 'Define Tables' parameter. ```text Company=Company* ``` -------------------------------- ### Get Differences Response Example Source: https://docs.matillion.com/metl/docs/5971755 Example response structure for retrieving differences, detailing branches and changes. The 'type' can be ADDED, REMOVED, or MODIFIED. ```json { "success": true, "result": { "branches": [ "master" ], "changes": { "dotfileChanges": [], "nonDotfileChanges": [ { "type": "MODIFIED", "filePath": "ROOT/o1.ORCHESTRATION" }, { "type": "MODIFIED", "filePath": "version" } ] } } } ``` -------------------------------- ### Universal Matillion ETL package examples Source: https://docs.matillion.com/metl/docs/6661919 Examples of the universally compatible build format introduced in version 1.60. ```text emerald-metl-1.60.2-.rpm (matillion-universal package) emerald-metl-1.60.2-.rpm (matillion-universal package) ``` -------------------------------- ### Example Response for GET Notice Source: https://docs.matillion.com/metl/docs/4653721 A successful GET request to the notice endpoint returns a JSON object containing a list of notice IDs. ```json { "mtln-cdc-queue-error", "123453", "mtln-sqs-error", "12345", "12343", "12346" } ``` -------------------------------- ### Verify Successful Startup Message Source: https://docs.matillion.com/metl/docs/reverting-from-external-to-internal-security Expected log output indicating the server has successfully started. ```text org.apache.catalina.startup.Catalina.start Server startup in ? ms ``` -------------------------------- ### Get Task History by Start Time Range Source: https://docs.matillion.com/metl/docs/2972278 Retrieves task history entries that started within a specific date and time range. ```APIDOC ## GET /rest/v1/group/name//project/name//task/filter/by/start/range/date//time//to/date//time/ ### Description Retrieves all entries in the task history which started within a specific time range. ### Method GET ### Endpoint `/rest/v1/group/name//project/name//task/filter/by/start/range/date//time//to/date//time/` ### Parameters #### Path Parameters - **groupName** (string) - Required - The name of the group. - **projectName** (string) - Required - The name of the project. - **startDate** (string) - Required - The start date in `yyyy-mm-dd` format. - **startTime** (string) - Required - The start time in `hh:mm` format. - **endDate** (string) - Required - The end date in `yyyy-mm-dd` format. - **endTime** (string) - Required - The end time in `hh:mm` format. ### Response #### Success Response (200) - **taskHistory** (array) - A list of task history entries within the specified start time range. ``` -------------------------------- ### Start Konlet Startup Service Source: https://docs.matillion.com/data-productivity-cloud/cdc/docs/2536271 Initiate the `konlet-startup` service, which is responsible for creating a new Docker container instance with the updated image based on defined container metadata. ```bash sudo systemctl start konlet-startup ``` -------------------------------- ### Package Version Output Source: https://docs.matillion.com/metl/docs/6661919 Example output formats for installed matillion-emerald packages. ```text matillion-emerald-byol- ``` ```text matillion-emerald- ``` -------------------------------- ### GET by/start/after/date/ Source: https://docs.matillion.com/metl/docs/2920263 List all entries in the task history which started on or after the given date. ```APIDOC ## GET by/start/after/date/ ### Description List all entries in the task history which started on or after the given date. ### Method GET ### Endpoint http:///rest/v1/group/name//project/name//task/filter/by/start/after/date/ ``` -------------------------------- ### Load Configuration from JSON File Source: https://docs.matillion.com/assets/snippets/modular/configure Demonstrates loading configuration settings from a JSON file in Python. Ensure the JSON file is correctly formatted. ```python import json with open("config.json", "r") as f: config = json.load(f) print(config["database_url"]) ``` -------------------------------- ### Matillion Server Migration Tool Help Source: https://docs.matillion.com/metl/private-docs/2904980 Displays the help message for the `migrate_server.py` script, showing all available command-line arguments and their descriptions. Use this to understand the migration options. ```bash $ ./migrate_server.py --help ``` -------------------------------- ### GET Export Job Metadata Response Source: https://docs.matillion.com/metl/docs/2956236 Example JSON response when exporting shared job metadata. ```json { "objects": [ { "metadata": {...}, "orchestrationJobs": {...}, "transformationJobs": {...}, }, ], "version": "master", "environment": "redshift" } ``` -------------------------------- ### Example Matillion ETL Database Configuration Source: https://docs.matillion.com/metl/docs/8564237 This is an example of the `Emerald.properties` configuration for connecting to an Azure PostgreSQL database. ```bash PERSISTENCE_STORE_NAME=postgres PERSISTENCE_USERNAME_POSTGRES=username@shorthostname PERSISTENCE_PASSWORD_POSTGRES={enc:base64}bjMxMEthR2frght PERSISTENCE_URL_POSTGRES=jdbc:postgresql://shorthostname.postgres.database.azure.com:5432/postgres?ssl=true&sslmode=require ``` -------------------------------- ### GET Shared Job List Response Source: https://docs.matillion.com/metl/docs/2956236 Example JSON response when retrieving a list of shared jobs. ```json [ "SharedJob" ] ``` -------------------------------- ### Successful GET Export Environment Variables Response Source: https://docs.matillion.com/metl/docs/2768621 Example of a successful response when exporting environment variables. ```JSON { "objects": [ { "name": "Example_variable", "value": "88.8" }, ], "version": "master", "environment": "redshift" } ``` -------------------------------- ### Install cloud-utils-growpart Source: https://docs.matillion.com/metl/docs/tech-note-disk-partition-169 Installs the necessary utility to grow disk partitions on CentOS-based systems. ```bash sudo su yum install cloud-utils-growpart ``` -------------------------------- ### Example Partition Key-Value Combinations Source: https://docs.matillion.com/metl/docs/2936386 Illustrates potential partition key-value combinations for a 'year' column, showing how data for different years would be organized in S3 folders. ```text year=2010 year=2011 year=2012 ``` -------------------------------- ### GET by/start/after/date//time/