Maia

https://docs.matillion.com
Admin
Maia is a low-code data integration and ETL platform that enables users to design, deploy, and...
Tokens:968,543
Snippets:14,303
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Specific Python3 Library Example

Source: https://docs.matillion.com/metl/docs/8366145

Example of installing a specific version of the boto3 Python3 library.

```bash
sudo pip3 install boto3=1.17.45
```

--------------------------------

### Install Specific Python2 Library Example

Source: https://docs.matillion.com/metl/docs/8366145

Example of installing a specific version of the boto3 Python2 library.

```bash
sudo pip install boto3==1.14.53
```

--------------------------------

### Install Boto3 Module with Pip

Source: https://docs.matillion.com/metl/docs/2234735

Example command to install the Boto3 module, a popular AWS SDK for Python, using pip.

```bash
sudo pip install boto3

```

--------------------------------

### API Key as Query Parameter Example

Source: https://docs.matillion.com/metl/docs/authentication-with-api-profiles

Demonstrates how to include an API key as a query parameter in a GET request.

```http
GET /https://api.exchangeratesapi.io/latest?api_key=abcdef12345
```

--------------------------------

### Resource Metadata Example (GET Response)

Source: https://docs.matillion.com/metl/docs/2916124

This JSON response shows the metadata for a project resource, including available child endpoints like 'getSchedule'.

```json
{
  "endpoints": [
    {
      "httpMethod": "PATH",
      "name": "ProjectInstanceService",
      "children": [
        {
          "httpMethod": "PATH",
          "name": "getSchedule",
          "description": "Allows accessing Schedule APIs",
          "path": "/schedule",
          "children": [
            {
              "httpMethod": "GET",
              "name": "export",
              "description": "Exports all of the schedules in the project",
              "path": "/export"
            },
            {
              "httpMethod": "PATH",
              "name": "getBySomething",
              "description": "Get a Schedule API by query parameters",
              "path": "/instance"
            },
                ......

```

--------------------------------

### Matillion ETL API v1 Get Specific Group Example

Source: https://docs.matillion.com/metl/docs/2916124

Example of a GET request to access a specific group by its name, using the group/name/<groupName> endpoint.

```http
GET http://<InstanceAddress>/rest/v1/group/name/Example-group
```

--------------------------------

### Execute Installation Script

Source: https://docs.matillion.com/metl/docs/2478877

Download and run the orchestration installation script using the provided variables file.

```bash
curl https://bitbucket.org/matillion/matillion-etl-system-configuration/raw/production/orchestrate-install.sh --output /tmp/orchestrate-install.sh
sudo bash /tmp/orchestrate-install.sh /tmp/installation.vars
```

--------------------------------

### Matillion ETL API v1 Get All Groups Example

Source: https://docs.matillion.com/metl/docs/2916124

Example of a GET request to retrieve a list of all available groups defined within the Matillion ETL instance.

```http
GET https://<InstanceAddress>/rest/v1/group
```

--------------------------------

### Install MkDocs and Plugins

Source: https://docs.matillion.com/data-productivity-cloud/private-docs/contribute-to-docs/contribute-prerequisites

Run this command to install MkDocs, the Material theme, and several essential plugins for documentation management. If 'pip' is not found, try 'pip3'.

```bash
pip install mkdocs mkdocs-material mkdocs-include-markdown-plugin mkdocs-redirects mkdocs-glightbox mkdocs-open-in-new-tab mkdocs-macros-plugin

```

--------------------------------

### Check Installed Matillion ETL Version

Source: https://docs.matillion.com/metl/docs/updating-to-specific-releases

Verify the currently installed version of Matillion ETL by listing installed packages starting with 'matillion*'.

```bash
yum list installed matillion*
```

--------------------------------

### Configuration File Example (JSON)

Source: https://docs.matillion.com/assets/snippets/modular/configure

A sample JSON file structure for storing configuration parameters. This format is widely used for its readability and ease of parsing.

```json
{
  "database_url": "postgresql://user:password@host:port/database",
  "api_key": "your_api_key"
}

```

--------------------------------

### Python Pushdown: Key-Based Lookup Example

Source: https://docs.matillion.com/data-productivity-cloud/designer/docs/grid-variables

Demonstrates using a key-based lookup function with the `LoadOptions` grid variable in Python Pushdown to retrieve specific configuration values.

```python
rows = LoadOptions  # Returns a 2D Python list

def get_value_by_key(key):
    for row in rows:
        if row[0] == key:
            return row[1]
    return None

# Look up values by key and assign them to Python variables
file_format = get_value_by_key('File Format')           # will return 'CSV'
compression = get_value_by_key('Compression')           # will return 'GZIP'
```

--------------------------------

### GET /permission/user

Source: https://docs.matillion.com/metl/docs/2156378

To get the list of users for the specified permission. This example using GET method REST API call to export the permission users.

```APIDOC
## GET /permission/user

### Description
To get the list of users for the specified permission.

### Method
GET

### Endpoint
http://[InstanceAddress]/rest/v1/permission/user

### Parameters
#### Query Parameters
- **[InstanceAddress]** (string) - Required - The server IP address or domain name.

### Response
#### Success Response (200)
- **user-1** (string) - Description of user-1
- **user-2** (string) - Description of user-2
- **user-3** (string) - Description of user-3

### Response Example
{
  "user-1": "",
  "user-2": "",
  "user-3": ""
}
```

--------------------------------

### Get Shared Job Packages Response Example

Source: https://docs.matillion.com/metl/docs/2956236

Example JSON response when retrieving shared job packages.

```json
[
    "TestPackage-1",
    "TestPackage-2",
    "TestPackage-3"
]
```

--------------------------------

### Download default installation.vars file

Source: https://docs.matillion.com/metl/docs/2478877

Use this command to download the default `installation.vars` file for configuring the Matillion ETL installation. This file specifies service versions and other configuration options.

```bash
curl https://bitbucket.org/matillion/matillion-etl-system-configuration/raw/production/example-config/installation.vars --output /tmp/installation.vars
```

--------------------------------

### Example Key Pattern for Define Tables

Source: https://docs.matillion.com/metl/docs/2965324

Illustrates how to group keys under a single table using a key pattern in the 'Define Tables' parameter.

```text
Company=Company*
```

--------------------------------

### Get Differences Response Example

Source: https://docs.matillion.com/metl/docs/5971755

Example response structure for retrieving differences, detailing branches and changes. The 'type' can be ADDED, REMOVED, or MODIFIED.

```json
{
    "success": true,
    "result": {
        "branches": [
            "master"
        ],
        "changes": {
            "dotfileChanges": [],
            "nonDotfileChanges": [
                {
                    "type": "MODIFIED",
                    "filePath": "ROOT/o1.ORCHESTRATION"
                },
                {
                    "type": "MODIFIED",
                    "filePath": "version"
                }
            ]
        }
    }
}
```

--------------------------------

### Universal Matillion ETL package examples

Source: https://docs.matillion.com/metl/docs/6661919

Examples of the universally compatible build format introduced in version 1.60.

```text
emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)
emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)
```

--------------------------------

### Example Response for GET Notice

Source: https://docs.matillion.com/metl/docs/4653721

A successful GET request to the notice endpoint returns a JSON object containing a list of notice IDs.

```json
{
    "mtln-cdc-queue-error",
    "123453",
    "mtln-sqs-error",
    "12345",
    "12343",
    "12346"
}

```

--------------------------------

### Verify Successful Startup Message

Source: https://docs.matillion.com/metl/docs/reverting-from-external-to-internal-security

Expected log output indicating the server has successfully started.

```text
org.apache.catalina.startup.Catalina.start Server startup in ? ms
```

--------------------------------

### Get Task History by Start Time Range

Source: https://docs.matillion.com/metl/docs/2972278

Retrieves task history entries that started within a specific date and time range.

```APIDOC
## GET /rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/range/date/<startDate>/time/<startTime>/to/date/<endDate>/time/<endTime>

### Description
Retrieves all entries in the task history which started within a specific time range.

### Method
GET

### Endpoint
`<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/range/date/<startDate>/time/<startTime>/to/date/<endDate>/time/<endTime>`

### Parameters
#### Path Parameters
- **groupName** (string) - Required - The name of the group.
- **projectName** (string) - Required - The name of the project.
- **startDate** (string) - Required - The start date in `yyyy-mm-dd` format.
- **startTime** (string) - Required - The start time in `hh:mm` format.
- **endDate** (string) - Required - The end date in `yyyy-mm-dd` format.
- **endTime** (string) - Required - The end time in `hh:mm` format.

### Response
#### Success Response (200)
- **taskHistory** (array) - A list of task history entries within the specified start time range.
```

--------------------------------

### Start Konlet Startup Service

Source: https://docs.matillion.com/data-productivity-cloud/cdc/docs/2536271

Initiate the `konlet-startup` service, which is responsible for creating a new Docker container instance with the updated image based on defined container metadata.

```bash
sudo systemctl start konlet-startup
```

--------------------------------

### Package Version Output

Source: https://docs.matillion.com/metl/docs/6661919

Example output formats for installed matillion-emerald packages.

```text
matillion-emerald-byol-<some version>
```

```text
matillion-emerald-<some version>
```

--------------------------------

### GET by/start/after/date/<date>

Source: https://docs.matillion.com/metl/docs/2920263

List all entries in the task history which started on or after the given date.

```APIDOC
## GET by/start/after/date/<date>

### Description
List all entries in the task history which started on or after the given date.

### Method
GET

### Endpoint
http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/after/date/<date>
```

--------------------------------

### Load Configuration from JSON File

Source: https://docs.matillion.com/assets/snippets/modular/configure

Demonstrates loading configuration settings from a JSON file in Python. Ensure the JSON file is correctly formatted.

```python
import json

with open("config.json", "r") as f:
    config = json.load(f)

print(config["database_url"])

```

--------------------------------

### Matillion Server Migration Tool Help

Source: https://docs.matillion.com/metl/private-docs/2904980

Displays the help message for the `migrate_server.py` script, showing all available command-line arguments and their descriptions. Use this to understand the migration options.

```bash
$ ./migrate_server.py --help
```

--------------------------------

### GET Export Job Metadata Response

Source: https://docs.matillion.com/metl/docs/2956236

Example JSON response when exporting shared job metadata.

```json
{
    "objects": [
        {
            "metadata": {...},
            "orchestrationJobs": {...},
            "transformationJobs": {...},
        },
    ],
    "version": "master",
    "environment": "redshift"
}
```

--------------------------------

### Example Matillion ETL Database Configuration

Source: https://docs.matillion.com/metl/docs/8564237

This is an example of the `Emerald.properties` configuration for connecting to an Azure PostgreSQL database.

```bash
PERSISTENCE_STORE_NAME=postgres
PERSISTENCE_USERNAME_POSTGRES=username@shorthostname
PERSISTENCE_PASSWORD_POSTGRES={enc:base64}bjMxMEthR2frght
PERSISTENCE_URL_POSTGRES=jdbc:postgresql://shorthostname.postgres.database.azure.com:5432/postgres?ssl=true&sslmode=require

```

--------------------------------

### GET Shared Job List Response

Source: https://docs.matillion.com/metl/docs/2956236

Example JSON response when retrieving a list of shared jobs.

```json
[
    "SharedJob"
]
```

--------------------------------

### Successful GET Export Environment Variables Response

Source: https://docs.matillion.com/metl/docs/2768621

Example of a successful response when exporting environment variables.

```JSON
{
    "objects": [
        {
            "name": "Example_variable",
            "value": "88.8"
        },
    ],
    "version": "master",
    "environment": "redshift"
}
```

--------------------------------

### Install cloud-utils-growpart

Source: https://docs.matillion.com/metl/docs/tech-note-disk-partition-169

Installs the necessary utility to grow disk partitions on CentOS-based systems.

```bash
sudo su
yum install cloud-utils-growpart
```

--------------------------------

### Example Partition Key-Value Combinations

Source: https://docs.matillion.com/metl/docs/2936386

Illustrates potential partition key-value combinations for a 'year' column, showing how data for different years would be organized in S3 folders.

```text
year=2010
year=2011
year=2012
```

--------------------------------

### GET by/start/after/date/<date>/time/<time>

Source: https://docs.matillion.com/metl/docs/2920263

List all entries in the task history which started on or after the given date/time.

```APIDOC
## GET by/start/after/date/<date>/time/<time>

### Description
List all entries in the task history which started on or after the given date/time.

### Method
GET

### Endpoint
http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/after/date/<date>/time/<time>
```

--------------------------------

### GET by/start/before/date/<date>/time/<time>

Source: https://docs.matillion.com/metl/docs/2920263

List all entries in the task history which started before the given date/time.

```APIDOC
## GET by/start/before/date/<date>/time/<time>

### Description
List all entries in the task history which started before the given date/time.

### Method
GET

### Endpoint
http://<InstanceAddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/task/filter/by/start/before/date/<date>/time/<time>
```

--------------------------------

### List of Projects Example (GET Response)

Source: https://docs.matillion.com/metl/docs/2916124

This JSON array lists the names of projects available within a group. Resource names are case-sensitive and may require URL encoding.

```json
[
    "Example-project ",
    "Example-project 1",
    "Example-project 2"
]
```
Tokens:
Raw