Mage Pro

https://docs.mage.ai/
Admin
Mage Pro is a powerful, production-grade, cloud-native platform for building, running, and scaling...
Tokens:213,625
Snippets:1,453
Trust Score:9.7
Update:1 year ago
Show doc for...
Tokens:
Raw
### Set up Mage with Docker Compose (Windows)

Source: https://docs.mage.ai/getting-started/setup

Initializes Mage using Docker Compose on Windows by cloning the quickstart repository, copying the environment file, and starting the services. This allows for a customizable Mage setup.

```bash
git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart
cd mage-quickstart
cp dev.env .env
rm dev.env
docker compose up
```

--------------------------------

### Install Mage Dependencies

Source: https://docs.mage.ai/getting-started/setup

This command installs the Mage AI package with all add-on packages. It can be run directly from the Mage terminal or added to a requirements.txt file for project dependency management.

```bash
pip install "mage-ai[all]"
```

--------------------------------

### Start Mage Project with Docker (GCP VM Instance)

Source: https://docs.mage.ai/getting-started/setup

Initiates a Mage Docker container on a GCP VM instance, ensuring project files are persisted through volume mounting. The command starts a Mage server accessible on http://localhost:6789.

```bash
sudo docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]
```

--------------------------------

### Create and Launch Mage Project

Source: https://docs.mage.ai/getting-started/setup

Creates a new Mage project with the specified project name and launches the Mage tool. This command is used after installing Mage via pip or conda.

```bash
mage start [project_name]
```

--------------------------------

### Set up Mage with Docker Compose (Mac/Linux)

Source: https://docs.mage.ai/getting-started/setup

Clones the Mage Docker Compose quickstart repository, configures environment variables, and starts a Mage server using Docker Compose on Mac/Linux. This method is ideal for customizing the Mage environment or integrating with other services.

```bash
git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart 
&& cd mage-quickstart 
&& cp dev.env .env && rm dev.env 
&& docker compose up
```

--------------------------------

### Install Mage Spark Dependency

Source: https://docs.mage.ai/getting-started/setup

Installs the Mage AI package with Spark support, allowing for the use of Spark in Mage pipelines. This command can be executed in the Mage terminal or listed in a requirements.txt file.

```bash
pip install "mage-ai[spark]"
```

--------------------------------

### Add Mage Spark Dependency to requirements.txt

Source: https://docs.mage.ai/getting-started/setup

Specifies the Mage AI package with Spark support to be installed from a requirements.txt file, ensuring consistent dependency management for projects using Spark.

```text
mage-ai[spark]
```

--------------------------------

### Start Mage Project with Docker (Windows Command Line)

Source: https://docs.mage.ai/getting-started/setup

Executes a Docker container for Mage on Windows Command Line, mapping the current directory to persist project files. The Mage server will be available at http://localhost:6789.

```bash
docker run -it -p 6789:6789 -v "%cd%":/home/src mageai/mageai /app/run_app.sh mage start [project_name]
```

--------------------------------

### Start Mage Project with Docker (PowerShell)

Source: https://docs.mage.ai/getting-started/setup

Launches a Docker container for Mage using PowerShell, with volume mounting for project data persistence. Access the Mage server via http://localhost:6789.

```powershell
docker run -it -p 6789:6789 -v ${PWD}:/home/src mageai/mageai /app/run_app.sh mage start [project_name]
```

--------------------------------

### Install Mage with conda

Source: https://docs.mage.ai/getting-started/setup

Installs the Mage Python package using conda from the conda-forge channel. This method is suitable for users who manage their environments with Anaconda or Miniconda.

```bash
conda install -c conda-forge mage-ai
```

--------------------------------

### Start Mage Project Locally with Docker

Source: https://docs.mage.ai/production/ci-cd/local-cloud/repository-setup

Initiates a new Mage project named 'demo_project' within a Docker container. It maps the current directory to the container's volume and exposes the Mage port. Ensure Docker is installed and running.

```bash
docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
  /app/run_app.sh mage start demo_project
```

--------------------------------

### Manual Mage Installation and Start on EC2 (Bash)

Source: https://docs.mage.ai/production/deploying-to-cloud/aws-without-terraform

Manually installs and starts Mage on an EC2 instance after SSHing into it. This method involves installing Mage using pip and then initializing and starting the Mage application.

```bash
ssh -i path/to/key/pair.pem ec2_username@ec2_public_dns_name
pip install mage-ai

mage init <your_repo_name>
mage start <your_repo_name>
```

--------------------------------

### Set up Mage with Docker Compose (Windows PowerShell/CMD)

Source: https://docs.mage.ai/getting-started/setup

Sets up Mage with Docker Compose on Windows using PowerShell or Command Prompt. It involves cloning a repository, preparing the environment configuration, and launching the Mage server.

```powershell
git clone https://github.com/mage-ai/compose-quickstart.git mage-quickstart
cd mage-quickstart
copy dev.env .env
del dev.env
docker compose up
```

--------------------------------

### Python: Load Data with Basic Profile Usage

Source: https://docs.mage.ai/development/io_config_setup

Provides a Python example of loading data from a PostgreSQL database using a specified IO configuration profile ('default'). It utilizes `ConfigFileLoader` and `Postgres` from Mage AI.

```python
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

@data_loader
def load_data_from_postgres(**kwargs):
    query = 'SELECT * FROM your_table'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'  # or any other profile name

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

--------------------------------

### Run Mage in Kubernetes

Source: https://docs.mage.ai/getting-started/setup

Instructions for deploying Mage AI in a Kubernetes environment using kubectl. This involves creating the application from a YAML file, checking pod status, and setting up port forwarding.

```bash
kubectl create -f kube/app.yaml
kubectl get pods -o wide
kubectl port-forward mage-server 6789:6789
```

--------------------------------

### GET /search API Example

Source: https://docs.mage.ai/data-integrations/sources/api

This example demonstrates a GET request to a search API, including query parameters, headers, and response parsing.

```APIDOC
## GET /search

### Description
This endpoint searches for documents based on a query. It demonstrates how to specify query parameters, headers, and a response parser to extract specific data from the response.

### Method
GET

### Endpoint
https://api.plos.org/search

### Parameters
#### Query Parameters
- **q** (string) - Required - The search query string.

#### Headers
- **Content-Type** (string) - Required - Specifies the content type of the request.

### Request Example
```json
{
  "url": "https://api.plos.org/search",
  "query": {
    "q": "title:DNA"
  },
  "headers": {
    "Content-Type": "application/json"
  },
  "response_parser": "response.docs[0].author_display",
  "columns": [
    "author_display"
  ]
}
```

### Response
#### Success Response (200)
- **author_display** (array) - An array of author names for the first document found.

#### Response Example
```json
[
  "Rayna I. Kraeva",
  "Dragomir B. Krastev",
  "Assen Roguev",
  "Anna Ivanova",
  "Marina N. Nedelcheva-Veleva",
  "Stoyno S. Stoynov"
]
```
```

--------------------------------

### Configure and Run Mage with dbt (Bash)

Source: https://docs.mage.ai/guides/dbt/setup-dbt

This command clones the dbt quickstart repository, sets up environment variables by copying `dev.env` to `.env`, and starts Mage and a Postgres database using Docker Compose. It's intended for Mac/Linux systems.

```bash
git clone https://github.com/mage-ai/dbt-quickstart mage-dbt-quickstart \
  && cd mage-dbt-quickstart \
  && cp dev.env .env && rm dev.env \
  && docker compose up
```

--------------------------------

### GET /coins/markets API Example

Source: https://docs.mage.ai/data-integrations/sources/api

This example shows a GET request to an API that retrieves cryptocurrency market data, focusing on query parameters.

```APIDOC
## GET /coins/markets

### Description
This endpoint retrieves market data for various cryptocurrencies. It takes a query parameter to specify the currency against which the market data should be presented.

### Method
GET

### Endpoint
https://api.coingecko.com/api/v3/coins/markets

### Parameters
#### Query Parameters
- **vs_currency** (string) - Required - The currency against which to return the coin data (e.g., 'usd').

### Request Example
```json
{
  "url": "https://api.coingecko.com/api/v3/coins/markets",
  "query": {
    "vs_currency": "usd"
  }
}
```

### Response
#### Success Response (200)
- **id** (string) - The unique identifier of the cryptocurrency.
- **symbol** (string) - The symbol of the cryptocurrency.
- **name** (string) - The name of the cryptocurrency.
- **image** (string) - URL of the cryptocurrency's image.
- **current_price** (number) - The current price of the cryptocurrency.
- **market_cap** (number) - The market capitalization of the cryptocurrency.
- **market_cap_rank** (integer) - The market cap rank of the cryptocurrency.
- **fully_diluted_valuation** (number) - The fully diluted valuation of the cryptocurrency.
- **total_volume** (number) - The total trading volume over the last 24 hours.
- **high_24h** (number) - The highest price in the last 24 hours.
- **low_24h** (number) - The lowest price in the last 24 hours.
- **price_change_24h** (number) - The price change in the last 24 hours.
- **price_change_percentage_24h** (number) - The percentage price change in the last 24 hours.
- **market_cap_change_24h** (number) - The market cap change in the last 24 hours.
- **market_cap_change_percentage_24h** (number) - The percentage market cap change in the last 24 hours.
- **circulating_supply** (number) - The circulating supply of the cryptocurrency.
- **total_supply** (number) - The total supply of the cryptocurrency.
- **max_supply** (number) - The maximum supply of the cryptocurrency.
- **ath** (number) - The all-time high price.
- **ath_change_percentage** (number) - The percentage change from the all-time high.
- **ath_date** (string) - The date of the all-time high.
- **atl** (number) - The all-time low price.
- **atl_change_percentage** (number) - The percentage change from the all-time low.
- **atl_date** (string) - The date of the all-time low.
- **roi** (object or null) - The return on investment.
- **last_updated** (string) - The last updated timestamp.

#### Response Example
```json
[
  {
    "id": "bitcoin",
    "symbol": "btc",
    "name": "Bitcoin",
    "image": "https://assets.coingecko.com/coins/images/1/large/bitcoin.png?1547033579",
    "current_price": 17154.23,
    "market_cap": 329331527834,
    "market_cap_rank": 1,
    "fully_diluted_valuation": 359638803581,
    "total_volume": 14339246353,
    "high_24h": 17227.56,
    "low_24h": 17107.75,
    "price_change_24h": 18.57,
    "price_change_percentage_24h": 0.10839,
    "market_cap_change_24h": -425260465.15130615,
    "market_cap_change_percentage_24h": -0.12896,
    "circulating_supply": 19230300.0,
    "total_supply": 21000000.0,
    "max_supply": 21000000.0,
    "ath": 69045,
    "ath_change_percentage": -75.16662,
    "ath_date": "2021-11-10T14:24:11.849Z",
    "atl": 67.81,
    "atl_change_percentage": 25185.95387,
    "atl_date": "2013-07-06T00:00:00.000Z",
    "roi": null,
    "last_updated": "2022-12-11T00:32:01.695Z"
  }
]
```
```

--------------------------------

### Basic IO Config File Structure - YAML

Source: https://docs.mage.ai/development/io_config_setup

Defines the fundamental structure of the io_config.yaml file. It includes an optional version field and a default profile section for configurations.

```yaml
version: 0.1.1  # Optional but recommended
default:
  # Your default profile configuration here
```

--------------------------------

### Stage and Commit Mage Project Files

Source: https://docs.mage.ai/migrations/dbt-to-mage-pro

Stages all project files, commits them with a message, and pushes the changes to the 'main' branch of the origin repository. This is a standard Git workflow for initializing a project.

```bash
git add .
git commit -m "Initial Mage Pro project setup"
git push origin main
```

--------------------------------

### Start PostgreSQL Docker Container for Mage AI

Source: https://docs.mage.ai/development/docker/connecting-a-database

This command starts a PostgreSQL Docker container, connecting it to the `mage-app` network. It exposes the default PostgreSQL port and mounts a volume for data persistence. Replace placeholders with your desired username, password, and database name.

```bash
docker run --network mage-app --network-alias postgres_db \
   -it -p 5432:5432 -v pgdata:/var/lib/postgresql/data \
   -e POSTGRES_USER=<username> -e POSTGRES_PASSWORD=<password> \
   -e POSTGRES_DB=<database> -e PG_DATA=/var/lib/postgresql/data/pgdata \
   postgres:13-alpine3.17 postgres

```

--------------------------------

### Test Custom Mage Docker Image

Source: https://docs.mage.ai/production/ci-cd/local-cloud/repository-setup

Runs the newly built custom Docker image ('mageprod:latest') to start a Mage project named 'demo_project'. This command verifies that the Docker image was built successfully and the Mage environment is operational.

```bash
docker run -it -p 6789:6789 mageprod:latest /app/run_app.sh mage start demo_project
```

--------------------------------

### Google Cloud Pub/Sub Emulator Setup Commands (Shell)

Source: https://docs.mage.ai/guides/streaming/destinations/google-cloud-pubsub

Provides shell commands to set up and run the Google Cloud Pub/Sub emulator locally. This includes starting the emulator with a specific project ID and initializing environment variables for its use.

```shell
gcloud beta emulators pubsub start --project=test-project-id

$(gcloud beta emulators pubsub env-init)
```

--------------------------------

### Configure and Run Mage with dbt (Windows Batch)

Source: https://docs.mage.ai/guides/dbt/setup-dbt

This script clones the dbt quickstart repository, copies the `dev.env` file to `.env`, removes the original `dev.env`, and then starts Mage and a Postgres database using Docker Compose. This is tailored for Windows environments.

```bash
git clone https://github.com/mage-ai/dbt-quickstart mage-dbt-quickstart
cd mage-dbt-quickstart
cp dev.env .env
rm dev.env
docker compose up
```

--------------------------------

### Start MSSQL Docker Container for Mage AI

Source: https://docs.mage.ai/development/docker/connecting-a-database

This command starts a Microsoft SQL Server (MSSQL) Docker container, connecting it to the `mage-app` network. It maps the default MSSQL port and sets the necessary environment variables for accepting the EULA and setting the SA password. Use a strong password for production environments.

```bash
docker run --network mage-app --network-alias mssql_db \
-it -p 1433:1433 -e "ACCEPT_EULA=Y" \
-e "MSSQL_SA_PASSWORD=test_Password123" \
mcr.microsoft.com/mssql/server:2022-latest

```

--------------------------------

### Create and Configure dbt profiles.yml

Source: https://docs.mage.ai/migrations/dbt-to-mage-pro

Creates a 'profiles.yml' file within the dbt project directory and provides an example configuration for connecting to BigQuery using a service account. This file is crucial for dbt to connect to your data warehouse.

```bash
touch profiles.yml
```

```yaml
mage_dbt_demos:
  outputs:
    dev:
      dataset: BigQuery dataset
      job_execution_timeout_seconds: 300
      job_retries: 1
      keyfile: /home/src/<project-name>/<file-name.json>
      location: US
      method: service-account
      priority: interactive
      project: CGP_PROJECT_ID
      threads: 4
      type: bigquery
  target: dev
```

--------------------------------

### Load dbt Dependencies with `dbt deps`

Source: https://docs.mage.ai/migrations/dbt-to-mage-pro

Run the `dbt deps` command to install dependencies declared in your `packages.yml` file. This is crucial when migrating dbt projects from sources like GitHub. The command fetches and installs all required packages, creating a `dbt_packages` directory.

```shell
dbt deps
```

--------------------------------

### React ProBanner Component for Mage.ai

Source: https://docs.mage.ai/getting-started/setup

This React component displays a promotional banner for Mage Pro, offering a free trial or direct sign-up. It includes customizable titles, descriptions, and button labels, linking to the Mage Pro sign-up page.

```javascript
export const ProBanner = ({button = 'Try Mage Pro for free', description, source = 'documentation', title = 'Our fully managed solution for teams is now available!'}) => <a href={`https://cloud.mage.ai/sign-up?source=${source}`} className="block my-4 px-5 py-4 overflow-hidden rounded-xl flex gap-3 border border-emerald-500/20 bg-emerald-50/50 dark:border-emerald-500/30 dark:bg-emerald-500/10" target="_blank">
    <div style={{
  display: 'flex',
  alignItems: 'center',
  width: '100%'
}}>
      <div className="text-sm prose min-w-0 text-emerald-900 dark:text-emerald-200" style={{
  flex: 1
}}>
        {title}
        {description && <br />}
        {description && <p className="normal">{description}</p>}
      </div>

      <div> </div>

      <div>
        <ProButton label={button} href={`https://cloud.mage.ai/sign-up?source=${source}`} />
      </div>
    </div>
  </a>;
```

--------------------------------

### Start Google Cloud Pub/Sub Emulator

Source: https://docs.mage.ai/streaming/sources/google-cloud-pubsub

Commands to start the Google Cloud Pub/Sub emulator locally. This involves starting the emulator with a specified project ID and initializing the environment variables for the emulator.

```bash
gcloud beta emulators pubsub start --project=test-project-id

$(gcloud beta emulators pubsub env-init)

```

--------------------------------

### Python: Load Data with Dynamic Profile Selection

Source: https://docs.mage.ai/development/io_config_setup

Demonstrates dynamic profile selection in Python, where the IO configuration profile can be determined at runtime, for example, from pipeline variables. It defaults to 'default' if no profile is provided.

```python
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

@data_loader
def load_data_from_postgres(**kwargs):
    query = 'SELECT * FROM your_table'
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    
    # Get profile from pipeline variables or use default
    config_profile = kwargs.get('profile', 'default')
    
    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

--------------------------------

### Legacy IO Config Format - YAML

Source: https://docs.mage.ai/development/io_config_setup

Presents the verbose, legacy format for io_config.yaml. This structure groups configurations by service type, such as PostgreSQL and AWS, with nested Redshift configuration under AWS.

```yaml
default:
  PostgreSQL:
    database: your_database
    user: your_username
    password: your_password
    host: your_host
    port: 5432
  AWS:
    region: your_region
    Redshift:
      database: your_redshift_database
      port: 5439
```

--------------------------------

### Load Secrets from Local Files (Mage Pro)

Source: https://docs.mage.ai/development/io_config_setup

Illustrates loading sensitive data directly from local files using the `file` function. This also includes an example of extracting JSON values from a file. This is a Mage Pro feature.

```yaml
version: 0.1.1
default:
  # Load from local file
  POSTGRES_PASSWORD: "{{ file('secrets/postgres_password.txt') }}"
  # Load JSON field from file
  SNOWFLAKE_PASSWORD: "{{ json_value(file('config/database.json'), 'snowflake_password') }}"
```

--------------------------------

### Import ConfigFileLoader in Python

Source: https://docs.mage.ai/development/io_config_setup

Demonstrates the correct way to import the ConfigFileLoader class from the mage_ai.io.config module in Python. This is essential for loading configuration files within Mage AI projects.

```python
from mage_ai.io.config import ConfigFileLoader
```

--------------------------------

### Define Multiple IO Configuration Profiles

Source: https://docs.mage.ai/development/io_config_setup

Shows how to define multiple profiles within an `io_config.yaml` file for different environments like development, staging, and production. This allows for flexible configuration management.

```yaml
version: 0.1.1
default:
  # Default profile configuration
  POSTGRES_DBNAME: production_db
development:
  # Development environment configuration
  POSTGRES_DBNAME: development_db
staging:
  # Staging environment configuration
  POSTGRES_DBNAME: staging_db
```

--------------------------------

### Install ESLint Global Dependencies

Source: https://docs.mage.ai/contributing/frontend/overview

Installs necessary ESLint packages globally for local development. This setup ensures consistent code linting across the project.

```bash
yarn global add eslint-config-next
yarn global add eslint_d
```

--------------------------------

### Standard IO Config Format with Database Credentials - YAML

Source: https://docs.mage.ai/development/io_config_setup

Illustrates the recommended standard format for io_config.yaml, showing how to configure credentials for PostgreSQL, Snowflake, and AWS. It utilizes environment variables for sensitive information like passwords.

```yaml
version: 0.1.1
default:
  # PostgreSQL configuration
  POSTGRES_DBNAME: your_database
  POSTGRES_USER: your_username
  POSTGRES_PASSWORD: "{{ env_var('POSTGRES_PASSWORD') }}"
  POSTGRES_HOST: your_host
  POSTGRES_PORT: 5432

  # Snowflake configuration
  SNOWFLAKE_USER: your_username
  SNOWFLAKE_PASSWORD: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
  SNOWFLAKE_ACCOUNT: your_account
  SNOWFLAKE_DEFAULT_WH: your_warehouse
  SNOWFLAKE_DEFAULT_DB: your_database
  SNOWFLAKE_DEFAULT_SCHEMA: your_schema

  # AWS configuration
  AWS_ACCESS_KEY_ID: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
  AWS_SECRET_ACCESS_KEY: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
  AWS_REGION: your_region
```

--------------------------------

### Start Kafka Docker Compose - Bash

Source: https://docs.mage.ai/guides/streaming/tutorials/streaming-pipeline

Commands to start and manage the Kafka Docker environment using docker-compose. It includes commands to bring up the services, execute commands within the Kafka container, and create/list Kafka topics. Ensure you have cloned the kafka-docker repository and have Docker installed.

```bash
docker-compose up

```

```bash
docker compose up

```

```bash
docker exec -i -t -u root $(docker ps | grep docker_kafka | cut -d' ' -f1) /bin/bash

```

```bash
$KAFKA_HOME/bin/kafka-topics.sh --create --partitions 4 --bootstrap-server kafka:9092 --topic test

```

```bash
$KAFKA_HOME/bin/kafka-topics.sh --bootstrap-server kafka:9092 --list

```

```bash
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list kafka:9092 --topic=test

```

```bash
>{ "hello": 1 }

```

```bash
>{ "this is a test": 1 }

```

```bash
>{ "test": 1 }

```

```bash
>{ "test": 2 }

```

```bash
>{ "test": 3 }

```

```bash
$KAFKA_HOME/bin/kafka-console-consumer.sh --from-beginning --bootstrap-server kafka:9092 --topic=test

```

```bash
{ "hello": 1 }

```

```bash
{ "this is a test": 1 }

```

```bash
{ "test": 1 }

```

```bash
{ "test": 2 }

```

```bash
{ "test": 3 }

```

--------------------------------

### POST /users API Example

Source: https://docs.mage.ai/data-integrations/sources/api

This example demonstrates a POST request to create a user, including payload, headers, and response parsing.

```APIDOC
## POST /users

### Description
This endpoint creates a new user. It requires a JSON payload with user details and authentication headers.

### Method
POST

### Endpoint
https://api.something.com/users

### Parameters
#### Request Body
- **user** (object) - Required - Contains user information.
  - **first_name** (string) - Required - The first name of the user.
  - **power** (integer) - Required - A power level associated with the user.

#### Headers
- **Content-Type** (string) - Required - Specifies the content type of the request, typically 'application/json'.
- **token** (string) - Required - Authentication token.

### Request Example
```yaml
{
  "url": "https://api.something.com/users",
  "method": "POST",
  "payload": {
    "user": {
      "first_name": "Urza",
      "power": 10
    }
  },
  "headers": {
    "Content-Type": "application/json",
    "token": "abc123"
  },
  "response_parser": "user"
}
```

### Response
#### Success Response (201)
- **user** (object) - The created user object, including any server-generated fields.

#### Response Example
```json
{
  "user": {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "first_name": "Urza",
    "power": 10,
    "created_at": "2023-10-27T10:00:00Z"
  }
}
```
```

--------------------------------

### Python: Load Data with Environment-based Profile Selection

Source: https://docs.mage.ai/development/io_config_setup

Illustrates how to automatically select an IO configuration profile in Python based on an environment variable (e.g., 'ENV'). This allows for seamless switching between development, staging, and production configurations.

```python
import os
from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.postgres import Postgres
from os import path

@data_loader
def load_data(**kwargs):
    query = 'SELECT * FROM your_table'
    config_path = path.join(get_repo_path(), 'io_config.yaml')

    # Automatically select profile based on environment
    environment = os.getenv('ENV', 'development')
    config_profile = environment  # 'development', 'staging', 'production'

    with Postgres.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
        return loader.load(query)
```

--------------------------------

### Standard Configuration File Example (YAML)

Source: https://docs.mage.ai/design/data-loading

An example of a standard format for `io_config.yaml` files. This format uses a subset of `ConfigKey` and organizes settings into profiles.

```yaml
version: 0.1.0
default:
    AWS_ACCESS_KEY_ID: AWS Access Key ID credential
    AWS_SECRET_ACCESS_KEY: AWS Secret Access Key credential
    AWS_REGION: AWS Region
    REDSHIFT_DBNAME: Name of Redshift database to connect to
    REDSHIFT_HOST: Redshift Cluster hostname
    REDSHIFT_PORT: Redshift Cluster port. Optional, defaults to 5439
    REDSHIFT_TEMP_CRED_USER: Redshift temp credentials username
    REDSHIFT_TEMP_CRED_PASSWORD: Redshift temp credentials password
```

--------------------------------

### Load Environment Variables for Secrets

Source: https://docs.mage.ai/development/io_config_setup

Demonstrates how to load sensitive information like passwords from environment variables using `env_var`. This is a common practice for managing credentials securely.

```yaml
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ env_var('POSTGRES_PASSWORD') }}"
  SNOWFLAKE_PASSWORD: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
```

--------------------------------

### Dockerfile for Mage Project Customization

Source: https://docs.mage.ai/production/ci-cd/local-cloud/repository-setup

A Dockerfile template to build a custom Mage Docker image. It sets up the project environment, copies project files, installs dependencies from requirements.txt, and configures environment variables. Replace '[project_name]' with your actual project name.

```dockerfile
FROM mageai/mageai:latest

ARG PROJECT_NAME=[project_name]
ARG MAGE_CODE_PATH=/home/mage_code
ARG USER_CODE_PATH=${MAGE_CODE_PATH}/${PROJECT_NAME}

WORKDIR ${MAGE_CODE_PATH}

# Replace [project_name] with the name of your project (e.g. demo_project)
COPY ${PROJECT_NAME} ${PROJECT_NAME}

# Set the USER_CODE_PATH variable to the path of user project.
# The project path needs to contain project name.
# Replace [project_name] with the name of your project (e.g. demo_project)
ENV USER_CODE_PATH=${USER_CODE_PATH}

# Install custom Python libraries
RUN pip3 install -r ${USER_CODE_PATH}/requirements.txt
# Install custom libraries within 3rd party libraries (e.g. dbt packages)
RUN python3 /app/install_other_dependencies.py --path ${USER_CODE_PATH}

ENV PYTHONPATH="${PYTHONPATH}:/home/mage_code"

CMD ["/bin/sh", "-c", "/app/run_app.sh"]
```

--------------------------------

### Clone dbt Project into Mage

Source: https://docs.mage.ai/migrations/dbt-to-mage-pro

Navigates to the 'dbt' directory within the Mage environment and clones a specified dbt project from a GitHub repository. This integrates an existing dbt project into Mage.

```bash
cd dbt
git clone https://github.com/your-org/your-dbt-project.git
cd your-dbt-project
```

--------------------------------

### Make API Calls to List Pipelines

Source: https://docs.mage.ai/extensibility/api-reference/overview

This example demonstrates how to make a GET request to the Mage AI API to retrieve a list of all pipelines. It utilizes a configured session and constructs the correct API endpoint URL.

```python
pipelines_url = base_url + '/api/pipelines'
r = s.get(pipelines_url)
list_pipelines = r.json()

print(list_pipelines)

```

--------------------------------

### Verbose Configuration File Example (YAML)

Source: https://docs.mage.ai/design/data-loading

An example of a verbose format for `io_config.yaml` files, used for backwards compatibility. This format organizes settings by data migration client.

```yaml
version: 0.0.0
default:
    AWS:
        Redshift:
            database: Name of Redshift database to connect to
            host: Redshift Cluster hostname
            port: Redshift Cluster port. Optional, defaults to 5439
            user: Redshift temp credentials username
            password: Redshift temp credentials password
        access_key_id: AWS Access Key ID credential
        secret_access_key: AWS Secret Access Key credential
        region: AWS Region
```

--------------------------------

### Comprehensive AI Block Configuration in Mage

Source: https://docs.mage.ai/ai/blocks

A detailed configuration for an AI block in Mage, demonstrating various options including prompt definition, output settings (code generation with validation, or structured JSON schema), and tool configuration for triggering other pipelines. This example illustrates the full capabilities of AI block setup.

```yaml
prompt: |
  Enter what you want AI to do
output:
  # Choose either `code` or `format`, not both
  code:
    save: true
    validation: Ensure this code runs without errors
  format:
    name:
      type: string
      description: The name of the person
    age:
      type: integer
      minimum: 0
validation: Validation prompt for output
tools:
  prompt: A prompt to decide which pipeline(s) to trigger
  required: false
  pipelines:
    - uuid: pipeline_uuid
      description: Description of the pipeline
      blocks:
        - uuid: block_uuid
      variables:
        user_id:
          type: string
```

--------------------------------

### Build Docker Image and Run Mage Services

Source: https://docs.mage.ai/contributing/development-environment

This bash script builds the development Docker image and starts all necessary Mage AI services. It's designed for a full development setup. Use the alternative script if only working on the backend.

```bash
./scripts/dev.sh [project_name]
```

--------------------------------

### Start Mage with Spark Docker Container

Source: https://docs.mage.ai/integrations/compute/spark-pyspark

Starts a Mage AI instance using a pre-built Docker container with Spark support. This command maps ports, mounts the current directory, and specifies the project to start.

```bash
docker run -it --name mage_spark -p 6789:6789 -v $(pwd):/home/src mage_spark \
  /app/run_app.sh mage start demo_project
```

--------------------------------

### Installing Mage AI

Source: https://docs.mage.ai/getting-started/runtime-variable

Command to install the Mage AI package using pip. This is a prerequisite for running pipelines from the command line.

```bash
pip install mage-ai
```

--------------------------------

### Amplitude Source Configuration Example

Source: https://docs.mage.ai/guides/data-integration-pipeline

Example YAML configuration for connecting to an Amplitude data source. It includes placeholders for credentials and demonstrates environment variable interpolation for sensitive information like passwords.

```yaml
account: ...
database: ...
password: "{{ env_var('PASSWORD') }}"
schema: ...
table: ...
username: ...
warehouse: ...
```

--------------------------------

### Initialize Dockerfile (Bash)

Source: https://docs.mage.ai/development/project/setup

Creates an empty Dockerfile. This file will later be populated with content to define the Docker image for your Mage project.

```bash
echo "" > Dockerfile
```

--------------------------------

### Add Qdrant Dependencies to requirements.txt

Source: https://docs.mage.ai/integrations/databases/Qdrant

This snippet lists the necessary libraries to install for Qdrant integration. Add these lines to your project's `requirements.txt` file and install them to ensure Qdrant functionality.

```text
qdrant-client==1.6.9
sentence-transformers==2.2.2
```

--------------------------------

### Start Mage Backend Services Only

Source: https://docs.mage.ai/contributing/development-environment

This bash script is optimized for backend development by starting only the essential backend services. This can improve performance when frontend development is not required.

```bash
./scripts/start.sh [project_name]
```

--------------------------------

### GET API Request without Response Parsing (JSON)

Source: https://docs.mage.ai/data-integrations/sources/api

This example demonstrates a GET request to the CoinGecko API to retrieve market data for cryptocurrencies. Since no `response_parser` is specified, the entire API response is returned. The `columns` configuration is not needed here because the response items are already JSON objects.

```json
{
  "theme": {"system"}
  "url": "https://api.coingecko.com/api/v3/coins/markets",
  "query": {
    "vs_currency": "usd"
  }
}
```

--------------------------------

### Create Mage Project Directory (Bash)

Source: https://docs.mage.ai/development/project/setup

Creates a new directory to serve as the root for your Mage project. This is the first step in organizing your project files.

```bash
mkdir data_monorepo
```

--------------------------------

### Start Mage AI Project

Source: https://docs.mage.ai/integrations/airflow/integrate-mage-airflow

Commands to start the Mage AI application within your DAGs folder. This makes the Mage UI accessible at http://localhost:6789, allowing you to create and manage pipelines. Separate commands are provided for Docker and pip installations.

```bash
docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai /app/run_app.sh mage start demo_project
```

```bash
mage start demo_project
```

--------------------------------

### Initialize Terraform Project

Source: https://docs.mage.ai/production/deploying-to-cloud/azure/setup

Initializes the Terraform working directory. This command downloads necessary provider plugins and sets up the backend configuration. It's a prerequisite for other Terraform commands.

```bash
cd azure
terraform init
```

--------------------------------

### Install SparkMagic and Configure for PySpark Kernel

Source: https://docs.mage.ai/integrations/compute/spark-pyspark

Installs the SparkMagic library and configures it for use with a PySpark kernel in Mage AI. This involves downloading an example configuration file and modifying it to point to the correct Spark host and port. Ensure you are in the correct directory before running these commands.

```bash
pip install sparkmagic
mkdir ~/.sparkmagic
wget https://raw.githubusercontent.com/jupyter-incubator/sparkmagic/master/sparkmagic/example_config.json
mv example_config.json ~/.sparkmagic/config.json
sed -i 's/localhost:8998/host.docker.internal:9999/g' ~/.sparkmagic/config.json
```

--------------------------------

### Terraform Initialization

Source: https://docs.mage.ai/production/deploying-to-cloud/digitalocean

Initializes the Terraform working directory. This command downloads the necessary provider plugins, such as the DigitalOcean provider, and sets up the backend configuration.

```shell
cd digitalocean
terraform init
```

--------------------------------

### GET API Request with Response Parsing (JSON)

Source: https://docs.mage.ai/data-integrations/sources/api

This example shows a GET request to the PLOS API to search for articles containing 'DNA' in their title. It utilizes a `response_parser` to extract specific fields (author_display) from the nested JSON response. The `columns` configuration is used when the parsed data is not a JSON object, ensuring proper formatting of the final output.

```json
{
  "theme": {"system"}
  "url": "https://api.plos.org/search",
  "query": {
    "q": "title:DNA"
  },
  "headers": {
    "Content-Type": "application/json"
  },
  "response_parser": "response.docs[0].author_display",
  "columns": [
    "author_display"
  ]
}
```

--------------------------------

### GET /api/pipelines/:pipeline_uuid/logs

Source: https://docs.mage.ai/extensibility/api-reference/logs/read-logs

Retrieves logs for a specific pipeline run. Supports filtering by the number of logs to return, an offset for pagination, and a start timestamp.

```APIDOC
## GET /api/pipelines/:pipeline_uuid/logs

### Description
Retrieves logs for a specific pipeline run. Supports filtering by the number of logs to return, an offset for pagination, and a start timestamp.

### Method
GET

### Endpoint
/api/pipelines/:pipeline_uuid/logs

### Parameters
#### Path Parameters
- **pipeline_uuid** (string) - Required - Pipeline UUID of the logs.

#### Query Parameters
- **_limit** (integer) - Optional - Maximum number of logs to be returned in the API response.
- **_offset** (integer) - Optional - Read logs after N number of logs, where N equals `_offset`.
- **start_timestamp** (integer) - Optional - Read logs on or after a specified timestamp.

### Request Example
```json
{
  "example": "curl --request GET \
    --url 'http://localhost:6789/api/pipelines/example_pipeline/logs?_limit=20&_offset=0&start_timestamp=1677648116&api_key=zkWlN0PkIKSN0C11CfUHUj84OT5XOJ6tDZ6bDRO2' \
    --header 'OAUTH-TOKEN=some_really_long_string'"
}
```

### Response
#### Success Response (200)
- **logs** (array) - An array containing log entries.
  - **block_run_logs** (array) - Logs for individual blocks within a pipeline run.
    - **name** (string) - Name of the block log file.
    - **path** (string) - Path to the block log file.
    - **content** (string) - Content of the block log.
  - **pipeline_run_logs** (array) - Logs for the overall pipeline run.
    - **name** (string) - Name of the pipeline log file.
    - **path** (string) - Path to the pipeline log file.
    - **content** (string) - Content of the pipeline log.
  - **total_block_run_log_count** (integer) - Total count of block run logs.
  - **total_pipeline_run_log_count** (integer) - Total count of pipeline run logs.
- **metadata** (object) - Metadata associated with the response.

#### Response Example
```json
{
  "logs": [
    {
      "block_run_logs": [
        {
          "name": "solitary_brook.log",
          "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/solitary_brook.log",
          "content": "very_long_content"
        },
        {
          "name": "load_titanic.log",
          "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/54/20230308T045259/load_titanic.log",
          "content": "very_long_content"
        }
      ],
      "pipeline_run_logs": [
        {
          "name": "pipeline.log",
          "path": "/root/.mage_data/default_repo/pipelines/example_pipeline/.logs/55/20230308T045408/pipeline.log",
          "content": "very_long_content \"timestamp\": 1678251278.430712, \"uuid\": \"3de507ebca894100b20ae4a1645724d7\"}\\n"
        },
        {
          "name": "pipeline.log",
          "path": "very_long_content"
        }
      ],
      "total_block_run_log_count": 2,
      "total_pipeline_run_log_count": 2
    }
  ],
  "metadata": {}
}
```
```

--------------------------------

### Mage AI Cursor-Based Pagination (Single Cursor YAML)

Source: https://docs.mage.ai/data-integrations/sources/api

Configures cursor-based pagination for API syncing when the API returns a next cursor or token in the response body. This example shows a single cursor parameter setup.

```yaml
pagination_config:
  type: cursor
  cursor_param: next
  initial_cursor_value: null
  next_cursor_path: meta.next_cursor
```

--------------------------------

### Create .gitignore File (Bash)

Source: https://docs.mage.ai/development/project/setup

Initializes an empty .gitignore file. This file specifies intentionally untracked files that Git should ignore, preventing them from being committed to the repository.

```bash
echo "" > .gitignore
```

--------------------------------

### Load Mage Secrets for Sensitive Data

Source: https://docs.mage.ai/development/io_config_setup

Shows how to retrieve secrets stored within Mage AI using `mage_secret_var`. This is useful for secrets managed directly by Mage.

```yaml
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ mage_secret_var('postgres_password') }}"
  SNOWFLAKE_PASSWORD: "{{ mage_secret_var('snowflake_password') }}"
```

--------------------------------

### Navigate to Project Directory (Bash)

Source: https://docs.mage.ai/development/project/setup

Changes the current working directory to the newly created Mage project parent folder. This allows subsequent commands to operate within the project's root.

```bash
cd data_monorepo
```

--------------------------------

### Generate dbt Documentation and Dependencies

Source: https://docs.mage.ai/guides/dbt/docs

This bash script executes two dbt commands to generate documentation. `dbt deps` installs any necessary package dependencies, and `dbt docs generate` creates the static documentation files for the dbt project.

```bash
dbt deps
dbt docs generate
```

--------------------------------

### Load AWS Secrets Manager Credentials

Source: https://docs.mage.ai/development/io_config_setup

Illustrates fetching secrets from AWS Secrets Manager using `aws_secret_var`. This allows integration with AWS's secret management service.

```yaml
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ aws_secret_var('prod/postgres/password') }}"
  SNOWFLAKE_PASSWORD: "{{ aws_secret_var('prod/snowflake/password') }}"
```

--------------------------------

### Kafka Data Formatting - Full Message Example

Source: https://docs.mage.ai/guides/streaming/destinations/kafka

Demonstrates a comprehensive Kafka message structure, including both 'data' and 'metadata' fields. This example shows how to specify a destination topic, a key for partitioning, and a precise timestamp.

```python
message = {
    'data': {
        'bees': 23,
        'ants': 30,
    },
    'metadata': {
        'dest_topic': 'census',
        'key': 'key',
        'time': 1693396630163,  # timestamp with ms precision
    }
}
```

--------------------------------

### Restart PostgreSQL Service (Bash)

Source: https://docs.mage.ai/data-integrations/sources/postgresql

Commands to stop and start the PostgreSQL service using bash, typically required after modifying the configuration file to apply changes. This is common on Linux systems.

```bash
sudo service postgresql stop
sudo service postgresql start
```

--------------------------------

### Load Google Cloud Secret Manager Secrets (Mage Pro)

Source: https://docs.mage.ai/development/io_config_setup

Shows how to access secrets from Google Cloud Secret Manager using `gcp_secret_var`. This feature is available in Mage Pro.

```yaml
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ gcp_secret_var('postgres-password') }}"
  SNOWFLAKE_PASSWORD: "{{ gcp_secret_var('snowflake-password') }}"
```

--------------------------------

### Load Azure Key Vault Secrets

Source: https://docs.mage.ai/development/io_config_setup

Demonstrates retrieving secrets from Azure Key Vault using `azure_secret_var`. This integrates Mage AI with Azure's secret management capabilities.

```yaml
version: 0.1.1
default:
  POSTGRES_PASSWORD: "{{ azure_secret_var('postgres-password') }}"
  SNOWFLAKE_PASSWORD: "{{ azure_secret_var('snowflake-password') }}"
```

--------------------------------

### Install Dependencies using pip

Source: https://docs.mage.ai/ai/ml/train-model

Installs project dependencies listed in requirements.txt using pip. This is a common step before running any Python project to ensure all necessary libraries are available.

```bash
pip install -r demo_project/requirements.txt
```

--------------------------------

### FileIO Client Operation Output

Source: https://docs.mage.ai/design/data-loading

Example output from the FileIO client when initialized with `verbose=True`, showing the steps involved in loading data from a URL and exporting it to a file.

```bash
FileIO initialized
├─ Loading data frame from 'https://raw.githubusercontent.com/datasciencdojodatasets/master/titanic.csv'...DONE
└─ Exporting data frame to './titanic_survival.json'...DONE
```

--------------------------------

### Run MongoDB Locally with Docker

Source: https://docs.mage.ai/data-integrations/sources/mongodb

This command starts a local MongoDB instance using Docker. It maps the default MongoDB port (27017) to the host machine and sets root credentials for authentication. Ensure Docker is installed and running.

```bash
docker run -p 27017:27017 \
  -e MONGO_INITDB_ROOT_USERNAME=admin \
  -e MONGO_INITDB_ROOT_PASSWORD=password \
  --name mongo \
  mongo:latest
```

--------------------------------

### MySQL Connector Setup to Resume from Position (YAML)

Source: https://docs.mage.ai/guides/streaming/sources/mysql-cdc

A YAML configuration for a MySQL connector that specifies the starting log file and position to resume replication from. This is useful for restarting replication after an interruption without losing data.

```yaml
connector_type: mysql
host: localhost
port: 3306
user: repl_user
password: password
start_log_file: "mysql-bin.000123"
start_log_pos: 456789
databases: ["mydb"]
return_db_records_only: true
```

--------------------------------

### DuckDB Export Logs

Source: https://docs.mage.ai/guides/load-api-data

Example log output showing the process of initializing DuckDB, opening a connection, and exporting data from an upstream block to a table. This indicates the successful completion of the data export step.

```bash
DuckDB initialized
└─ Opening connection to DuckDB...DONE
Exporting data from upstream block floral_star to restaurant_user_transactions.
Type: integer
Type: integer
Type: string
Type: integer
Type: string
Type: integer
Type: string
Type: integer
Type: integer
Type: integer

```

--------------------------------

### CLI Terraform Deployment for Mage AI

Source: https://docs.mage.ai/production/deploying-to-cloud/aws

Commands to initialize, troubleshoot, and apply Terraform for deploying Mage AI via CLI. Includes steps for resolving provider installation errors and the actual deployment command.

```bash
cd aws
terraform init

# To resolve provider installation errors:
brew install kreuzwerker/taps/m1-terraform-provider-helper
m1-terraform-provider-helper activate
m1-terraform-provider-helper install hashicorp/template -v v2.2.0
rm .terraform.lock.hcl
terraform init --upgrade

# To deploy:
terraform apply
```

--------------------------------

### Initialize Terraform for GCP Deployment

Source: https://docs.mage.ai/production/deploying-to-cloud/gcp

This command initializes a Terraform working directory. It downloads the necessary provider plugins (in this case, hashicorp/google and hashicorp/http) and sets up the backend for state management. This is a prerequisite before applying any Terraform configurations.

```bash
terraform init
```
Tokens:
Raw