Popsink

https://docs.popsink.com
Admin
Popsink is a real-time data integration platform that enables automated capture, transformation, and...
Tokens:39,186
Snippets:480
Trust Score:7.1
Update:6 months ago
Show doc for...
Tokens:
Raw
### Full Popsink Client Workflow Example

Source: https://docs.popsink.com/on-prem/on-prem

Demonstrates a complete workflow using the PopsinkClient, including initialization, user registration, login, environment and team creation, pipeline setup, and pipeline execution.

```python
def main():
    # Initialize client
    client = PopsinkClient(SERVER, verify_ssl=VERIFY_SSL)

    # 1. Register
    print("1. Registering user...")
    user = client.register("demo@example.com", "SecurePassword123!")
    print(f"   ✓ User created: {user['id']}")

    # 2. Login
    print("\n2. Logging in...")
    token = client.login("demo@example.com", "SecurePassword123!")
    print(f"   ✓ Logged in (token: {token[:20]}...)")

    # 3. Create environment
    print("\n3. Creating environment...")
    env = client.create_environment("demo-env", "Demo environment")
    print(f"   ✓ Environment created: {env['id']}")

    # 4. Create team
    print("\n4. Creating team...")
    team = client.create_team("Demo Team", "Demo team for testing")
    print(f"   ✓ Team created: {team['id']}")

    # 5. Create pipeline
    print("\n5. Creating pipeline...")
    pipeline_config = {
        "source_name": "demo-kafka-source",
        "source_type": "KAFKA_SOURCE",
        "source_config": {
            "topic": "demo-topic",
            "bootstrap_servers": "kafka:9092",
            "consumer_group_id": "demo-group",
            "security_protocol": "PLAINTEXT"
        },
        "target_name": "demo-kafka-target",
        "target_type": "KAFKA_TARGET",
        "target_config": {
            "bootstrap_server": "kafka:9092",
            "security_protocol": "PLAINTEXT"
        },
        "subscription_target_table_name": "events",
        "subscription_backfill": False
    }
    pipeline = client.create_pipeline("demo-pipeline", team['id'], pipeline_config)
    print(f"   ✓ Pipeline created: {pipeline['id']}")

    # 6. Start pipeline
    print("\n6. Starting pipeline...")
    state = client.start_pipeline(pipeline['id'])
    print(f"   ✓ Pipeline state: {state}")
```

--------------------------------

### Quick Start: User Registration, Login, and API Access

Source: https://docs.popsink.com/on-prem/on-prem

Use these curl commands to quickly register a user, obtain an authentication token via login, and then use that token to access the API. Ensure you replace 'https://your-server' with your actual server address and 'YOUR_TOKEN_HERE' with the token received after login. The '-k' flag is used to bypass SSL certificate verification, which may be necessary in some on-premises setups.

```bash
# 1. Create a user
curl -X POST "https://your-server/api/auth/register" \
  -H "Content-Type: application/json" \
  -d '{"email":"user@example.com","password":"secure123","is_active":true}' -k

# 2. Login to get your token
curl -X POST "https://your-server/api/auth/jwt/login" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d 'username=user@example.com&password=secure123' -k

# 3. Use the token in all subsequent requests
curl -X GET "https://your-server/api/pipelines/" \
  -H "Authorization: Bearer YOUR_TOKEN_HERE" -k
```

--------------------------------

### Start Pipeline

Source: https://docs.popsink.com/on-prem/on-prem

Initiate the deployment of a pipeline. Returns a 202 Accepted status.

```bash
curl -X POST "https://your-server/api/pipelines/$PIPELINE_ID/start" \
  -H "Authorization: Bearer $TOKEN" -k
```

```json
"BUILDING"
```

--------------------------------

### Start Pipeline

Source: https://docs.popsink.com/on-prem/on-prem

Initiates the building and deployment process for a pipeline.

```APIDOC
## POST /api/pipelines/$PIPELINE_ID/start

### Description
Starts the pipeline, transitioning it from a draft or paused state to the building state, and subsequently to live.

### Method
POST

### Endpoint
/api/pipelines/$PIPELINE_ID/start

### Parameters
#### Path Parameters
- **PIPELINE_ID** (string) - Required - The unique identifier of the pipeline to start.

### Response
#### Success Response (202 Accepted)
Returns the current state of the pipeline, which will be "BUILDING".

#### Response Example
```json
"BUILDING"
```
```

--------------------------------

### List Pipelines (Example)

Source: https://docs.popsink.com/on-prem/on-prem

Demonstrates how to list all available pipelines using cURL, HTTPie, Postman, and Python.

```APIDOC
## GET /api/pipelines/

### Description
Retrieves a list of all pipelines accessible to the authenticated user. This endpoint is often used for initial setup or overview.

### Method
GET

### Endpoint
/api/pipelines/

### Parameters
#### Query Parameters
(No query parameters are explicitly defined in the provided text for this endpoint, but common parameters like `limit` or `offset` might be supported.)

### Request Example (cURL)
```bash
export SERVER="https://your-server"
export TOKEN="your-token-here"

curl -X GET "$SERVER/api/pipelines/" \
  -H "Authorization: Bearer $TOKEN" -k
```

### Request Example (HTTPie)
```bash
http GET https://your-server/api/pipelines/ \
  Authorization:"Bearer $TOKEN"
```

### Request Example (Postman)
1. Create a new request.
2. Set the method to GET.
3. Enter the URL: `https://your-server/api/pipelines/`
4. Add a header: `Authorization` with the value `Bearer YOUR_TOKEN`.
5. Click "Send".

### Request Example (Python)
```python
import requests

SERVER = "https://your-server"
TOKEN = "your-token-here"

headers = {
    "Authorization": f"Bearer {TOKEN}"
}

response = requests.get(
    f"{SERVER}/api/pipelines/",
    headers=headers,
    verify=False  # Only for self-signed certificates
)

print(response.json())
```
```

--------------------------------

### Start a Pipeline

Source: https://docs.popsink.com/on-prem/on-prem

Initiate the execution of a specified pipeline. Requires the pipeline ID. The method returns the current state of the pipeline after starting.

```python
def start_pipeline(self, pipeline_id: str) -> str:
    """Start a pipeline"""
    response = requests.post(
        f"{self.server}/api/pipelines/{pipeline_id}/start",
        headers=self._headers(),
        verify=self.verify_ssl
    )
    response.raise_for_status()
    return response.json()
```

--------------------------------

### Pipeline Creation Response

Source: https://docs.popsink.com/on-prem/on-prem

Example JSON response returned after successfully creating a new pipeline.

```json
{
  "id": "5efd3d63-a6eb-4b0c-afe9-6ee1d8f5f820",
  "name": "kafka-to-oracle-pipeline",
  "state": "draft",
  "version": 1,
  "team_id": "881e0fce-5caa-4f5c-82c6-5eb883c47100",
  "team_name": "Data Engineering Team",

  "existing_source_id": null,
  "source_name": "kafka-events-source",
  "source_type": "KAFKA_SOURCE",
  "source_config": { "topic": "user-events", ... },

  "existing_target_id": null,
  "target_name": "oracle-warehouse",
  "target_type": "ORACLE_TARGET",
  "target_config": { "host": "oracle.company.com", ... },

  "subscription_target_table_name": "events",
  "subscription_backfill": false,
  "subscription_mapper_config": [...],

  "created_at": "2025-11-19T10:30:00Z",
  "updated_at": "2025-11-19T10:30:00Z"
}
```

--------------------------------

### Initialize API Script

Source: https://docs.popsink.com/on-prem/on-prem

Boilerplate imports and setup for a Python script interacting with the Popsink API.

```python
#!/usr/bin/env python3
"""
Complete Popsink API example script
"""
import requests
import json
from typing import Optional
```

--------------------------------

### Install Popsink Data-Plane Helm Chart

Source: https://docs.popsink.com/deployment/selfhosted

Install the Popsink data-plane using the provided Helm chart. Ensure you specify the correct release name and chart version.

```bash
helm install release-name oci://ghcr.io/popsink/charts/data-plane --version 0.1.0-alpha.8
```

--------------------------------

### API Response Example

Source: https://docs.popsink.com/on-prem/on-prem

This is an example of a typical JSON response format from the Popsink API, showing resource details like ID, name, and creation timestamp.

```json
{
  "id": "uuid-here",
  "name": "resource-name",
  "created_at": "2025-11-19T10:00:00Z"
}
```

--------------------------------

### Example FQDN for Helm Values

Source: https://docs.popsink.com/deployment/selfhosted

Provide a Fully Qualified Domain Name (FQDN) for your deployment. This value must also be set in the `ingressurl` field of your `values.yaml` file.

```yaml
demo-popsink.tld
```

--------------------------------

### Run tn5250j Terminal Emulator

Source: https://docs.popsink.com/snowflake-ibmi/db2-setup

Launch the tn5250j terminal emulator from the command line. Ensure JAVA is installed and tn5250j.jar is in the current directory.

```shell
java -jar ./tn5250j.jar
```

--------------------------------

### Enable Journaling on Source Tables

Source: https://docs.popsink.com/snowflake-ibmi/journaling.md

Starts journaling for specific physical files (tables) using the created journal.

```text
STRJRNPF POPSINK1/${TABLE} POPSINK1/MY_JOURNAL
```

```text
STRJRNPF POPSINK1/CATEGORIES POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/SUPPLIERS POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/PRODUCTS POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/STOCK POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/PURCHASES POPSINK1/MY_JOURNAL
STRJRNPF POPSINK1/SALES POPSINK1/MY_JOURNAL
```

--------------------------------

### List Pipelines with Filters (Bash)

Source: https://docs.popsink.com/on-prem/on-prem

Examples of using curl to list pipelines with various filters like state, team ID, or search terms. Requires a valid Bearer token for authorization.

```bash
# Get all pipelines in a specific state
curl -X GET "https://your-server/api/pipelines/?state=live" \
  -H "Authorization: Bearer $TOKEN" -k
```

```bash
# Get pipelines for a specific team
curl -X GET "https://your-server/api/pipelines/?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN" -k
```

```bash
# Search pipelines by name
curl -X GET "https://your-server/api/pipelines/?search=kafka" \
  -H "Authorization: Bearer $TOKEN" -k
```

--------------------------------

### Get Pipeline Logs (WebSocket)

Source: https://docs.popsink.com/on-prem/on-prem

Establishes a WebSocket connection to receive real-time logs for a pipeline.

```APIDOC
## WebSocket /api/pipelines/$PIPELINE_ID/logs

### Description
Provides a real-time stream of logs for a specific pipeline via a WebSocket connection. Useful for monitoring pipeline activity and debugging.

### Method
WebSocket

### Endpoint
/api/pipelines/$PIPELINE_ID/logs

### Parameters
#### Path Parameters
- **PIPELINE_ID** (string) - Required - The unique identifier of the pipeline for which to retrieve logs.

#### Headers
- **Authorization** (string) - Required - Bearer token for authentication (e.g., `Bearer YOUR_TOKEN`).

### Request Example (JavaScript)
```javascript
const ws = new WebSocket(
  `wss://your-server/api/pipelines/${PIPELINE_ID}/logs`,
  ['Authorization', `Bearer ${TOKEN}`]
);

ws.onmessage = (event) => {
  console.log('Log:', event.data);
};
```
```

--------------------------------

### Get Pipeline Logs via WebSocket

Source: https://docs.popsink.com/on-prem/on-prem

Establish a WebSocket connection to stream real-time pipeline logs.

```javascript
// WebSocket connection example
const ws = new WebSocket(
  `wss://your-server/api/pipelines/${PIPELINE_ID}/logs`,
  ['Authorization', `Bearer ${TOKEN}`]
);

ws.onmessage = (event) => {
  console.log('Log:', event.data);
};
```

--------------------------------

### Handle API Errors with Requests (Python)

Source: https://docs.popsink.com/on-prem/on-prem

Python example using the requests library to fetch pipeline data and handle potential HTTP or connection errors. Uses verify=False for simplicity, which is not recommended for production.

```python
import requests

try:
    response = requests.get(
        "https://your-server/api/pipelines/",
        headers={"Authorization": f"Bearer {TOKEN}"},
        verify=False
    )
    response.raise_for_status()  # Raises exception for 4xx/5xx
    data = response.json()
except requests.exceptions.HTTPError as e:
    print(f"HTTP Error: {e}")
    print(f"Response: {e.response.text}")
except requests.exceptions.ConnectionError as e:
    print(f"Connection Error: {e}")
except Exception as e:
    print(f"Error: {e}")
```

--------------------------------

### Get Pipelines

Source: https://docs.popsink.com/on-prem/on-prem

Retrieve a list of all pipelines.

```APIDOC
## GET /api/pipelines/

### Description
Retrieve a list of all pipelines accessible to the authenticated user.

### Method
GET

### Endpoint
/api/pipelines/

### Parameters
#### Query Parameters
- **skip** (integer) - Optional - Number of pipelines to skip for pagination.
- **limit** (integer) - Optional - Maximum number of pipelines to return.

### Response
#### Success Response (200)
- **id** (string) - The unique identifier for the pipeline.
- **name** (string) - The name of the pipeline.
- **created_at** (string) - The timestamp when the pipeline was created.

#### Response Example
[
  {
    "id": "pipeline-uuid-1",
    "name": "Data Ingestion Pipeline",
    "created_at": "2025-11-19T10:00:00Z"
  },
  {
    "id": "pipeline-uuid-2",
    "name": "Data Transformation Pipeline",
    "created_at": "2025-11-19T10:05:00Z"
  }
]
```

--------------------------------

### Create a New Environment

Source: https://docs.popsink.com/on-prem/on-prem

Set up a new workspace for your teams and pipelines. This requires authentication and should be done after logging in. The response provides the environment ID.

```bash
curl -X POST "https://your-server/api/envs/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "production-environment",
    "description": "Production data processing environment",
    "use_retention": true
  }' -k
```

--------------------------------

### Create Pipeline with New Connectors

Source: https://docs.popsink.com/on-prem/on-prem

Initializes a new pipeline by defining source and target configurations inline.

```bash
curl -X POST "https://your-server/api/pipelines/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "kafka-to-oracle-pipeline",
    "team_id": "'$TEAM_ID'",

    "source_name": "kafka-events-source",
    "source_type": "KAFKA_SOURCE",
    "source_config": {
      "topic": "user-events",
      "bootstrap_servers": "kafka.company.com:9092",
      "consumer_group_id": "pipeline-consumer",
      "sasl_username": "kafka_user",
      "sasl_password": "kafka_pass",
      "sasl_mechanism": "PLAIN",
      "security_protocol": "SASL_SSL"
    },

    "target_name": "oracle-warehouse",
    "target_type": "ORACLE_TARGET",
    "target_config": {
      "host": "oracle.company.com",
      "port": "1521",
      "database": "ORCL",
      "server_name": "XE",
      "server_id": "oraclesrv01",
      "user": "etl_user",
      "password": "etl_pass"
    },

    "subscription_target_table_name": "events",
    "subscription_backfill": false,
    "subscription_mapper_config": [
      {
        "key": "user_id",
        "path": "$.user_id",
        "cast": "string",
        "primary_key": true,
        "nullable": false
      },
      {
        "key": "event_type",
        "path": "$.event_type",
        "cast": "string",
        "primary_key": false,
        "nullable": false
      }
    ]
  }' -k
```

--------------------------------

### POST /api/envs/

Source: https://docs.popsink.com/on-prem/on-prem

Creates a new environment for teams and pipelines.

```APIDOC
## POST /api/envs/

### Description
Creates a workspace for teams and pipelines.

### Method
POST

### Endpoint
https://your-server/api/envs/

### Parameters
#### Request Body
- **name** (string) - Required - Environment name (must be unique)
- **description** (string) - Optional - Description of the environment's purpose
- **use_retention** (boolean) - Optional - Enable data retention policies (default: false)

### Response
#### Success Response (201)
- **id** (string) - Environment ID
- **name** (string) - Environment name
- **description** (string) - Environment description
- **use_retention** (boolean) - Retention status
- **retention_configuration** (null) - Configuration details

#### Response Example
{
  "id": "d455a981-be17-455e-a03c-4d9c3224db72",
  "name": "production-environment",
  "description": "Production data processing environment",
  "use_retention": true,
  "retention_configuration": null
}
```

--------------------------------

### GET /api/pipelines/

Source: https://docs.popsink.com/on-prem/on-prem

Retrieves a list of pipelines with support for filtering and pagination.

```APIDOC
## GET /api/pipelines/

### Description
Retrieves a list of pipelines. Supports filtering by state, team, or name, and includes pagination parameters.

### Method
GET

### Endpoint
/api/pipelines/

### Parameters
#### Query Parameters
- **state** (string) - Optional - Filter by pipeline state (e.g., 'live')
- **team_id** (string) - Optional - Filter by team identifier
- **search** (string) - Optional - Search pipelines by name
- **skip** (int) - Optional - Number of items to skip for pagination
- **limit** (int) - Optional - Maximum number of items to return
```

--------------------------------

### Create User Account

Source: https://docs.popsink.com/on-prem/on-prem

Registers a new user account. Requires an email and password.

```bash
curl -X POST "https://your-server/api/auth/register" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane.smith@company.com",
    "password": "SecurePass456!",
    "is_active": true
  }' -k
```

```json
{
  "id": "32d4b09d-4527-40d5-923b-e166d85cb459",
  "email": "jane.smith@company.com",
  "is_active": true
}
```

--------------------------------

### Pipeline ID Environment Variable

Source: https://docs.popsink.com/on-prem/on-prem

Example of storing the generated pipeline ID as an environment variable.

```bash
PIPELINE_ID=5efd3d63-a6eb-4b0c-afe9-6ee1d8f5f820
```

--------------------------------

### Create Pipeline with Existing Connectors

Source: https://docs.popsink.com/on-prem/on-prem

Initializes a pipeline by referencing existing connector IDs.

```bash
curl -X POST "https://your-server/api/pipelines/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "reuse-connectors-pipeline",
    "team_id": "'$TEAM_ID'",

    "existing_source_id": "a1b2c3d4-e5f6-4a5b-8c9d-0e1f2a3b4c5d",
    "existing_target_id": "f6e5d4c3-b2a1-4c5d-9e8f-5d4c3b2a1f0e",

    "subscription_target_table_name": "events_copy",
    "subscription_backfill": true
  }' -k
```

--------------------------------

### Initialize Popsink Client

Source: https://docs.popsink.com/on-prem/on-prem

Instantiate the PopsinkClient with server URL and SSL verification setting. Set VERIFY_SSL to False for development or testing environments.

```python
SERVER = "https://your-server"
VERIFY_SSL = False  # Set to True in production

class PopsinkClient:
    def __init__(self, server: str, verify_ssl: bool = True):
        self.server = server.rstrip('/')
        self.verify_ssl = verify_ssl
        self.token: Optional[str] = None
```

--------------------------------

### Register a New User

Source: https://docs.popsink.com/on-prem/on-prem

Use the register method to create a new user account. Requires email and password. Ensure the server URL and SSL verification are correctly configured.

```python
def register(self, email: str, password: str) -> dict:
    """Register a new user"""
    response = requests.post(
        f"{self.server}/api/auth/register",
        json={
            "email": email,
            "password": password,
            "is_active": True
        },
        verify=self.verify_ssl
    )
    response.raise_for_status()
    return response.json()
```

--------------------------------

### Register a New User

Source: https://docs.popsink.com/on-prem/on-prem

Use this endpoint to create a new user account. Ensure the email is unique. The response includes the user's ID, which is needed for subsequent authentication.

```bash
curl -X POST "https://your-server/api/auth/register" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "john.doe@company.com",
    "password": "MySecurePassword123!",
    "is_active": true,
    "is_superuser": false,
    "is_verified": false
  }' -k
```

--------------------------------

### Insert Sample Data

Source: https://docs.popsink.com/snowflake-ibmi/db2-setup

Populates the previously created tables with initial records for testing purposes.

```oracle-sql
INSERT INTO Categories (CategoryName, Description) VALUES
('Electronics', 'Devices and gadgets'),
('Groceries', 'Daily consumables and food items'),
('Stationery', 'Office and school supplies');


INSERT INTO Suppliers (SupplierName, ContactName, Address, Phone, Email) VALUES
('TechSource Ltd.', 'Alice Johnson', '123 Tech Ave', '555-1234', 'alice@techsource.com'),
('FoodWorld Inc.', 'Bob Smith', '789 Market St', '555-5678', 'bob@foodworld.com'),
('OfficePlus', 'Carol Lee', '456 Office Rd', '555-9012', 'carol@officeplus.com');


INSERT INTO Products (ProductName, CategoryID, SupplierID, UnitPrice, ReorderLevel, Discontinued) VALUES
('USB Flash Drive 32GB', 1, 1, 12.99, 20, 0),
('Organic Apples 1kg', 2, 2, 3.49, 50, 0),
('Notebook A4', 3, 3, 1.25, 100, 0);

INSERT INTO Stock (ProductID, QuantityInStock) VALUES
(1, 150),
(2, 300),
(3, 500);

INSERT INTO Purchases (ProductID, SupplierID, Quantity, PurchaseDate, UnitCost) VALUES
(1, 1, 200, DATE('2025-09-10'), 10.00),
(2, 2, 500, DATE('2025-09-15'), 2.50),
(3, 3, 1000, DATE('2025-09-12'), 0.90);

INSERT INTO Sales (ProductID, Quantity, SaleDate, UnitPrice) VALUES
(1, 50, DATE('2025-09-18'), 14.99),
(2, 100, DATE('2025-09-19'), 3.99),
(3, 250, DATE('2025-09-20'), 1.50);
```

--------------------------------

### Handle 422 Validation Errors

Source: https://docs.popsink.com/on-prem/on-prem

Example of a JSON error response structure and a valid request payload for pipeline creation.

```json
{
  "detail": [
    {
      "loc": ["body", "source_name"],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}
```

```json
{
  "name": "my-pipeline",
  "team_id": "uuid-here",
  "source_name": "my-source",
  "source_type": "KAFKA_SOURCE",
  "source_config": {...}
}
```

--------------------------------

### Create Multiple Pipelines in Bulk (Python)

Source: https://docs.popsink.com/on-prem/on-prem

Python script to create multiple pipelines by iterating over a list of configurations and sending POST requests. Assumes SERVER and headers are defined elsewhere.

```python
# Create multiple pipelines
pipeline_configs = [
    {"name": "pipeline-1", "team_id": TEAM_ID, ...},
    {"name": "pipeline-2", "team_id": TEAM_ID, ...},
    {"name": "pipeline-3", "team_id": TEAM_ID, ...}
]

for config in pipeline_configs:
    response = requests.post(
        f"{SERVER}/api/pipelines/",
        headers=headers,
        json=config
    )
    print(f"Created: {response.json()['id']}")
```

--------------------------------

### Create a New Team

Source: https://docs.popsink.com/on-prem/on-prem

Create a new team for organizing resources. Requires a name and optionally a description. Authentication is handled via internal headers.

```python
def create_team(self, name: str, description: str = "") -> dict:
    """Create a team"""
    response = requests.post(
        f"{self.server}/api/teams/",
        headers=self._headers(),
        json={
            "name": name,
            "description": description
        },
        verify=self.verify_ssl
    )
    response.raise_for_status()
    return response.json()
```

--------------------------------

### Get Authentication Headers

Source: https://docs.popsink.com/on-prem/on-prem

Internal helper method to retrieve authorization headers. It requires the user to be logged in, otherwise, it raises a ValueError. The token must be present.

```python
def _headers(self) -> dict:
    """Get headers with auth token"""
    if not self.token:
        raise ValueError("Not logged in. Call login() first.")
    return {
        "Authorization": f"Bearer {self.token}",
        "Content-Type": "application/json"
    }
```

--------------------------------

### MySQL User Privileges for Connector

Source: https://docs.popsink.com/connectors/source/mysql

Required SQL privileges for the user account that the MySQL Source Connector will use to connect to the database. Ensure these are granted before setup.

```sql
SELECT
RELOAD
SHOW DATABASES
REPLICATION SLAVE
REPLICATION CLIENT
```

--------------------------------

### POST /api/teams/

Source: https://docs.popsink.com/on-prem/on-prem

Creates a new team within an environment.

```APIDOC
## POST /api/teams/

### Description
Creates a team within your environment.

### Method
POST

### Endpoint
https://your-server/api/teams/

### Parameters
#### Request Body
- **name** (string) - Required - Team name (unique within environment)
- **description** (string) - Optional - Purpose and responsibilities of the team

### Response
#### Success Response (201)
- **id** (string) - Team ID
- **name** (string) - Team name
- **description** (string) - Team description
- **env_id** (string) - Associated environment ID

#### Response Example
{
  "id": "881e0fce-5caa-4f5c-82c6-5eb883c47100",
  "name": "Data Engineering Team",
  "description": "Team responsible for ETL pipelines",
  "env_id": "d455a981-be17-455e-a03c-4d9c3224db72"
}
```

--------------------------------

### Create a New Team

Source: https://docs.popsink.com/on-prem/on-prem

Create a team within your environment. The creator automatically becomes a team owner. This action requires an active environment and authentication.

```bash
curl -X POST "https://your-server/api/teams/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Data Engineering Team",
    "description": "Team responsible for ETL pipelines"
  }' -k
```

--------------------------------

### Create a Popsink Pipeline

Source: https://docs.popsink.com/on-prem/on-prem

Use this endpoint to create a new pipeline. Ensure you replace placeholder values like 'your-team-id' and 'my-source' with your actual configuration.

```python
pipeline_data = {
    "name": "my-pipeline",
    "team_id": "your-team-id",
    "source_name": "my-source",
    "source_type": "KAFKA_SOURCE",
    "source_config": {...}
}

response = requests.post(
    f"{SERVER}/api/pipelines/",
    headers=headers,
    json=pipeline_data,
    verify=False
)

print(response.json())
```

--------------------------------

### Create a Pipeline

Source: https://docs.popsink.com/on-prem/on-prem

Define and create a new pipeline. Requires a name, team ID, and a configuration dictionary. The configuration specifies source, target, and subscription details.

```python
def create_pipeline(self, name: str, team_id: str, config: dict) -> dict:
    """Create a pipeline"""
    pipeline_data = {
        "name": name,
        "team_id": team_id,
        **config
    }
    response = requests.post(
        f"{self.server}/api/pipelines/",
        headers=self._headers(),
        json=pipeline_data,
        verify=self.verify_ssl
    )
    response.raise_for_status()
    return response.json()
```

--------------------------------

### Create Pipeline

Source: https://docs.popsink.com/on-prem/on-prem

Creates a new pipeline with the specified configuration.

```APIDOC
## POST /api/pipelines/

### Description
Create a new pipeline.

### Method
POST

### Endpoint
/api/pipelines/

### Request Body
- **name** (string) - Required - The name of the pipeline.
- **team_id** (string) - Required - The ID of the team the pipeline belongs to.
- **source_name** (string) - Required - The name of the data source.
- **source_type** (string) - Required - The type of the data source (e.g., KAFKA_SOURCE).
- **source_config** (object) - Required - Configuration details for the data source.

### Request Example
```json
{
  "name": "my-pipeline",
  "team_id": "your-team-id",
  "source_name": "my-source",
  "source_type": "KAFKA_SOURCE",
  "source_config": {}
}
```

### Response
#### Success Response (200)
- **pipeline_id** (string) - The ID of the created pipeline.
- **message** (string) - A confirmation message.

#### Response Example
```json
{
  "pipeline_id": "pipeline-123",
  "message": "Pipeline created successfully."
}
```
```

--------------------------------

### Login and Obtain Token

Source: https://docs.popsink.com/on-prem/on-prem

Authenticate with the server using email and password to obtain an access token. The token is stored internally and returned by the method. Ensure credentials are correct.

```python
def login(self, email: str, password: str) -> str:
    """Login and get token"""
    response = requests.post(
        f"{self.server}/api/auth/jwt/login",
        data={
            "username": email,
            "password": password
        },
        verify=self.verify_ssl
    )
    response.raise_for_status()
    data = response.json()
    self.token = data["access_token"]
    return self.token
```

--------------------------------

### Configure Pipeline Source Options

Source: https://docs.popsink.com/on-prem/on-prem

Choose between using an existing source ID or defining a new source configuration.

```json
{
  "existing_source_id": "uuid-here"
}
```

```json
{
  "source_name": "my-source",
  "source_type": "KAFKA_SOURCE",
  "source_config": {...}
}
```

--------------------------------

### Apply *BOTH Journaling to All Tables

Source: https://docs.popsink.com/snowflake-ibmi/journaling.md

Execute these commands to update the journaling settings for all specified tables to use *BOTH images. This ensures that all tables involved in replication capture the necessary pre- and post-change data.

```CL
CHGJRNOBJ OBJ((POPSINK1/CATEGORIES *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

```CL
CHGJRNOBJ OBJ((POPSINK1/SUPPLIERS *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

```CL
CHGJRNOBJ OBJ((POPSINK1/PRODUCTS *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

```CL
CHGJRNOBJ OBJ((POPSINK1/STOCK *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

```CL
CHGJRNOBJ OBJ((POPSINK1/PURCHASES *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

```CL
CHGJRNOBJ OBJ((POPSINK1/SALES *FILE)) ATR(*IMAGES) IMAGES(*BOTH)
```

--------------------------------

### Test Pipeline API with HTTPie

Source: https://docs.popsink.com/on-prem/on-prem

Use HTTPie for a more user-friendly command-line interface.

```bash
# Install
pip install httpie

# Use
http GET https://your-server/api/pipelines/ \
  Authorization:"Bearer $TOKEN"
```

--------------------------------

### Verify Project Completion

Source: https://docs.popsink.com/on-prem/on-prem

Prints a success message and executes the main function to confirm all steps are finished.

```python
print("\n✓ All steps completed successfully!")


if __name__ == "__main__":
    main()
```

--------------------------------

### Create a New Environment

Source: https://docs.popsink.com/on-prem/on-prem

Create a new environment within Popsink. Requires a name and optionally a description. The `use_retention` parameter is set to True by default.

```python
def create_environment(self, name: str, description: str = "") -> dict:
    """Create an environment"""
    response = requests.post(
        f"{self.server}/api/envs/",
        headers=self._headers(),
        json={
            "name": name,
            "description": description,
            "use_retention": True
        },
        verify=self.verify_ssl
    )
    response.raise_for_status()
    return response.json()
```

--------------------------------

### POST /api/auth/register

Source: https://docs.popsink.com/on-prem/on-prem

Registers a new user account in the system.

```APIDOC
## POST /api/auth/register

### Description
Registers a new user account.

### Method
POST

### Endpoint
https://your-server/api/auth/register

### Parameters
#### Request Body
- **email** (string) - Required - Your email address (must be unique)
- **password** (string) - Required - Your password (min 8 characters recommended)
- **is_active** (boolean) - Required - Set to true to activate the account immediately
- **is_superuser** (boolean) - Optional - Admin privileges (default: false)
- **is_verified** (boolean) - Optional - Email verification status (default: false)

### Request Example
{
  "email": "john.doe@company.com",
  "password": "MySecurePassword123!",
  "is_active": true,
  "is_superuser": false,
  "is_verified": false
}

### Response
#### Success Response (201)
- **id** (string) - User ID
- **email** (string) - User email
- **is_active** (boolean) - Active status
- **is_superuser** (boolean) - Superuser status
- **is_verified** (boolean) - Verification status
- **active_env_id** (null) - Active environment ID

#### Response Example
{
  "id": "3b0c298a-062e-42d1-a01b-277fa66908da",
  "email": "john.doe@company.com",
  "is_active": true,
  "is_superuser": false,
  "is_verified": false,
  "active_env_id": null
}
```

--------------------------------

### Paginate Pipeline Results (Bash)

Source: https://docs.popsink.com/on-prem/on-prem

Demonstrates how to paginate through pipeline results using skip and limit parameters with curl. Adjust skip and limit values to retrieve different pages.

```bash
# Get first page (default 50 items)
curl -X GET "https://your-server/api/pipelines/?skip=0&limit=50" \
  -H "Authorization: Bearer $TOKEN" -k
```

```bash
# Get second page
curl -X GET "https://your-server/api/pipelines/?skip=50&limit=50" \
  -H "Authorization: Bearer $TOKEN" -k
```

--------------------------------

### Display Library List on IBMi

Source: https://docs.popsink.com/snowflake-ibmi/db2-setup

Execute the DSPLIBL command within the tn5250j terminal to view the list of libraries (schemas) available on the IBMi system.

```cl
DSPLIBL
```

--------------------------------

### Create Publication for Tables

Source: https://docs.popsink.com/connectors/source/postgres

Create a SQL publication to specify which tables should be included in the replication stream for Popsink.

```sql
CREATE PUBLICATION popsink FOR TABLE <your table list here>;
```

--------------------------------

### Enable Supplemental Logging at Table Level

Source: https://docs.popsink.com/connectors/source/oracle

Enable supplemental logging for a specific table. Use this if you only need to replicate specific tables.

```sql
ALTER TABLE **{db.name}** ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
```

--------------------------------

### User Registration

Source: https://docs.popsink.com/on-prem/on-prem

Register a new user account with the API.

```APIDOC
## POST /api/auth/register

### Description
Register a new user account.

### Method
POST

### Endpoint
/api/auth/register

### Request Body
- **email** (string) - Required - The email address for the new user.
- **password** (string) - Required - The password for the new user.
- **is_active** (boolean) - Optional - Whether the user account should be active upon creation.

### Request Example
{
  "email": "user@example.com",
  "password": "secure123",
  "is_active": true
}

### Response
#### Success Response (201)
- **id** (string) - The unique identifier for the newly created user.
- **email** (string) - The email address of the user.
- **is_active** (boolean) - Indicates if the user account is active.

#### Response Example
{
  "id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "email": "user@example.com",
  "is_active": true
}
```
Tokens:
Raw