DungBeetle

https://github.com/zerodha/dungbeetle
Admin
DungBeetle is a lightweight distributed job server for queuing and asynchronously executing SQL read...
Tokens:12,111
Snippets:80
Trust Score:8
License:MIT
Update:5 months ago
Show doc for...
Tokens:
Raw
### Go Client Library Examples

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Examples demonstrating how to use the DungBeetle Go client library for various operations.

```APIDOC
## Go Client Library

DungBeetle provides a Go HTTP client library for programmatic interaction with the server.

### Example Usage
```go
package main

import (
    "fmt"
    "log"
    "log/slog"
    "net/http"
    "time"

    "github.com/zerodha/dungbeetle/v2/client"
    "github.com/zerodha/dungbeetle/v2/models"
)

func main() {
    // Initialize the client
    c := client.New(&client.Opt{
        RootURL:    "http://localhost:6060",
        HTTPClient: &http.Client{Timeout: 30 * time.Second},
        Logger:     slog.Default(),
    })

    // Post a new job
    jobResp, err := c.PostJob(models.JobReq{
        TaskName: "get_profit_entries_by_date",
        JobID:    "go_client_job_001",
        Args:     []string{"USER123", "2024-01-01", "2024-12-31"},
        Queue:    "default",
        Retries:  3,
    })
    if err != nil {
        log.Fatalf("failed to post job: %v", err)
    }
    fmt.Printf("Job queued: %s\n", jobResp.JobID)

    // Poll for job completion
    for {
        status, err := c.GetJobStatus(jobResp.JobID)
        if err != nil {
            log.Fatalf("failed to get job status: %v", err)
        }

        fmt.Printf("Job state: %s\n", status.State)

        if status.State == "SUCCESS" {
            fmt.Printf("Job completed! Rows returned: %d\n", status.Count)
            break
        } else if status.State == "FAILURE" {
            fmt.Printf("Job failed: %s\n", status.Error)
            break
        }

        time.Sleep(2 * time.Second)
    }

    // Post a group of jobs
    groupResp, err := c.PostJobGroup(models.GroupReq{
        GroupID: "batch_reports",
        Jobs: []models.JobReq{
            {TaskName: "get_profit_summary", JobID: "batch_1", Args: []string{"DEPT_A"}},
            {TaskName: "get_profit_summary", JobID: "batch_2", Args: []string{"DEPT_B"}},
        },
    })
    if err != nil {
        log.Fatalf("failed to post job group: %v", err)
    }
    fmt.Printf("Group queued: %s with %d jobs\n", groupResp.GroupID, len(groupResp.Jobs))

    // Check group status
    groupStatus, err := c.GetGroupStatus(groupResp.GroupID)
    if err != nil {
        log.Fatalf("failed to get group status: %v", err)
    }
    fmt.Printf("Group state: %s\n", groupStatus.State)

    // Get pending jobs in queue
    pending, err := c.GetPendingJobs("default")
    if err != nil {
        log.Fatalf("failed to get pending jobs: %v", err)
    }
    fmt.Printf("Pending jobs in queue: %d\n", len(pending))

    // Delete a job
    if err := c.DeleteJob("go_client_job_001", true); err != nil {
        log.Fatalf("failed to delete job: %v", err)
    }

    // Delete a group
    if err := c.DeleteGroupJob("batch_reports", true); err != nil {
        log.Fatalf("failed to delete group: %v", err)
    }
}
```
```

--------------------------------

### List Registered Tasks API

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Examples of using the `GET /tasks` API endpoint to retrieve a list of registered SQL tasks. Shows how to get just task names or full task details including SQL queries.

```bash
# Get list of task names only
curl http://localhost:6060/tasks

# Response:
{
  "status": "success",
  "data": ["get_profit_summary", "get_profit_entries", "get_profit_entries_by_date"]
}

# Get full task details including SQL queries
curl "http://localhost:6060/tasks?sql=1"

# Response:
{
  "status": "success",
  "data": {
    "get_profit_summary": {
      "name": "get_profit_summary",
      "queue": "default",
      "concurrency": 10,
      "raw": "SELECT SUM(amount) AS total, entry_date FROM entries WHERE user_id = ? GROUP BY entry_date;"
    },
    "get_profit_entries": {
      "name": "get_profit_entries",
      "queue": "high_priority",
      "concurrency": 10,
      "raw": "SELECT * FROM entries WHERE user_id = ?;"
    }
  }
}

```

--------------------------------

### Start Dungbeetle Server

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Use this command to start the Dungbeetle server, specifying the configuration file and SQL query directory. Run 'dungbeetle --help' for all arguments.

```shell
dungbeetle --config /path/to/config.toml --sql-directory /path/to/your/sql/queries
```

--------------------------------

### Start DungBeetle Server

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Commands to generate a sample configuration file and start the DungBeetle server with various options, including worker-only mode.

```bash
dungbeetle --new-config
```

```bash
dungbeetle --config /path/to/config.toml \
           --sql-directory /path/to/sql/queries \
           --server 127.0.0.1:6060 \
           --queue default \
           --worker-name my_worker \
           --worker-concurrency 10
```

```bash
dungbeetle --config /path/to/config.toml \
           --sql-directory /path/to/sql/queries \
           --worker-only
```

--------------------------------

### Define SQL Tasks with goyesql

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Examples of SQL tasks defined in `.sql` files using the goyesql format. Demonstrates simple queries, queries with specific database/queue/result backend tags, and raw queries.

```sql
-- queries.sql

-- name: get_profit_summary
-- Simple query attached to all configured DBs
SELECT SUM(amount) AS total, entry_date FROM entries WHERE user_id = ? GROUP BY entry_date;

-- name: get_profit_entries
-- db: my_db, backup_db
-- queue: high_priority
-- results: my_results
-- Query attached to specific DBs, queue, and result backend
SELECT * FROM entries WHERE user_id = ?;

-- name: get_profit_entries_by_date
-- Query with multiple positional arguments
SELECT * FROM entries WHERE user_id = ? AND timestamp > ? AND timestamp < ?;

-- name: get_raw_report
-- raw: 1
-- Raw query (not prepared) - useful for dynamic SQL
SELECT * FROM entries WHERE user_id = ? AND status IN ('active', 'pending');

```

--------------------------------

### Running Multiple Workers with Priority Queues

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Configuration and usage examples for running multiple DungBeetle worker instances with different queues and concurrency levels.

```APIDOC
## Running Multiple Workers with Priority Queues

DungBeetle supports running multiple worker instances with different queues and concurrency levels for handling jobs with different priorities.

### Command Line Examples

```bash
# Run primary worker with HTTP server for high priority jobs
dungbeetle --config /path/to/config.toml \
           --sql-directory /path/to/sql \
           --queue "high_priority" \
           --worker-name "high_priority_worker" \
           --worker-concurrency 30 \
           --server 127.0.0.1:6060

# Run secondary worker (worker-only mode) for low priority jobs
dungbeetle --config /path/to/config.toml \
           --sql-directory /path/to/sql \
           --queue "low_priority" \
           --worker-name "low_priority_worker" \
           --worker-concurrency 5 \
           --worker-only
```

### Submitting Jobs to Different Queues

```bash
# Submit a job to the high priority queue
curl -X POST http://localhost:6060/tasks/get_profit_summary/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_id": "urgent_report", "args": ["USER1"], "queue": "high_priority"}'

# Submit a job to the low priority queue
curl -X POST http://localhost:6060/tasks/get_profit_entries/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_id": "batch_export", "args": ["USER2"], "queue": "low_priority"}'
```
```

--------------------------------

### DungBeetle TOML Configuration

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Example TOML configuration file for DungBeetle, detailing settings for application, Redis job queue, Redis state store, PostgreSQL results database, PostgreSQL source database, and ClickHouse source database.

```toml
[app]
log_level = "DEBUG"
default_job_ttl = "60s"

# Redis broker for job queue
[job_queue.broker]
type = "redis"
addresses = ["localhost:6379"]
password = ""
db = 1
max_active = 50
max_idle = 20
dial_timeout = "1s"
read_timeout = "1s"
write_timeout = "1s"

# Redis state store for job status
[job_queue.state]
type = "redis"
addresses = ["localhost:6379"]
password = ""
db = 1
max_active = 50
max_idle = 20
dial_timeout = "1s"
read_timeout = "1s"
write_timeout = "1s"
expiry = "3000s"
meta_expiry = "3600s"

# Results database (where query results are stored)
[results.my_results]
type = "postgres"
dsn = "postgres://user:pass@127.0.0.1:5432/resultsDB?sslmode=disable"
max_idle = 10
max_active = 100
connect_timeout = "10s"
results_table = "results_%s"

# Source database (where queries are executed)
[db.my_db]
type = "postgres"
dsn = "postgres://user:pass@127.0.0.1:5432/sourceDB?sslmode=disable"
max_idle = 10
max_active = 100
connect_timeout = "10s"

# ClickHouse source database example
[db.ch_db]
type = "clickhouse"
dsn = "clickhouse://default:@127.0.0.1:9000/default?sslmode=disable"
max_idle = 10
max_active = 100
connect_timeout = "10s"

```

--------------------------------

### Run primary DungBeetle worker

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Starts a DungBeetle worker with the HTTP control interface. Specify configuration, SQL directory, queue name, worker name, and concurrency level.

```shell
dungbeetle --config /path/to/config.toml --sql-directory /path/to/sql/dir \
	--queue "high_priority" \
    --worker-name "high_priority_worker" \
    --worker-concurrency 30
```

--------------------------------

### Run secondary DungBeetle worker (worker-only)

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Starts a DungBeetle worker without the HTTP control interface, suitable for handling low-priority jobs. Requires configuration, SQL directory, queue name, worker name, and concurrency.

```shell
dungbeetle --config /path/to/config.toml --sql-directory /path/to/sql/dir \
	--queue "low_priority" \
    --worker-name "low_priority_worker" \
    --worker-concurrency 5 \
    --worker-only
```

--------------------------------

### GET /tasks - List Registered Tasks

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Retrieves a list of all registered SQL tasks. The response can optionally include the raw SQL queries for each task.

```APIDOC
## GET /tasks - List Registered Tasks

### Description
Returns the list of all registered SQL tasks loaded from the SQL files. Optionally include the raw SQL queries.

### Method
GET

### Endpoint
/tasks

### Query Parameters
- **sql** (integer) - Optional - If set to 1, include the raw SQL queries in the response.

### Response
#### Success Response (200)
- **status** (string) - Indicates the status of the operation ('success').
- **data** (array or object) - If 'sql' parameter is not set, an array of task names. If 'sql' parameter is set, an object where keys are task names and values are task details (including name, queue, concurrency, and raw SQL).

### Response Example (without sql parameter)
```json
{
  "status": "success",
  "data": ["get_profit_summary", "get_profit_entries", "get_profit_entries_by_date"]
}
```

### Response Example (with sql=1 parameter)
```json
{
  "status": "success",
  "data": {
    "get_profit_summary": {
      "name": "get_profit_summary",
      "queue": "default",
      "concurrency": 10,
      "raw": "SELECT SUM(amount) AS total, entry_date FROM entries WHERE user_id = ? GROUP BY entry_date;"
    },
    "get_profit_entries": {
      "name": "get_profit_entries",
      "queue": "high_priority",
      "concurrency": 10,
      "raw": "SELECT * FROM entries WHERE user_id = ?;"
    }
  }
}
```
```

--------------------------------

### GET /jobs/queue/{queue} - List Pending Jobs in Queue

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Retrieves a list of all pending jobs within a specified queue.

```APIDOC
## GET /jobs/queue/{queue}

### Description
Returns all pending jobs in a specific queue, useful for monitoring queue depth and job backlog.

### Method
GET

### Endpoint
/jobs/queue/{queue}

### Parameters
#### Path Parameters
- **queue** (string) - Required - The name of the queue to list jobs from.

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.
- **data** (array) - A list of pending jobs in the specified queue.
  - **id** (string) - The ID of the job.
  - **task** (string) - The name of the task associated with the job.
  - **queue** (string) - The queue the job belongs to.

#### Response Example
```json
{
  "status": "success",
  "data": [
    {
      "id": "job_abc123",
      "task": "get_profit_summary",
      "queue": "default"
    },
    {
      "id": "job_def456",
      "task": "get_profit_entries",
      "queue": "default"
    }
  ]
}
```
```

--------------------------------

### List Pending Jobs in Queue

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Get a list of all jobs currently pending in a specified queue. This is helpful for monitoring the backlog and understanding queue depth.

```bash
curl http://localhost:6060/jobs/queue/default
```

```bash
curl http://localhost:6060/jobs/queue/high_priority
```

--------------------------------

### GET /groups/{groupID} - Check Group Status

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Retrieves the status of a job group, including the states of all individual jobs within it.

```APIDOC
## GET /groups/{groupID}

### Description
Returns the status of a job group including the state of all individual jobs within the group.

### Method
GET

### Endpoint
/groups/{groupID}

### Parameters
#### Path Parameters
- **groupID** (string) - Required - The unique identifier of the job group to check.

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.
- **data** (object) - Contains the status of the job group and its individual jobs.
  - **group_id** (string) - The ID of the job group.
  - **jobs** (array) - A list of jobs within the group, with their current states.
    - **job_id** (string) - The ID of the individual job.
    - **state** (string) - The current execution state of the job.
    - **task** (string) - The name of the task.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "group_id": "monthly_reports_2024",
    "jobs": [
      {
        "job_id": "sales_report",
        "state": "SUCCESS",
        "task": "get_profit_summary"
      },
      {
        "job_id": "marketing_report",
        "state": "PENDING",
        "task": "get_profit_summary"
      },
      {
        "job_id": "ops_report",
        "state": "STARTED",
        "task": "get_profit_entries_by_date"
      }
    ]
  }
}
```
```

--------------------------------

### GET /jobs/{jobID} - Check Job Status

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Retrieves the current status of a specific job, including its execution state, row count, and any associated errors.

```APIDOC
## GET /jobs/{jobID}

### Description
Returns the current status of a job including execution state, result row count, and any errors.

### Method
GET

### Endpoint
/jobs/{jobID}

### Parameters
#### Path Parameters
- **jobID** (string) - Required - The unique identifier of the job to check.

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.
- **data** (object) - Contains the job status details.
  - **job_id** (string) - The ID of the job.
  - **state** (string) - The current execution state of the job (e.g., PENDING, STARTED, SUCCESS, FAILURE, RETRY).
  - **count** (integer) - The number of rows processed or affected by the job.
  - **error** (string) - Any error message if the job failed.

#### Response Example (Job in progress)
```json
{
  "status": "success",
  "data": {
    "job_id": "report_user123_2024",
    "state": "STARTED",
    "count": 0,
    "error": ""
  }
}
```

#### Response Example (Job completed successfully)
```json
{
  "status": "success",
  "data": {
    "job_id": "report_user123_2024",
    "state": "SUCCESS",
    "count": 1542,
    "error": ""
  }
}
```

#### Response Example (Job failed)
```json
{
  "status": "success",
  "data": {
    "job_id": "report_user123_2024",
    "state": "FAILURE",
    "count": 0,
    "error": "task execution failed: get_profit_entries_by_date: pq: relation \"entries\" does not exist"
  }
}
```
```

--------------------------------

### Define SQL Tasks with Metadata

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

SQL queries can be defined in .sql files with metadata tags like `-- db:`, `-- queue:`, and `-- results:`. The `raw: 1` tag prevents query preparation.

```sql
-- queries.sql

-- name: get_profit_summary
SELECT SUM(amount) AS total, entry_date FROM entries WHERE user_id = ? GROUP BY entry_date;

-- name: get_profit_entries
-- db: db0, other_db
-- queue: myqueue
-- results: my_res_db
SELECT * FROM entries WHERE user_id = ?;

-- name: get_profit_entries_by_date
SELECT * FROM entries WHERE user_id = ? AND timestamp > ? and timestamp < ?;

-- name: get_profit_entries_by_date
-- raw: 1
-- This query will not be prepared (raw=1)
SELECT * FROM entries WHERE user_id = ? AND timestamp > ? and timestamp < ?;
```

--------------------------------

### Go Client: Post and Poll a Single Job

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Initialize the DungBeetle Go client and use it to post a new job, then poll its status until completion or failure. Ensure the HTTP client has a suitable timeout.

```go
package main

import (
    "fmt"
    "log"
    "log/slog"
    "net/http"
    "time"

    "github.com/zerodha/dungbeetle/v2/client"
    "github.com/zerodha/dungbeetle/v2/models"
)

func main() {
    // Initialize the client
    c := client.New(&client.Opt{
        RootURL:    "http://localhost:6060",
        HTTPClient: &http.Client{Timeout: 30 * time.Second},
        Logger:     slog.Default(),
    })

    // Post a new job
    jobResp, err := c.PostJob(models.JobReq{
        TaskName: "get_profit_entries_by_date",
        JobID:    "go_client_job_001",
        Args:     []string{"USER123", "2024-01-01", "2024-12-31"},
        Queue:    "default",
        Retries:  3,
    })
    if err != nil {
        log.Fatalf("failed to post job: %v", err)
    }
    fmt.Printf("Job queued: %s\n", jobResp.JobID)

    // Poll for job completion
    for {
        status, err := c.GetJobStatus(jobResp.JobID)
        if err != nil {
            log.Fatalf("failed to get job status: %v", err)
        }

        fmt.Printf("Job state: %s\n", status.State)

        if status.State == "SUCCESS" {
            fmt.Printf("Job completed! Rows returned: %d\n", status.Count)
            break
        } else if status.State == "FAILURE" {
            fmt.Printf("Job failed: %s\n", status.Error)
            break
        }

        time.Sleep(2 * time.Second)
    }

    // Post a group of jobs
    groupResp, err := c.PostJobGroup(models.GroupReq{
        GroupID: "batch_reports",
        Jobs: []models.JobReq{
            {TaskName: "get_profit_summary", JobID: "batch_1", Args: []string{"DEPT_A"}},
            {TaskName: "get_profit_summary", JobID: "batch_2", Args: []string{"DEPT_B"}},
        },
    })
    if err != nil {
        log.Fatalf("failed to post job group: %v", err)
    }
    fmt.Printf("Group queued: %s with %d jobs\n", groupResp.GroupID, len(groupResp.Jobs))

    // Check group status
    groupStatus, err := c.GetGroupStatus(groupResp.GroupID)
    if err != nil {
        log.Fatalf("failed to get group status: %v", err)
    }
    fmt.Printf("Group state: %s\n", groupStatus.State)

    // Get pending jobs in queue
    pending, err := c.GetPendingJobs("default")
    if err != nil {
        log.Fatalf("failed to get pending jobs: %v", err)
    }
    fmt.Printf("Pending jobs in queue: %d\n", len(pending))

    // Delete a job
    if err := c.DeleteJob("go_client_job_001", true); err != nil {
        log.Fatalf("failed to delete job: %v", err)
    }

    // Delete a group
    if err := c.DeleteGroupJob("batch_reports", true); err != nil {
        log.Fatalf("failed to delete group: %v", err)
    }
}
```

--------------------------------

### Schedule Job with Delayed Execution (ETA)

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Schedule a job to run at a specific future time by providing an ETA. The job will be queued and executed once the specified time is reached.

```bash
curl -X POST http://localhost:6060/tasks/get_profit_summary/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "scheduled_report",
    "args": ["USER456"],
    "eta": "2024-12-25 09:00:00"
  }'
```

--------------------------------

### POST /groups - Schedule a Group of Jobs

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Creates and queues multiple jobs for concurrent execution as a group.

```APIDOC
## POST /groups

### Description
Creates and queues multiple jobs as a group for concurrent execution. Useful when you need to run several related reports and track their collective completion.

### Method
POST

### Endpoint
/groups

### Parameters
#### Request Body
- **group_id** (string) - Required - A unique identifier for the job group.
- **concurrency** (integer) - Optional - The maximum number of jobs in the group to run concurrently. Defaults to 1.
- **jobs** (array) - Required - A list of job objects to be included in the group.
  - **job_id** (string) - Required - A unique identifier for the individual job.
  - **task** (string) - Required - The name of the task to execute.
  - **args** (array) - Optional - Arguments for the task.
  - **queue** (string) - Optional - The queue for the job.
  - **retries** (integer) - Optional - Number of retries for the job.
  - **eta** (string) - Optional - Desired execution time for the job.
  - **db** (string) - Optional - Target database for the job.

### Request Example
```json
{
  "group_id": "monthly_reports_2024",
  "concurrency": 3,
  "jobs": [
    {
      "job_id": "sales_report",
      "task": "get_profit_summary",
      "args": ["SALES_DEPT"]
    },
    {
      "job_id": "marketing_report",
      "task": "get_profit_summary",
      "args": ["MARKETING_DEPT"]
    },
    {
      "job_id": "ops_report",
      "task": "get_profit_entries_by_date",
      "args": ["OPS_DEPT", "2024-01-01", "2024-01-31"]
    }
  ]
}
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.
- **data** (object) - Contains details of the scheduled job group.
  - **group_id** (string) - The ID of the job group.
  - **jobs** (array) - A list of jobs scheduled within the group, with their details.
    - **job_id** (string) - The ID of the individual job.
    - **task** (string) - The name of the task.
    - **queue** (string) - The queue assigned to the job.
    - **eta** (string|null) - The scheduled execution time.
    - **retries** (integer) - The number of retries configured.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "group_id": "monthly_reports_2024",
    "jobs": [
      {
        "job_id": "sales_report",
        "task": "get_profit_summary",
        "queue": "default",
        "eta": null,
        "retries": 0
      },
      {
        "job_id": "marketing_report",
        "task": "get_profit_summary",
        "queue": "default",
        "eta": null,
        "retries": 0
      },
      {
        "job_id": "ops_report",
        "task": "get_profit_entries_by_date",
        "queue": "default",
        "eta": null,
        "retries": 0
      }
    ]
  }
}
```
```

--------------------------------

### Schedule a Group of Jobs

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Schedule multiple jobs to run concurrently as a group. Define the group ID, concurrency limit, and the list of jobs to be executed.

```bash
curl -X POST http://localhost:6060/groups \
  -H "Content-Type: application/json" \
  -d '{
    "group_id": "monthly_reports_2024",
    "concurrency": 3,
    "jobs": [
      {
        "job_id": "sales_report",
        "task": "get_profit_summary",
        "args": ["SALES_DEPT"]
      },
      {
        "job_id": "marketing_report",
        "task": "get_profit_summary",
        "args": ["MARKETING_DEPT"]
      },
      {
        "job_id": "ops_report",
        "task": "get_profit_entries_by_date",
        "args": ["OPS_DEPT", "2024-01-01", "2024-01-31"]
      }
    ]
  }'
```

--------------------------------

### Schedule Job with Custom ID and Arguments

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Use this endpoint to schedule a job with a specific job ID and provide arguments for the task. Specify the queue and number of retries if needed.

```bash
curl -X POST http://localhost:6060/tasks/get_profit_entries_by_date/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "report_user123_2024",
    "args": ["USER123", "2024-01-01", "2024-12-31"],
    "queue": "high_priority",
    "retries": 3
  }'
```

--------------------------------

### Schedule a single job with curl

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Use this command to schedule a single job. Ensure the correct task name and arguments are provided.

```shell
$ curl localhost:6060/tasks/get_profit_entries_by_date/jobs -H "Content-Type: application/json" -X POST --data '{"job_id": "myjob", "args": ["USER1", "2017-12-01", "2017-01-01"]}'

{"status":"success","data":{"job_id":"myjob","task_name":"get_profit_entries_by_date","queue":"sqljob_queue","eta":null,"retries":0}}
```

--------------------------------

### POST /tasks/{task_name}/jobs - Schedule a Job

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Schedules a new job for a specific task with optional arguments, queue, retries, and delayed execution (ETA).

```APIDOC
## POST /tasks/{task_name}/jobs

### Description
Schedules a new job for a specific task with optional arguments, queue, retries, and delayed execution (ETA).

### Method
POST

### Endpoint
/tasks/{task_name}/jobs

### Parameters
#### Request Body
- **job_id** (string) - Required - A unique identifier for the job.
- **args** (array) - Optional - A list of arguments to pass to the task.
- **queue** (string) - Optional - The queue to which the job should be assigned. Defaults to 'default'.
- **retries** (integer) - Optional - The number of times the job should be retried if it fails. Defaults to 0.
- **eta** (string) - Optional - The desired execution time for the job in 'YYYY-MM-DD HH:MM:SS' format.
- **db** (string) - Optional - Specifies a target database for the job.

### Request Example
```json
{
  "job_id": "report_user123_2024",
  "args": ["USER123", "2024-01-01", "2024-12-31"],
  "queue": "high_priority",
  "retries": 3
}
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.
- **data** (object) - Contains details of the scheduled job.
  - **job_id** (string) - The ID of the scheduled job.
  - **task** (string) - The name of the task that was scheduled.
  - **queue** (string) - The queue the job was assigned to.
  - **eta** (string|null) - The scheduled execution time, if provided.
  - **retries** (integer) - The number of retries configured for the job.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "job_id": "report_user123_2024",
    "task": "get_profit_entries_by_date",
    "queue": "high_priority",
    "eta": null,
    "retries": 3
  }
}
```
```

--------------------------------

### POST /tasks/{taskName}/jobs - Schedule a Job

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Creates and queues a new job for execution based on a registered task. The API returns immediately with job details, allowing for asynchronous status polling.

```APIDOC
## POST /tasks/{taskName}/jobs - Schedule a Job

### Description
Creates and queues a new job for execution against a registered task. Returns immediately with job details for status polling.

### Method
POST

### Endpoint
/tasks/{taskName}/jobs

### Path Parameters
- **taskName** (string) - Required - The name of the task to schedule a job for.

### Request Body
- **args** (array) - Optional - Positional arguments to pass to the SQL query.
- **db** (string) - Optional - Specifies the source database to use for this job. Overrides the task's default database configuration.
- **queue** (string) - Optional - Specifies the job queue to use. Overrides the task's default queue configuration.
- **results** (string) - Optional - Specifies the results backend to use. Overrides the task's default results backend configuration.

### Request Example
```json
{
  "args": [123, "2023-01-01", "2023-12-31"],
  "db": "my_db",
  "queue": "high_priority",
  "results": "my_results"
}
```

### Response
#### Success Response (200 or 201)
- **status** (string) - Indicates the status of the operation ('success').
- **data** (object) - Contains details of the created job.
  - **job_id** (string) - The unique identifier for the job.
  - **task_name** (string) - The name of the task that was scheduled.
  - **status** (string) - The initial status of the job (e.g., 'queued').
  - **created_at** (string) - Timestamp when the job was created.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "job_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
    "task_name": "get_profit_entries_by_date",
    "status": "queued",
    "created_at": "2023-10-27T10:00:00Z"
  }
}
```
```

--------------------------------

### Schedule Job Targeting Specific Database

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Schedule a job and specify a particular database to be used for its execution. This is useful for tasks that require access to a specific data source.

```bash
curl -X POST http://localhost:6060/tasks/get_profit_entries/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "specific_db_job",
    "args": ["USER789"],
    "db": "backup_db"
  }'
```

--------------------------------

### Schedule a group of jobs with curl

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Schedule multiple jobs to run concurrently. The `group_id` identifies the group, and `concurrency` limits simultaneous jobs within the group. Jobs within the group run independently.

```shell
$ curl localhost:6060/groups -H "Content-Type: application/json" -X POST --data '{"group_id": "mygroup", "concurrency": 3, "jobs": [{"job_id": "myjob", "task": "get_profit_entries_by_date", "args": ["USER1", "2017-12-01", "2017-01-01"]}, {"job_id": "myjob2", "task": "get_profit_entries_by_date", "args": ["USER1", "2017-12-01", "2017-01-01"]}]'

{"status":"success","data":{"group_id":"mygroup","jobs":[{"job_id":"myjob","task":"test1","queue":"sqljob_queue","eta":null,"retries":0},{"job_id":"myjob2","task":"test2","queue":"sqljob_queue","eta":null,"retries":0}]}}
```

--------------------------------

### Task Management API

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Endpoints for retrieving task information and scheduling jobs.

```APIDOC
## GET /tasks

### Description
Returns the list of registered SQL tasks.

### Method
GET

### Endpoint
/tasks

## POST /tasks/{taskName}/jobs

### Description
Schedules a job for a given task.

### Method
POST

### Endpoint
/tasks/{taskName}/jobs

### Parameters
#### Path Parameters
- **taskName** (string) - Required - The name of the task to schedule a job for.

### Request Body
- **job_id** (string) - Optional - Alphanumeric ID for the job. If not passed, the server generates one.
- **queue** (string) - Optional - Queue to send the job to. Workers listening on this queue will receive the job.
- **eta** (string) - Optional - Timestamp (yyyy-mm-dd hh:mm:ss) at which the job should start. If not provided, the job is queued immediately.
- **retries** (int) - Optional - The number of times a failed job should be retried. Defaults to 0.
- **args[]** (string) - Optional - Positional argument to pass to the SQL query. Can be passed multiple times.

### Request Example
```json
{
  "job_id": "my-custom-job-id",
  "queue": "high-priority",
  "eta": "2023-10-27 10:00:00",
  "retries": 3,
  "args": ["arg1", "arg2"]
}
```

### Response
#### Success Response (200)
- **jobID** (string) - The ID of the scheduled job.

#### Response Example
```json
{
  "jobID": "generated-job-id-12345"
}
```
```

--------------------------------

### Run Workers with Priority Queues

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Configure and run multiple DungBeetle worker instances, specifying queues, concurrency, and whether to run the HTTP server. Submit jobs to specific queues using cURL.

```bash
# Run primary worker with HTTP server for high priority jobs
dungbeetle --config /path/to/config.toml \
           --sql-directory /path/to/sql \
           --queue "high_priority" \
           --worker-name "high_priority_worker" \
           --worker-concurrency 30 \
           --server 127.0.0.1:6060

# Run secondary worker (worker-only mode) for low priority jobs
dungbeetle --config /path/to/config.toml \
           --sql-directory /path/to/sql \
           --queue "low_priority" \
           --worker-name "low_priority_worker" \
           --worker-concurrency 5 \
           --worker-only

# Submit jobs to different queues
curl -X POST http://localhost:6060/tasks/get_profit_summary/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_id": "urgent_report", "args": ["USER1"], "queue": "high_priority"}'

curl -X POST http://localhost:6060/tasks/get_profit_entries/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_id": "batch_export", "args": ["USER2"], "queue": "low_priority"}'
```

--------------------------------

### Send job to high priority queue

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Submits a job to the 'high_priority' queue using a curl request. Specify the job ID, queue, and arguments.

```shell
# Send a job to the high priority queue.
$ curl localhost:6060/tasks/get_profit_entries_by_date/jobs -H "Content-Type: application/json" --data '{"job_id": "myjob", "queue": "high_priority", "args": ["USER1", "2017-12-01", "2017-01-01"]}'
```

--------------------------------

### Send job to low priority queue

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Submits a job to the 'low_priority' queue using a curl request. Specify the job ID and queue.

```shell
# Send another job to the low priority queue.
$ curl localhost:6060/tasks/get_profit_entries_by_date/jobs -H "Content-Type: application/json" --data '{"job_id": "myjob", "queue": "low_priority"}'
```

--------------------------------

### Job Management API

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Endpoints for retrieving job status, managing job queues, and deleting jobs.

```APIDOC
## GET /jobs/{jobID}

### Description
Returns the status of a given job.

### Method
GET

### Endpoint
/jobs/{jobID}

#### Path Parameters
- **jobID** (string) - Required - The ID of the job to retrieve the status for.

## GET /jobs/queue/{queue}

### Description
Returns the list of all pending jobs in a specific queue.

### Method
GET

### Endpoint
/jobs/queue/{queue}

#### Path Parameters
- **queue** (string) - Required - The name of the queue to retrieve pending jobs from.

## DELETE /jobs/{jobID}

### Description
Deletes a pending job from the queue and immediately cancels its execution. If `purge=true` is sent as a query parameter, completed jobs will also be deleted. Note that only the Go PostgreSQL driver cancels queries mid-execution; MySQL server will continue execution. For MySQL, set `max_execution_time`.

### Method
DELETE

### Endpoint
/jobs/{jobID}

#### Path Parameters
- **jobID** (string) - Required - The ID of the job to delete.

#### Query Parameters
- **purge** (boolean) - Optional - If true, deletes completed jobs as well.
```

--------------------------------

### Group Management API

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Endpoints for scheduling and managing groups of jobs.

```APIDOC
## POST /groups

### Description
Schedules a group of jobs.

### Method
POST

### Endpoint
/groups

### Request Body
- **group_id** (string) - Optional - Alphanumeric ID for the group of jobs. If not passed, the server generates one.
- **concurrency** (int) - Optional - Number of jobs to run concurrently within the group.

### Request Example
```json
{
  "group_id": "my-job-group",
  "concurrency": 5
}
```

## GET /groups/{groupID}

### Description
Gets the status of a job group and its associated jobs.

### Method
GET

### Endpoint
/groups/{groupID}

#### Path Parameters
- **groupID** (string) - Required - The ID of the job group to retrieve status for.

## DELETE /groups/{groupID}

### Description
Deletes a pending job from the queue and immediately cancels its execution. If `purge=true` is sent as a query parameter, completed jobs will also be deleted. Note that only the Go PostgreSQL driver cancels queries mid-execution; MySQL server will continue execution. For MySQL, set `max_execution_time`.

### Method
DELETE

### Endpoint
/groups/{groupID}

#### Path Parameters
- **groupID** (string) - Required - The ID of the job group to delete.

#### Query Parameters
- **purge** (boolean) - Optional - If true, deletes completed jobs as well.
```

--------------------------------

### Check job status with curl

Source: https://github.com/zerodha/dungbeetle/blob/master/README.md

Retrieve the status and results of a specific job using its `job_id`. The `Results` field indicates the number of rows generated by the query.

```shell
$ curl localhost:6060/jobs/myjob
{"status":"success","data":{"job_id":"myjob","status":"SUCCESS","results":[{"Type":"int64","Value":2}],"error":""}}~                                                                               

# `Results` indicates the number of rows generated by the query.
```

--------------------------------

### DELETE /jobs/{jobID} - Cancel or Delete a Job

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Cancels a pending or running job. Optionally purges completed jobs.

```APIDOC
## DELETE /jobs/{jobID}

### Description
Cancels a pending or running job. For PostgreSQL, cancels query execution immediately. Use `purge=true` to delete completed jobs.

### Method
DELETE

### Endpoint
/jobs/{jobID}

### Parameters
#### Path Parameters
- **jobID** (string) - Required - The unique identifier of the job to cancel or delete.

#### Query Parameters
- **purge** (boolean) - Optional - If set to `true`, deletes completed jobs. Defaults to `false`.

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.
- **data** (boolean) - `true` if the operation was successful.

#### Response Example (Cancel job)
```json
{
  "status": "success",
  "data": true
}
```

#### Response Example (Delete completed job)
```json
{
  "status": "success",
  "data": true
}
```
```

--------------------------------

### Cancel a Job Group

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Use the DELETE /groups/{groupID} endpoint to cancel pending or running jobs within a group. Add `purge=true` to delete completed jobs.

```bash
# Cancel all jobs in a group
curl -X DELETE http://localhost:6060/groups/monthly_reports_2024
```

```bash
# Delete completed group jobs
curl -X DELETE "http://localhost:6060/groups/monthly_reports_2024?purge=true"
```

--------------------------------

### Check Job Status

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Retrieve the current status of a job using its job ID. This endpoint provides information on the job's state, execution progress, and any errors encountered.

```bash
curl http://localhost:6060/jobs/report_user123_2024
```

--------------------------------

### Cancel or Delete a Job

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Cancel a pending or running job using its job ID. Optionally, purge completed jobs by setting `purge=true` to remove them from the system.

```bash
curl -X DELETE http://localhost:6060/jobs/report_user123_2024
```

```bash
curl -X DELETE "http://localhost:6060/jobs/report_user123_2024?purge=true"
```

--------------------------------

### Check Group Status

Source: https://context7.com/zerodha/dungbeetle/llms.txt

Retrieve the status of a job group using its group ID. This provides an overview of the states of all individual jobs within the group.

```bash
curl http://localhost:6060/groups/monthly_reports_2024
```
Tokens:
Raw