### Initialize and Run Sample Digdag Workflow

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/getting_started.md

Initializes a new Digdag project with a sample workflow and then runs it. Assumes Digdag is installed and in the PATH.

```shell
digdag init <dir>
cd <dir>
digdag run <workflow_name>.dig
```

--------------------------------

### Start Digdag-UI Development Server

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Installs dependencies and starts the digdag-ui development server using npm.

```shell
cd digdag-ui/
npm install
npm run dev
```

--------------------------------

### Digdag Start Command Example

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Illustrates how to use the `digdag start` command to initiate a new session. It shows variations for specifying the session time and passing project-specific parameters.

```console
$ digdag start <project-name> <name> --session <hourly | daily | now | yyyy-MM-dd | "yyyy-MM-dd HH:mm:ss">

Starts a new session. This command requires project name, workflow name, and session_time. Examples:

$ digdag start myproj main --dry-run --session hourly
$ digdag start myproj main --session daily
$ digdag start myproj main --session "2016-01-01 00:00:00"
$ digdag start myproj main --session "2016-01-01" -p environment=staging -p user=frsyuki
```

--------------------------------

### Download and Install Digdag (Linux/macOS)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/getting_started.md

Downloads the latest Digdag executable using curl, makes it executable, and adds it to the system's PATH. Requires a Unix-like environment.

```shell
curl -o ~/bin/digdag --create-dirs -L "https://dl.digdag.io/digdag-latest"
chmod +x ~/bin/digdag
echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
```

--------------------------------

### Start Digdag Server

Source: https://github.com/treasure-data/digdag/blob/master/examples/REST_API.md

Launches the Digdag server with memory optimization enabled. This command is a prerequisite for interacting with the Digdag API.

```shell
$ digdag server --memory
```

--------------------------------

### Download and Install Digdag (Windows PowerShell)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/getting_started.md

Downloads the Digdag batch file using PowerShell on Windows, sets the TLS version, creates a bin directory, and adds it to the system's PATH. Requires PowerShell.

```powershell
PowerShell -Command "& {[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12; mkdir -Force $env:USERPROFILE\bin; Invoke-WebRequest http://dl.digdag.io/digdag-latest.jar -OutFile $env:USERPROFILE\bin\digdag.bat}"
setx PATH "%PATH%;%USERPROFILE%\bin"
```

--------------------------------

### Setup Python Virtual Environment for Docs

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Creates a Python virtual environment and installs necessary libraries for building Digdag documentation using Sphinx.

```shell
python3 -m venv .venv
source .venv/bin/activate
pip install -r digdag-docs/requirements.txt -c digdag-docs/constraints.txt
```

--------------------------------

### Install Node.js using Homebrew

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Installs Node.js using the Homebrew package manager on macOS.

```shell
brew install node
```

--------------------------------

### Install Node.js using nodebrew

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Installs a specific version of Node.js (v12.x) using the nodebrew package manager, including setting up the PATH environment variable.

```shell
curl -L git.io/nodebrew | perl - setup
export PATH=$HOME/.nodebrew/current/bin:$PATH
source ~/.bashrc
nodebrew install-binary v12.x
nodebrew use v12.x
```

--------------------------------

### Run Digdag Server with Swagger

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Starts the Digdag server with the `--enable-swagger` option to expose the REST API documentation.

```shell
./gradlew cli
./pkg/digdag-<current version>.jar server --memory --enable-swagger
```

--------------------------------

### Digdag ParamGet Operator Example

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/param_get.md

This example demonstrates how to use the param_get> operator to retrieve single and multiple values from a ParamServer and then display them using the sh> operator. It shows the basic syntax for fetching data and making it available in subsequent tasks.

```yaml
+get_single_value:
  param_get>:
  key1: key_of_store_parameter1

+get_multiple_values:
  param_get>:
  key2: key_of_store_parameter2
  key3: key_of_store_parameter3

+show_gotten_data:
  sh>: echo '${key_of_store_parameter1} ${key_of_store_parameter2} ${key_of_store_parameter3}'
```

--------------------------------

### Fix Digdag SSL Certificate Error

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/getting_started.md

Resolves SSL certificate verification errors encountered with curl on certain Linux distributions by copying the correct certificate bundle.

```shell
sudo mkdir -p /etc/pki/tls/certs
sudo cp /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt
```

--------------------------------

### Digdag Client: Start Command Changes

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.5.0.rst

Highlights the changes in the 'start' command for Digdag's client mode. The '--now' or zoned timestamp argument is replaced by the '--session' argument, and a '--revision' option is added to specify a past revision.

```bash
digdag start --session YYYY-MM-DDTHH:MM:SSZ ...
```

```bash
digdag start --revision <name> ...
```

--------------------------------

### Create and Push Digdag Workflow

Source: https://github.com/treasure-data/digdag/blob/master/examples/REST_API.md

Creates a simple Digdag workflow file named 'resttest.dig' and pushes it to the Digdag server. The workflow is configured to echo a parameter named 'msg'.

```shell
mkdir resttest
cd resttest
cat << 'EODAG' > resttest.dig
timezone: Europe/Zurich

+echoparam:
  echo>: ${msg}
EODAG

digdag push resttest
```

--------------------------------

### Making HTTP GET Request

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/http.md

This example demonstrates how to make a simple HTTP GET request to a specified URI and store the response content.

```APIDOC
## POST /api/users

### Description
This endpoint allows you to create a new user in the system.

### Method
POST

### Endpoint
/api/users

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - The maximum number of users to return.

#### Request Body
- **name** (string) - Required - The name of the user.
- **email** (string) - Required - The email address of the user.

### Request Example
```json
{
  "name": "John Doe",
  "email": "john.doe@example.com"
}
```

### Response
#### Success Response (201)
- **id** (integer) - The unique identifier of the newly created user.
- **name** (string) - The name of the user.
- **email** (string) - The email address of the user.

#### Response Example
```json
{
  "id": 123,
  "name": "John Doe",
  "email": "john.doe@example.com"
}
```

#### Error Response (400)
- **message** (string) - A message describing the error.

#### Error Response Example
```json
{
  "message": "Email is required."
}
```
```

--------------------------------

### Backfill Digdag Schedule Sessions

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Starts sessions of a schedule for past session times. It requires a start time and optionally allows specifying the number of sessions to start and a unique name for the backfill attempts. A dry-run option is available for validation.

```console
$ digdag backfill <schedule-id>
$ digdag backfill <project-name> <name>

:command:`-f, --from 'yyyy-MM-dd[ HH:mm:ss]'`
  Timestamp to start backfill from (required). Sessions from this time (including this time) until current time will be started.

  Example: ``--from '2016-01-01'``

:command:`--count N`
  Starts given number of sessions. By default, this command starts all sessions until current time.

  Example: ``--count 5``

:command:`--name NAME`
  Unique name of the new attempts (required). This name is used not to run backfill sessions twice accidentally.

  Example: ``--name backfill1``

:command:`-d, --dry-run`
  Tries to backfill and validates the results but does nothing.
```

--------------------------------

### Initialize Digdag Project (`digdag init`)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Creates a new Digdag workflow project in a specified directory. It generates sample files like .dig, scripts, and .gitignore. You can specify an example project type, such as 'echo', 'sh', 'ruby', 'python', 'td', or 'postgresql'.

```console
$ digdag init <dir>
$ digdag init mydag
$ digdag init -t sh
```

--------------------------------

### Setup Treasure Data API Key and Database in Digdag

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/tutorials/bigdata_analytics_using_treasure_data.rst

This YAML snippet demonstrates how to set up authentication with Treasure Data by defining the API key and default database. It then shows three steps that execute SQL queries against the specified database.

```yaml
_export:
  td:
    apikey: YOUR/API_KEY
    database: www_access

+step1:
  td>: queries/step1.sql

+step2:
  td>: queries/step2.sql
  create_table: mytable_${session_date_compact}

+step3:
  td>: queries/step2.sql
  insert_into: mytable
```

--------------------------------

### bq_extract> Location Configuration Example

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq_extract.md

Provides an example of setting the 'location' parameter for the bq_extract> operator. This ensures that the BigQuery job runs in the specified geographic location, which must match the table and destination location.

```yaml
location: asia-northeast1
```

--------------------------------

### Digdag Command: new (Python 3 Compatibility)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.8.2.rst

Fixes the sample workflow generated by the `new` command to ensure compatibility with Python 3. This change targets users who generate new workflows and intend to use them with a Python 3 environment. It ensures that newly created projects start with a functional Python 3 setup.

```bash
# Example usage of the 'new' command (specific code not provided in release notes)
digdag new my_project
```

--------------------------------

### bq_extract> Compression Configuration Examples

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq_extract.md

Shows how to configure compression for the exported files using the 'compression' parameter with the bq_extract> operator. Options include GZIP and NONE.

```yaml
compression: NONE
```

```yaml
compression: GZIP
```

--------------------------------

### Digdag Workflow Scheduling (YAML)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/scheduling_workflow.rst

Example of setting up a daily schedule for a Digdag workflow. The `schedule` directive specifies the execution frequency and time.

```yaml
timezone: UTC

schedule:
  daily>: 07:00:00

+step1:
  sh>: tasks/shell_sample.sh
```

--------------------------------

### Configure Legacy SQL for BigQuery in Digdag

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq.md

This example shows how to configure the bq> operator to use legacy SQL instead of the default standard SQL for BigQuery queries. This is useful when migrating from older BigQuery setups or when specific legacy SQL features are required.

```yaml
_export:
  bq:
    use_legacy_sql: true
```

--------------------------------

### Launch Workflow via REST API

Source: https://github.com/treasure-data/digdag/blob/master/examples/REST_API.md

This endpoint allows you to launch a Digdag workflow by making a PUT request to the /api/attempts endpoint. You need to provide the workflow ID, session time, and any parameters for the workflow.

```APIDOC
## PUT /api/attempts

### Description
Launches a Digdag workflow attempt.

### Method
PUT

### Endpoint
`/api/attempts`

### Parameters
#### Query Parameters
None

#### Request Body
- **workflowId** (string) - Required - The ID of the workflow to launch.
- **sessionTime** (string) - Required - The session time for the workflow attempt in ISO 8601 format (e.g., "2019-05-01T13:38:52+09:00").
- **params** (object) - Optional - A JSON object containing parameters to pass to the workflow.
  - **msg** (string) - Example parameter for demonstration.

### Request Example
```json
{
  "params": {
    "msg": "Hello from REST API."
  },
  "sessionTime": "2019-05-01T13:38:52+09:00",
  "workflowId": "1"
}
```

### Response
#### Success Response (200)
- **id** (string) - The ID of the created attempt.
- **workflowId** (string) - The ID of the workflow that was attempted.
- **state** (string) - The state of the attempt (e.g., "running").
- **sessionTime** (string) - The session time of the attempt.

#### Response Example
```json
{
  "id": "attempt_12345",
  "workflowId": "1",
  "state": "running",
  "sessionTime": "2019-05-01T13:38:52+09:00"
}
```
```

--------------------------------

### Push Digdag Project

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Creates a project archive and uploads it to the Digdag server. This command uploads workflow definition files (.dig) and all other files from the current directory recursively. Options include specifying the project directory, revision, start time for schedules, and handling symlinks.

```console
$ digdag push <project> [options...]

Examples:

.. code-block:: console

    $ digdag push myproj -r "$(date +%Y-%m-%dT%H:%M:%S%z)"
    $ digdag push default -r "$(git show --pretty=format:'%T' | head -n 1)"

:command:`--project DIR`
  Use this directory as the project directory (default: current directory).

  Example: ``--project workflow/``

:command:`-r, --revision REVISION`
  Unique name of the revision. If this is not set, a random UUID is automatically generated. Typical argument is git's SHA1 hash (``git show --pretty=format:'%T' | head -n 1``) or timestamp (``date +%Y-%m-%dT%H:%M:%S%z``).

  Example: ``-r f40172ebc58f58087b6132085982147efa9e81fb``

:command:`--schedule-from "yyyy-MM-dd HH:mm:ss Z"`
  Start schedules from this time. If this is not set, system time of the server is used. Parameter must include time zone offset. You can run ``date "+%Y-%m-%d %H:%M:%S %z"`` command to get current local time.

  Example: ``--schedule-from "2017-07-29 00:00:00 +0200"``

:command:`--copy-outgoing-symlinks`
  Transform symlinks to regular files or directories if the symlink points a file or directory outside of the target directory. Without this option, such case fails because the files or directories won't be included unless copying.

  Example: ``--copy-outgoing-symlinks``

Digdag excludes files start with dot (``.``) from project archives.
```

--------------------------------

### Start Digdag Session with Custom Variables

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/workflow_definition.rst

Demonstrates how to pass custom variables to a Digdag workflow when starting a new session using the command line. The '-p' flag allows specifying key-value pairs, which can be used within the workflow. Multiple '-p' flags can be used to set multiple variables.

```console
$ digdag run -p my_var1=foo -p my_var2=abc
```

--------------------------------

### Digdag Server: New API Endpoint (GET /api/workflows/{id})

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.5.0.rst

Details the addition of the GET /api/workflows/{id} endpoint for retrieving workflow details.

```http
GET /api/workflows/{id}
```

--------------------------------

### Digdag Server Command with Task Logging

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.2.6.rst

This command starts the Digdag server or scheduler with an option to store task logs in a specified directory. This enables the new per-task logging feature.

```shell
digdag server -O, --task-log DIR
```

--------------------------------

### Digdag http> Example: Sending POST Request with JSON Content

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/http.md

This example illustrates how to make a POST request using the http> operator, sending dynamic session data as JSON content. It utilizes session variables to construct the request URI and body.

```yaml
+notify:
  http>: https://api.example.com/data/sessions/${session_uuid}
  method: POST
  content:
    status: RUNNING
    time: ${session_time}
```

--------------------------------

### Run Digdag Server

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Starts a Digdag server, which requires either the '--memory' or '--database' option for status storage. Available options control the server's port, bind address, status database, memory storage, task log directory, access log directory, and thread limits. It also includes options to disable local agent, executor loop, and scheduler.

```console
$ digdag server --memory
$ digdag server -o digdag-server
$ digdag server -o digdag-server -b 0.0.0.0
```

--------------------------------

### Configure Docker Command Executor with Custom Image Build

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_executor.md

This configuration demonstrates building a custom Docker image for the Docker Command Executor. It specifies the base image, commands to run during the build process (e.g., installing packages), and options for the `docker build` command.

```yaml
_export:
  docker:
    image: "azul/zulu-openjdk:8"
    docker: "/usr/local/bin/docker"
    run_options: [ "-m", "1G" ]
    build:
      - apt-get -y update
      - apt-get -y install software-properties-common
    build_options:
      - "--build-arg var1=test1"

+task1:
  py>: ...
```

--------------------------------

### Manage Multiple Databases in Digdag Tasks

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/tutorials/bigdata_analytics_using_treasure_data.rst

This example illustrates how to set a default database using `_export` and then override it for individual tasks. Step 1 targets 'mydb', while Step 2 uses the default 'www_access' database.

```yaml
_export:
  td:
    apikey: YOUR/API_KEY
    database: www_access

+step1:
  # this task uses mydb
  td>: queries/step1.sql
  database: mydb  # overwrites database setting

+step2:
  # this task uses www_access
  td>: queries/step2.sql
  create_table: mytable_${session_date_compact}
```

--------------------------------

### Run Digdag Workflow (`digdag run`)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Executes a Digdag workflow. You can specify a workflow file, a specific task to start from, or a task to end at. Options include setting the project directory, saving session status, rerunning tasks, specifying session times, limiting threads, storing logs, and passing parameters via command line or files. Dry-run and showing parameters are also supported.

```console
$ digdag run <workflow.dig> [+task] [options...]
$ digdag run workflow.dig
$ digdag run workflow.dig +step2
$ digdag run another.dig --start +step2 --end +step4
$ digdag run another.dig -g +step1 --hour
$ digdag run workflow.dig -p environment=staging -p user=frsyuki
$ digdag run workflow.dig --session hourly
$ digdag run workflow.dig --project workflow/
$ digdag run workflow.dig -o .digdag/status
$ digdag run workflow.dig --rerun
$ digdag run workflow.dig --start +step2
$ digdag run workflow.dig --goal +step2
$ digdag run workflow.dig --end +step4
$ digdag run workflow.dig --session 2016-01-01
$ digdag run workflow.dig --no-save
$ digdag run workflow.dig --max-task-threads 5
$ digdag run workflow.dig -O log/tasks
$ digdag run workflow.dig -p environment=staging
$ digdag run workflow.dig -P params.yml
$ digdag run workflow.dig -d
$ digdag run workflow.dig -dE
```

--------------------------------

### PostgreSQL ParamServer Configuration

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/param_set.md

Configuration example for connecting to a ParamServer using PostgreSQL. It specifies the database type, host, user, password, and database name. Other optional parameters like timeouts and SSL are also configurable.

```properties
param_server.type=postgresql
param_server.host=my_params.example.com
param_server.user=serizawa
param_server.password=QD8-_7nE4eMoaZ4FbKE2pA
param_server.database=digdag_param_server
```

--------------------------------

### Define Python Arguments in Digdag

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/py.md

Shows the Python code corresponding to the YAML examples for passing arguments. This includes class constructors and method signatures, with type hints for clarity.

```python
# tasks.py
from typing import Union


class MyWorkflow(object):
    def __init__(
      self,
      required1_1: str,
      required1_2: str,
      required2: dict[str, str],
      required3: int,
      required4: float,
      required5: list[Union[str, int, float]]
    ):
        print(f"{required1_1} same as {required1_2}")
        self.arg2 = required2
        print(f"{float(required3)} same as {required4}")
        self.arg5 = required5
    
    def my_task(self):
        pass
```

```python
# simple_tasks.py
def my_func(required1: str, required2: dict[str, str]):
  print(f"{required1}: {required2}")
```

```python
# tasks.py
class MyWorkflow:
  def __init__(self, required_class_arg: str):
    self.arg = required_class_arg
  
  def my_task(self, required_method_arg: list[str]):
    print(f"{self.arg}: {required_method_arg}")
```

--------------------------------

### Digdag Workflow Definition Example (YAML)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/workflow_definition.rst

A sample Digdag workflow file demonstrating timezone configuration and task definitions using shell, Python, and Ruby operators. This format is used for defining the sequence and execution logic of tasks.

```yaml
timezone: UTC

+step1:
  sh>: tasks/shell_sample.sh

+step2:
  py>: tasks.MyWorkflow.step2
  param1: this is param1

+step3:
  rb>: MyWorkflow.step3
  require: tasks/ruby_sample.rb
```

--------------------------------

### Run Digdag Scheduler

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Starts the Digdag scheduler to run workflows periodically. It scans for '.dig' files in the current directory. Options include specifying the project directory, port, bind address, database for status storage, task log directory, maximum task threads, and session parameters.

```console
$ digdag scheduler
$ digdag scheduler -d status
$ digdag scheduler -b 0.0.0.0
```

--------------------------------

### Digdag for_each Operator Example

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/for_each.md

Demonstrates the basic usage of the for_each operator in Digdag to repeat tasks. It iterates through a combination of 'fruit' and 'verb' variables, generating multiple echo tasks.

```yaml
+repeat:
  for_each>:
    fruit: [apple, orange]
    verb: [eat, throw]
  _do:
    echo>: ${verb} ${fruit}
```

--------------------------------

### Add --type flag to digdag init

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.8.16.rst

This change introduces a `--type` flag to the `digdag init` command, allowing users to specify the type of project to generate. This enhances the initial project setup process.

```shell
digdag init --type [project_type]
```

--------------------------------

### Redis ParamServer Configuration

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/param_set.md

Configuration example for connecting to a ParamServer using Redis. It includes the database type, host, password, and an SSL flag. These settings are essential for establishing a connection to the Redis instance.

```properties
param_server.type=redis
param_server.host=my_params.example.com
param_server.password=AQ5e5EvdiVzlLNsDI0Pm4A
param_server.ssl=true
```

--------------------------------

### Update Digdag Executable

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Updates the Digdag executable binary to the latest version or a specific version. This command simplifies keeping the Digdag installation up-to-date.

```console
$ digdag selfupdate
$ digdag selfupdate 0.10.5
```

--------------------------------

### Example Digdag Workflow with bq_extract>

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq_extract.md

Demonstrates a simple Digdag workflow that includes a BigQuery query execution and subsequent data export using the bq_extract> operator. This snippet shows how to define a dataset, execute a SQL query, and export the results to Google Cloud Storage with GZIP compression.

```yaml
_export:
  bq:
    dataset: my_dataset

+process:
  bq>: queries/analyze.sql
  destination_table: result

+export:
  bq_extract>: result
  destination: gs://my_bucket/result.csv.gz
  compression: GZIP
```

--------------------------------

### S3 Wait Operator with Separate Bucket and Key Parameters

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/s3_wait.md

This example illustrates using the `bucket` and `key` options separately to define the S3 file to wait for, offering an alternative to specifying the path directly in the operator.

```yaml
+wait_with_params:
  s3_wait>:
    bucket: another-bucket
    key: path/to/your/file.txt
```

--------------------------------

### bq_extract> Destination Format Examples

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq_extract.md

Illustrates the different 'destination_format' options available for the bq_extract> operator, including CSV, NEWLINE_DELIMITED_JSON, and AVRO.

```yaml
destination_format: CSV
```

```yaml
destination_format: NEWLINE_DELIMITED_JSON
```

```yaml
destination_format: AVRO
```

--------------------------------

### Making HTTP POST Request with Content

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/http.md

This example shows how to make an HTTP POST request with a JSON payload and specific headers.

```APIDOC
## POST /api/sessions/{session_uuid}/status

### Description
This endpoint updates the status of a specific session.

### Method
POST

### Endpoint
/api/sessions/${session_uuid}/status

### Parameters
#### Path Parameters
- **session_uuid** (string) - Required - The unique identifier of the session.

#### Request Body
- **status** (string) - Required - The new status of the session (e.g., "RUNNING").
- **time** (string) - Optional - The timestamp associated with the status update.

### Request Example
```json
{
  "status": "RUNNING",
  "time": "${session_time}"
}
```

### Response
#### Success Response (200)
- **message** (string) - A confirmation message indicating the status update was successful.

#### Response Example
```json
{
  "message": "Session status updated successfully."
}
```
```

--------------------------------

### Digdag Dry Run Option

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Performs a dry run of starting a new session attempt. This validates the process and results without executing any tasks or making changes.

```console
-d, --dry-run
```

--------------------------------

### Launch Digdag Workflow via REST API with curl

Source: https://github.com/treasure-data/digdag/blob/master/examples/REST_API.md

Invokes a pushed Digdag workflow using the REST API with curl. It specifies the workflow ID, session time, and parameters to be passed to the workflow. The 'workflowId' should be replaced with the actual ID obtained after pushing the workflow.

```shell
curl -X PUT "http://localhost:65432/api/attempts" \
       -H  "accept: application/json" \
       -H  "Content-Type: application/json" \
       -d "{ \"params\": { \"msg\": \"Hello from REST API.\" }, \"sessionTime\": \"2019-05-01T13:38:52+09:00\", \"workflowId\": \"1\"}"
```

--------------------------------

### S3 Wait Operator Basic Usage

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/s3_wait.md

This example demonstrates the basic usage of the s3_wait> operator to wait for a file in an S3 bucket. It specifies the bucket and key directly in the operator definition.

```yaml
+wait:
  s3_wait>: my-bucket/my-key
```

--------------------------------

### Digdag Secret Encryption Key Generation

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Demonstrates how to generate a base64 encoded 128-bit AES encryption key for Digdag secrets. It shows an example using openssl to create a 16-byte phrase and then encode it.

```bash
digdag.secret-encryption-key = MDEyMzQ1Njc4OTAxMjM0NQ==
# example
echo -n '16_bytes_phrase!' | openssl base64
MTZfYnl0ZXNfcGhyYXNlIQ==
```

--------------------------------

### Digdag http> Example: Fetching and Processing Data

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/http.md

This snippet demonstrates using the http> operator to fetch data from an API and then process the response content. It shows how to store the response and iterate over it for further actions.

```yaml
+fetch:
  http>: https://api.example.com/foobars
  store_content: true

+process:
  for_each>:
    foobar: ${http.last_content}
  _do:
    bq>: query.sql
```

--------------------------------

### bq_extract> Table Reference Examples

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq_extract.md

Illustrates various ways to specify the BigQuery table to be exported using the bq_extract> operator. This includes simple table names, dataset-qualified names, and project-qualified names.

```yaml
bq_extract>: my_table
```

```yaml
bq_extract>: my_dataset.my_table
```

```yaml
bq_extract>: my_project:my_dataset.my_table
```

--------------------------------

### Define Python Tasks in Digdag

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/py.md

Illustrates how to structure Python code for use with the Digdag py> operator. It shows examples for running class methods and standalone functions, including directory structures and Python script content.

```python
# __init__.py
class MyWorkflow(object):
    def my_task(self):
        print("awesome execution")
```

```python
# tasks.py
class MyWorkflow(object):
    def my_task(self):
        print("awesome execution")
```

```python
# simple_tasks.py
def my_func():
  print("simple execution")
```

--------------------------------

### Digdag Variable Substitution Example

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/workflow_definition.rst

Illustrates how to embed Digdag's built-in session and task variables within workflow definitions using the ${variable_name} syntax. These variables provide dynamic information about the current session and task.

```text
# Example usage of built-in variables:
# Timezone: ${timezone}
# Session UUID: ${session_uuid}
# Task Name: ${task_name}

# Example usage with scheduled jobs:
# Last Session Time: ${last_session_time}
# Next Session Time: ${next_session_time}
```

--------------------------------

### ECS Command Executor Setup and Configuration

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_executor.md

Configuration settings for the AWS ECS Command Executor, including cluster name, AWS credentials, launch type, region, subnets, and retry settings. It also includes configuration for temporal storage using AWS S3.

```properties
agent.command_executor.ecs.name = digdag-test
 agent.command_executor.ecs.digdag-test.access_key_id = <ACCESS KEY>
 agent.command_executor.ecs.digdag-test.secret_access_key = <SECRET KEY>
 agent.command_executor.ecs.digdag-test.launch_type = FARGATE
 agent.command_executor.ecs.digdag-test.region = us-east-1
 agent.command_executor.ecs.digdag-test.subnets = subnet-NNNNN
 agent.command_executor.ecs.digdag-test.max_retries = 3

 agent.command_executor.ecs.temporal_storage.type = s3
 agent.command_executor.ecs.temporal_storage.s3.bucket = <Bucket>
 agent.command_executor.ecs.temporal_storage.s3.endpoint = s3.amazonaws.com
 agent.command_executor.ecs.temporal_storage.s3.credentials.access-key-id = <ACCESS KEY>
 agent.command_executor.ecs.temporal_storage.s3.credentials.secret-access-key = <SECRET KEY>
```

--------------------------------

### Digdag: Conditional Execution with Else Block (if>)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/if.md

This example showcases the `if>` operator with both `_do` and `_else_do` blocks in Digdag. The `_do` tasks execute if the condition is true, while the `_else_do` tasks execute if the condition is false. This allows for comprehensive branching logic within a Digdag workflow, handling all possible boolean outcomes of a parameter or expression. It requires Digdag to be installed and configured.

```yaml
+run_if_param_is_false:
  if>: ${param}
  _do:
    echo>: ${param} == true
  _else_do:
    echo>: ${param} == false
```

--------------------------------

### Run Swagger-UI Locally

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Runs Swagger-UI using Docker and opens the Digdag API documentation in the browser.

```shell
docker run -dp 8080:8080 swaggerapi/swagger-ui
open http://localhost:8080/?url=http://localhost:65432/api/swagger.json
```

--------------------------------

### bq_extract> Destination URI Examples

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq_extract.md

Shows how to configure the destination for exported BigQuery data using the 'destination' parameter with the bq_extract> operator. Supports single and multiple Google Cloud Storage URIs.

```yaml
destination: gs://my_bucket/my_export.csv
```

```yaml
destination:
  - gs://my_bucket/my_export_1.csv
  - gs://my_bucket/my_export_2.csv
```

--------------------------------

### Perform Build, Site Generation, and Release Checks with Gradle

Source: https://github.com/treasure-data/digdag/blob/master/README.md

This Gradle command executes a series of tasks including cleaning the build, building the CLI and site, and performing checks to ensure the release is ready.

```bash
./gradlew clean cli site check releaseCheck
```

--------------------------------

### Build Digdag Documentation

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Builds the Digdag documentation locally using Gradle. It may require running `./gradlew clean` first if dependencies are not updated correctly.

```shell
(.venv)$ ./gradlew site
```

--------------------------------

### Get Workflow API

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.5.0.rst

The GET /api/workflows/{id} endpoint is now available to retrieve details for a specific workflow using its ID.

```APIDOC
## GET /api/workflows/{id}

### Description
Retrieves details for a specific workflow.

### Method
GET

### Endpoint
/api/workflows/{id}

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the workflow.
```

--------------------------------

### Digdag http> Example: Custom Headers

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/http.md

This code snippet shows how to include custom headers in an HTTP request made by the http> operator. It demonstrates adding 'Accept', 'X-Foo', and 'Baz' headers to the request.

```yaml
headers:
  - Accept: application/json
  - X-Foo: bar
  - Baz: quux
```

--------------------------------

### Get Truncated Session Time API

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.5.0.rst

A new GET /api/workflows/{id}/truncated_session_time endpoint has been added. This API calculates session time using the workflow's timezone, useful for preparing new session attempts.

```APIDOC
## GET /api/workflows/{id}/truncated_session_time

### Description
Calculates a session time using the specified workflow's time zone. This API is useful for preparing a new session attempt.

### Method
GET

### Endpoint
/api/workflows/{id}/truncated_session_time

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the workflow.

#### Query Parameters
- **session_time** (string) - Required - The base session time in ISO 8601 format.
```

--------------------------------

### Example Treasure Data Query for td_wait>

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/td_wait.md

An example SQL query that can be used with the td_wait> operator. This query checks if at least one record exists in 'target_table' within the session time, returning true if found.

```sql
select 1 from target_table where TD_TIME_RANGE(time, '${session_time}') limit 1
```

--------------------------------

### bq_extract> Field Delimiter Configuration Example

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/bq_extract.md

Demonstrates how to set a custom field delimiter for the exported data using the 'field_delimiter' option with the bq_extract> operator. This example uses a tab character as the delimiter.

```yaml
field_delimiter: "\t"
```

--------------------------------

### Format Session Time with Moment.js in Digdag

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/workflow_definition.rst

Demonstrates how to format timestamps using the bundled Moment.js library within Digdag YAML configurations. It shows examples for standard formatting, UTC conversion, adding days, and capturing the current execution time. Requires Digdag environment with Moment.js available.

```yaml
timezone: America/Los_Angeles

+format_session_time:
  # "2016-09-24 00:00:00 -0700"
  echo>: ${moment(session_time).format("YYYY-MM-DD HH:mm:ss Z")}

+format_in_utc:
  # "2016-09-24 07:00:00"
  echo>: ${moment(session_time).utc().format("YYYY-MM-DD HH:mm:ss")}

+format_tomorrow:
  # "September 24, 2016 12:00 AM"
  echo>: ${moment(session_time).add(1, 'days').format("LLL")}

+get_execution_time:
  # "2016-09-24 05:24:49 -0700"
  echo>: ${moment().format("YYYY-MM-DD HH:mm:ss Z")}
```

--------------------------------

### Create and Push Release Branch

Source: https://github.com/treasure-data/digdag/blob/master/README.md

This command sequence creates a new Git branch for the release, commits the changes, and pushes the branch to the origin repository. This is part of the release preparation process.

```bash
git checkout -b release_v<version>
git push origin release_v<version>
```

--------------------------------

### Digdag http> Example: XML Content with Content-Type Override

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/http.md

This example demonstrates sending XML content with the http> operator. It specifies the content as an XML string and explicitly sets the 'Content-Type' header to 'application/xml', overriding the inferred type.

```yaml
content: |
  <?xml version="1.0" encoding="UTF-8"?>
  <notification>
    <status>RUNNING</status>
    <time>${session_time}</time>
  </notification>
content_format: text
content_type: application/xml
```

--------------------------------

### Run Digdag Tests

Source: https://github.com/treasure-data/digdag/blob/master/README.md

Executes all tests using Gradle and generates test coverage and Findbugs reports.

```shell
./gradlew check
```

--------------------------------

### Digdag Server: New API Endpoint (GET /api/workflows/{id}/truncated_session_time)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.5.0.rst

Introduces a new API endpoint, GET /api/workflows/{id}/truncated_session_time, designed to calculate session times using a workflow's timezone, aiding in the preparation of new session attempts.

```http
GET /api/workflows/{id}/truncated_session_time
```

--------------------------------

### Embed Digdag Server with ServerBootstrap

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.4.2.rst

The `io.digdag.server.ServerBootstrap` class provides functionality to embed the Digdag server within a custom application. This allows for greater control over the server's lifecycle and integration into existing systems. No specific inputs or outputs are detailed, but it implies programmatic control.

```java
io.digdag.server.ServerBootstrap
```

--------------------------------

### Enable Digdag Metrics (JMX, Fluentd)

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/metrics.md

Configuration to enable the Digdag metrics framework, specifying JMX and Fluentd as enabled metrics exporters.

```properties
metrics.enable = jmx,fluency
```

--------------------------------

### Sending Email with Digdag Mail Operator

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/mail.md

Demonstrates how to use the Digdag mail operator to send emails. It shows examples of specifying the email body from a file or directly within the .dig file, setting the subject, and defining recipients for 'to', 'cc', and 'bcc' fields. Also includes an example of sending an email upon task failure.

```digdag
_export:
  mail:
    from: "you@gmail.com"

+step1:
  mail>: body.txt
  subject: workflow started
  to: [me@example.com]
  cc: [foo@example.com,bar@example.com]

+step2:
  mail>:
    data: this is email body embedded in a .dig file
  subject: workflow started
  to: [me@example.com]
  bcc: [foo@example.com,bar@example.com]

+step3:
  sh>: this_task_might_fail.sh
  _error:
    mail>: body.txt
    subject: this workflow failed
    to: [me@example.com]
```

--------------------------------

### Digdag CLI Parameters

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Describes command-line parameters for Digdag, such as setting environment variables and loading parameters from files.

```APIDOC
## Digdag CLI Parameters

### Description
Configuration options for the Digdag command-line interface, including setting environment variables and loading parameters from YAML files.

### Parameters

#### Query Parameters

- **-p environment** (string) - Sets the environment for the project. Note: Variables defined in `_export` are not overwritable by `--param`.
- **-P, --params-file PATH** (string) - Reads parameters from a YAML file. Nested parameters can be accessed using dot notation (e.g., `${mysql.user}`).
- **-c, --config PATH** (string) - Specifies the configuration file to load. Defaults to `~/.config/digdag/config`.

### Request Example

```bash
-p environment=staging
-P params.yml
-c digdag-server/server.properties
```
```

--------------------------------

### GET /api/projects/{id}/revisions

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.9.6.rst

Retrieves revisions for a given project. Includes a new 'userInfo' field.

```APIDOC
## GET /api/projects/{id}/revisions

### Description
Retrieves revisions for a specific project. This endpoint has been updated to include a `userInfo` field in its response.

### Method
GET

### Endpoint
/api/projects/{id}/revisions

### Parameters
#### Path Parameters
- **id** (integer) - Required - The ID of the project.

### Response
#### Success Response (200)
- **userInfo** (object) - Information about the user associated with the revision.
- **other_fields** (any) - Other fields related to project revisions.
```

--------------------------------

### Download Digdag Project

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Downloads a specified Digdag project archive and extracts it to a local directory. Options allow specifying an output directory and a specific project revision.

```console
$ digdag download <project>
$ digdag download myproj
$ digdag download myproj -o output
$ digdag download myproj -r rev20161106
```

--------------------------------

### GET /api/sessions/{id}/attempts

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.9.6.rst

Retrieves attempts for a given session. Includes a new 'index' field for attempts.

```APIDOC
## GET /api/sessions/{id}/attempts

### Description
Retrieves attempts for a specific session. This endpoint now includes an `index` field for each attempt, representing its sequence number within the session.

### Method
GET

### Endpoint
/api/sessions/{id}/attempts

### Parameters
#### Path Parameters
- **id** (integer) - Required - The ID of the session.

### Response
#### Success Response (200)
- **index** (integer) - The sequence number of the attempt within the session (1, 2, 3, ...).
- **other_fields** (any) - Other fields related to session attempts.
```

--------------------------------

### Digdag Client-Mode Common Options

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/command_reference.rst

Lists common command-line options for Digdag client-mode commands. These include specifying the server endpoint, adding custom HTTP headers, basic authentication, and loading configuration files.

```console
:command:`-e, --endpoint HOST`
  HTTP endpoint of the server (default: http://127.0.0.1:65432)

  Example: ``--endpoint digdag-server.example.com:65432``

:command:`-H, --header KEY=VALUE`
  Add a custom HTTP header. Use multiple times to set multiple headers.

:command:`--basic-auth <user:pass>`
  Add an Authorization header with the provided username and password.

:command:`-c, --config PATH`
  Configuration file to load. (default: ~/.config/digdag/config)

  Example: ``-c digdag-server/client.properties``
```

--------------------------------

### GET /api/repositories/{id}/revisions

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/releases/release-0.4.3.rst

Retrieves a list of revisions for a specific repository.

```APIDOC
## GET /api/repositories/{id}/revisions

### Description
Retrieves a list of revisions for a specific repository identified by its ID.

### Method
GET

### Endpoint
/api/repositories/{id}/revisions

### Parameters
#### Path Parameters
- **id** (string) - Required - The unique identifier of the repository.

### Response
#### Success Response (200)
- **revisions** (array) - A list of revision objects, where each object typically contains a revision ID and timestamp.

#### Response Example
{
  "revisions": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "timestamp": "2023-10-27T10:00:00Z"
    },
    {
      "id": "87654321-e89b-12d3-a456-426614174000",
      "timestamp": "2023-10-26T15:30:00Z"
    }
  ]
}
```

--------------------------------

### Run a Shell Command with sh>

Source: https://github.com/treasure-data/digdag/blob/master/digdag-docs/src/operators/sh.md

Demonstrates how to execute a simple shell command 'echo "hello world"' using the sh> operator in Digdag.

```yaml
+step1:
  sh>: echo "hello world"
```