Pipekit

https://docs.pipekit.io
Admin
Pipekit is a control plane for Argo Workflows that enables platform teams to manage data and CI...
Tokens:39,704
Snippets:644
Trust Score:7.9
Update:5 months ago
Show doc for...
Tokens:
Raw
### Dynamic Documentation Query Example

Source: https://docs.pipekit.io/pipekit/organization/byo-logs

An example of how to query the documentation dynamically using an HTTP GET request with the 'ask' query parameter. This is useful for retrieving specific information not explicitly present on the page.

```http
GET https://docs.pipekit.io/pipekit/organization/byo-logs.md?ask=<question>
```

--------------------------------

### Install Pipekit CLI with Scoop (Windows)

Source: https://docs.pipekit.io/cli

Install the Pipekit CLI on Windows using Scoop. First, add the Pipekit repository, then install the CLI.

```bash
scoop bucket add pipekit https://github.com/pipekit/scoop.git
scoop install pipekit/cli
```

--------------------------------

### Key-Value / Logfmt Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of logs formatted using the key-value or logfmt style, which are recognized by Pipekit for parsing.

```logfmt
level=error msg="Connection failed"
severity=warning component=api
loglevel=debug request_id=abc123
```

--------------------------------

### Example Log Parsing

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Demonstrates how Pipekit parses log lines to identify severity and keywords. This example shows a warning being identified before a keyword in the message.

```text
68:5   warning  Error: Cannot access refs during render
       ^        ^
       |        |
       wins     loses (appears later)

Result: `warning` (appears at position 7, before `Error` at position 17)
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.pipekit.io/cli

To get information not directly present in the documentation, perform an HTTP GET request on the page URL with the `ask` query parameter. The question should be specific and self-contained.

```http
GET https://docs.pipekit.io/cli.md?ask=<question>
```

--------------------------------

### Querying Documentation via HTTP GET

Source: https://docs.pipekit.io/pipekit/pipes/managing-pipes/run-sync-conditions

To get additional information not present on the current page, you can perform an HTTP GET request to the page URL with the `ask` query parameter. The question should be specific and self-contained. The response will include a direct answer and relevant documentation excerpts.

```http
GET https://docs.pipekit.io/pipekit/pipes/managing-pipes/run-sync-conditions.md?ask=<question>
```

--------------------------------

### Query Documentation with GET Request

Source: https://docs.pipekit.io/pipekit/organization

To get additional information not directly present on a page, perform an HTTP GET request to the page URL with the `ask` query parameter. The question should be specific and self-contained.

```http
GET https://docs.pipekit.io/pipekit/organization.md?ask=<question>
```

--------------------------------

### Query Pipekit Agent Documentation via HTTP GET

Source: https://docs.pipekit.io/pipekit-agent

To get information not directly on the page, make an HTTP GET request to the current page URL with the `ask` query parameter. The question should be specific and self-contained.

```http
GET https://docs.pipekit.io/pipekit-agent.md?ask=<question>
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.pipekit.io/pipekit/metrics

Perform an HTTP GET request to query the documentation dynamically. Include your question in the `ask` query parameter. The response will contain a direct answer and relevant excerpts.

```http
GET https://docs.pipekit.io/pipekit/metrics.md?ask=<question>
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/workflow-logs

Perform an HTTP GET request on the current page URL with the `ask` query parameter to dynamically query documentation. The question should be specific and self-contained.

```http
GET https://docs.pipekit.io/pipekit/pipes/pipe-runs/workflow-logs.md?ask=<question>
```

--------------------------------

### Querying Documentation via HTTP GET

Source: https://docs.pipekit.io/self-hosting-pipekit/pipekit-agent-configuration

To get information not explicitly present on the page, perform an HTTP GET request to the page URL with an 'ask' query parameter containing your question.

```http
GET https://docs.pipekit.io/self-hosting-pipekit/pipekit-agent-configuration.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.pipekit.io/getting-started

To get information not explicitly on the page, make an HTTP GET request to the current URL with an 'ask' query parameter containing your question.

```http
GET https://docs.pipekit.io/getting-started.md?ask=<question>
```

--------------------------------

### Querying Documentation with 'ask' Parameter

Source: https://docs.pipekit.io/pipekit/organization/permissions

Use this GET request to ask questions about the documentation. The response includes direct answers and relevant excerpts.

```http
GET https://docs.pipekit.io/pipekit/organization/permissions.md?ask=<question>
```

--------------------------------

### Query Documentation with HTTP GET

Source: https://docs.pipekit.io/pipekit-agent/helm-chart

Perform an HTTP GET request to the current page URL with the 'ask' query parameter to dynamically query the documentation. Use this when the answer is not explicitly present, for clarification, or to retrieve related sections.

```http
GET https://docs.pipekit.io/pipekit-agent/helm-chart.md?ask=<question>
```

--------------------------------

### Query Pipekit Documentation Dynamically

Source: https://docs.pipekit.io/pipekit/runs

Perform an HTTP GET request on a documentation page URL with the `ask` query parameter to get dynamic answers. The question should be specific and in natural language.

```http
GET https://docs.pipekit.io/pipekit/runs.md?ask=<question>
```

--------------------------------

### Bracketed Format Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of logs using bracketed prefixes to denote log levels, such as [ERROR] or [WARN]. Pipekit recognizes these for parsing.

```text
[ERROR] Connection failed
[WARN] Retry attempt 3
[INFO] Server started
[E] Short error format
[W] Short warning format
```

--------------------------------

### Install Pipekit CLI with Homebrew (macOS & Linux)

Source: https://docs.pipekit.io/cli

Use this command to install the Pipekit CLI on macOS and Linux systems using the Homebrew package manager.

```bash
brew install pipekit/tap/cli
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.pipekit.io/pipekit/pipes/managing-pipes/alerting

To get information not directly on the page, make an HTTP GET request to the current URL with the `ask` query parameter. The question should be specific and in natural language.

```http
GET https://docs.pipekit.io/pipekit/pipes/managing-pipes/alerting.md?ask=<question>
```

--------------------------------

### Syslog Format Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of logs adhering to the Syslog format, including priority values and message content. Pipekit supports parsing these logs.

```syslog
<3>Mar  6 12:00:00 myhost myapp: Connection failed
<27>Error message with facility
```

--------------------------------

### Timestamp-Prefixed Logs Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of logs that include a timestamp prefix, followed by the log level and message. Pipekit can parse these formats.

```text
2024-01-15T10:30:00Z ERROR Connection failed
2024-01-15 10:30:00 WARN Retry attempt
2024/01/15T10:30:00.123+00:00 INFO Server started
```

--------------------------------

### Install Pipekit Agent with Helm

Source: https://docs.pipekit.io/pipekit-agent/helm-chart

Install the Pipekit Agent using Helm, providing your Secret Access Key and Cluster ID. Ensure you have created the 'argo' namespace beforehand.

```bash
kubectl create ns argo
helm repo add pipekit https://helm.pipekit.io

helm upgrade -i -n argo \
pipekit-agent pipekit/pipekit-agent \
--set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \
--set secrets.pipekitClusterId="[provided Cluster ID]"
```

--------------------------------

### Query Pipekit Documentation via API

Source: https://docs.pipekit.io/self-hosting-pipekit/dependencies-and-prerequisites

Demonstrates how to query the Pipekit documentation dynamically using an HTTP GET request with the 'ask' query parameter.

```http
GET https://docs.pipekit.io/self-hosting-pipekit/dependencies-and-prerequisites.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.pipekit.io/additional-information/free-trial-cluster

Perform an HTTP GET request to the current page URL with the 'ask' query parameter to dynamically query the documentation. The question should be specific and self-contained.

```HTTP
GET https://docs.pipekit.io/additional-information/free-trial-cluster.md?ask=<question>
```

--------------------------------

### Colon-Separated Format Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of logs using a colon-separated format, where the log level precedes the message. Pipekit recognizes this structure.

```text
ERROR: Connection failed
WARN: Retry attempt 3
INFO: Server started on port 8080
```

--------------------------------

### Querying Documentation Dynamically

Source: https://docs.pipekit.io/self-hosting-pipekit/integrating-with-your-git-provider

Perform an HTTP GET request to query the documentation dynamically. Use the `ask` query parameter with a specific, self-contained question in natural language. The response will include a direct answer and relevant excerpts.

```http
GET https://docs.pipekit.io/self-hosting-pipekit/integrating-with-your-git-provider.md?ask=<question>
```

--------------------------------

### Querying Documentation Dynamically

Source: https://docs.pipekit.io/self-hosting-pipekit/container-images

Perform an HTTP GET request to query the documentation dynamically using the `ask` query parameter. This is useful for retrieving specific information not explicitly present on the page.

```http
GET https://docs.pipekit.io/self-hosting-pipekit/container-images.md?ask=<question>
```

--------------------------------

### Role Precedence Example

Source: https://docs.pipekit.io/pipekit/organization/access-control

Illustrates how higher-privilege roles apply when a user belongs to both an org-scoped and a workspace-scoped group.

```shell
idp:team:platform  → Org Admin   → all workspaces
idp:team:foo       → Viewer      → team-foo-dev

# A platform engineer in both groups gets Admin in team-foo-dev, not Viewer
```

--------------------------------

### Pipe-Separated Format Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of logs formatted with pipe separators, typically used to delineate application name, log level, and message. Pipekit parses these.

```text
myapp | ERROR | Connection failed
service | WARN | High memory usage
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.pipekit.io/pipekit/authentication/okta

Perform an HTTP GET request on a documentation page URL, including the 'ask' query parameter with a natural language question to retrieve specific information.

```http
GET https://docs.pipekit.io/pipekit/authentication/okta.md?ask=<question>
```

--------------------------------

### Namespace Permissions Configuration Example

Source: https://docs.pipekit.io/pipekit/organization/permissions

Define user access to Kubernetes namespaces across different clusters. Supports wildcard namespaces.

```yaml
version: v1alpha1
users:
  - username: test-user
    config:
      - clusterName: local-cluster
        namespaces:
          - default
          - dev
      - clusterName: local-cluster-2
        namespaces:
          - '*'
  - username: test-user-2
    config:
      - clusterName: local-cluster
        namespaces:
          - default
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs

Perform an HTTP GET request to query documentation dynamically. Use the 'ask' query parameter with a specific, self-contained question in natural language.

```http
GET https://docs.pipekit.io/pipekit/pipes/pipe-runs.md?ask=<question>
```

--------------------------------

### Query Documentation Dynamically

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/run-graph

Perform an HTTP GET request to query documentation dynamically. Use the 'ask' query parameter with a specific, self-contained question in natural language. The response includes a direct answer and relevant excerpts.

```HTTP
GET https://docs.pipekit.io/pipekit/pipes/pipe-runs/run-graph.md?ask=<question>
```

--------------------------------

### Example Workflow Using Secrets

Source: https://docs.pipekit.io/pipekit/pipes/managing-pipes/secrets

This example demonstrates how to access a secret, MY_SECRET, which is passed as an environment variable to the workflow. It checks if the secret's value matches the expected value and logs the result.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: test-secret
spec:
  entrypoint: test-secret
  templates:
    - name: test-secret
      container:
        image: ubuntu
        command:
            - /bin/bash
            - -c
            - |
            if [ "${MY_SECRET}" == "my-secret-value" ]
            then
                echo "Secrets are working!"
                echo ${MY_SECRET}
            else
                echo "Secrets are not working!"
                echo ${MY_SECRET}
                exit 1
            fi

```

--------------------------------

### Query Documentation via API

Source: https://docs.pipekit.io/python-sdk/jupyter-notebooks

To get additional information not directly present on a page, perform an HTTP GET request to the page URL with an 'ask' query parameter containing your question.

```http
GET https://docs.pipekit.io/python-sdk/jupyter-notebooks.md?ask=<question>
```

--------------------------------

### Apply Pipekit Helm Chart to Kubernetes Cluster

Source: https://docs.pipekit.io/self-hosting-pipekit/helm-chart

Create the 'pipekit' namespace and upgrade or install the Pipekit Helm chart using a custom values file.

```bash
kubectl create ns pipekit
helm upgrade -i -n pipekit \
-f myvalues.yaml \
pipekit pipekit/pipekit
```

--------------------------------

### Plain Text Fallback Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of plain text logs where log level keywords are detected anywhere within the message. This serves as a fallback when structured formats are not identified.

```text
Connection error occurred
Warning: disk space low
This is an info message about the system
```

--------------------------------

### Grant Workflow Repository Pull Access Example

Source: https://docs.pipekit.io/pipekit/pipes/managing-pipes

Use this example to grant a Workflow access to the git repository containing its Workflow yaml. The generated SSH key is mounted at `/root/.ssh/ssh-deploy-key`.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: clone-repo-
spec:
  entrypoint: clone-repo
  volumeClaimTemplates:
  - metadata:
      name: code
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName: nfs
      resources:
        requests:
          storage: 1Gi
  templates:
  - name: clone-repo
    container:
      image: alpine
      command:
        - sh
        - -c
        - |
          apk --update add openssh-client git
          eval `ssh-agent -s`
          mkdir -p /workdir/src/github.com/my_org
          cd /workdir/src/github.com/my_org
          ssh-add /root/.ssh/ssh-deploy-key
          ssh-keyscan github.com > /root/.ssh/known_hosts
          git config --global --add safe.directory '*'
          git clone git@github.com:my_org/my_repo.git
          cd my_repo
          git checkout $GIT_COMMIT
      volumeMounts:
      - name: code
        mountPath: /workdir
      resources:
        requests:
          memory: 1Gi
          cpu: 100m

```

--------------------------------

### Android Logcat Style Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of logs formatted in the Android Logcat style, using a single letter for the log level followed by the tag and message. Pipekit supports this format.

```text
E/MyApp: Connection failed
W/MyService: Retry attempt
I/MainActivity: onCreate called
```

--------------------------------

### Install Pipekit Agent with External Workflows Enabled

Source: https://docs.pipekit.io/pipekit-agent/helm-chart

Install or upgrade the Pipekit Agent Helm chart, setting necessary secrets and enabling external workflows. This command also creates the 'argo' namespace if it doesn't exist.

```bash
kubectl create ns argo
helm repo add pipekit https://helm.pipekit.io

helm upgrade -i -n argo \
pipekit-agent pipekit/pipekit-agent \
--set secrets.pipekitSecretAccessKey="[provided Secret Access Key]" \
--set secrets.pipekitClusterId="[provided Cluster ID]" \
--set configMap.allowExternalWorkflows=true
```

--------------------------------

### Copy Pipekit CLI Binary into a Docker Container

Source: https://docs.pipekit.io/cli

Example Dockerfile instruction to copy the Pipekit CLI binary from a builder image into your target container, such as a Jupyter notebook environment.

```dockerfile
FROM jupyter/scipy-notebook
COPY --from=pipekit13/cli /usr/local/bin/pipekit /usr/local/bin/pipekit
```

--------------------------------

### Org-Scoped Role Bindings

Source: https://docs.pipekit.io/pipekit/organization/access-control

Examples of assigning roles to IdP groups at the organization scope, which applies to all workspaces.

```shell
# Org-scoped — applies to all workspaces
idp:team:platform        → Org Admin       → all workspaces
idp:team:ops             → Viewer          → all workspaces
idp:team:shared-tools    → Editor          → all workspaces

# Workspace-scoped — applies to one workspace only
idp:team:data-engineers  → Runner          → team-data-dev
idp:team:data-engineers  → Viewer          → team-data-prod
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.pipekit.io/pipekit/authentication

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. The question should be specific, self-contained, and written in natural language.

```http
GET https://docs.pipekit.io/pipekit/authentication.md?ask=<question>
```

--------------------------------

### Get Existing Helm Chart Values

Source: https://docs.pipekit.io/self-hosting-pipekit/helm-chart

Retrieve the currently applied values for your Pipekit Helm installation in a specific namespace.

```bash
helm get values pipekit -n pipekit
```

--------------------------------

### Add Pipekit Helm Repository and Show Default Values

Source: https://docs.pipekit.io/self-hosting-pipekit/helm-chart

Add the Pipekit Helm repository and display the default values to create a local configuration file.

```bash
helm repo add pipekit https://helm.pipekit.io
helm show values pipekit/pipekit > myvalues.yaml
```

--------------------------------

### Query Documentation API

Source: https://docs.pipekit.io/pipekit/organization/access-control

To get information not directly present on the page, make a GET request to the page URL with an 'ask' query parameter containing your question.

```http
GET https://docs.pipekit.io/pipekit/organization/access-control.md?ask=<question>
```

--------------------------------

### JSON Structured Logs Examples

Source: https://docs.pipekit.io/pipekit/pipes/pipe-runs/pod-logs

Examples of JSON formatted logs that Pipekit can parse. Recognized field names for log level include 'level', 'severity', and 'lvl'.

```json
{"level": "error", "message": "Connection failed"}
{"severity": "warn", "msg": "Retry attempt 3"}
{"lvl": "INFO", "event": "User logged in"}
```

--------------------------------

### Create Postgres Databases for Pipekit

Source: https://docs.pipekit.io/self-hosting-pipekit/dependencies-and-prerequisites

Use these SQL commands to create the necessary databases for the Pipekit 'id' and 'users' services.

```bash
CREATE DATABASE users;
CREATE DATABASE id;
```

--------------------------------

### Initialize Pipekit Service with Token

Source: https://docs.pipekit.io/python-sdk/jupyter-notebooks

Create a PipekitService instance by passing an authentication token. This service object is used for all subsequent interactions with the Pipekit API.

```python
from pipekit_sdk.service import PipekitService

# Create a Pipekit service that is used to talk to the Pipekit API
pipekit = PipekitService(token="<token>")
```

--------------------------------

### Terraform Plan Generation Workflow Step

Source: https://docs.pipekit.io/use-cases/infrastructure-management

This step initializes Terraform, generates a plan, and summarizes it for review. It requires AWS credentials and mounts a working directory for source code and plan outputs.

```yaml
    - name: tfplan
      inputs:
        parameters:
          - name: path
      container:
        image: <your-registry>/terraform
        command:
          - bash
          - -c
          - |
            mkdir -p /tfplan-dir
            mkdir -p /workdir/terraform
            cp -R /workdir/src/github.com/<your-org>/<your-repo> /tfplan-dir
            cd /tfplan-dir/<your-repo>/{{inputs.parameters.path}}/
            tofu init
            tofu plan -lock-timeout=600s -out=tfplan
            tofu show -json tfplan | tf-summarize > /workdir/{{inputs.parameters.path}}-plan.txt
            tofu show tfplan
            tf-summarize tfplan
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-secret-access-key
        volumeMounts:
        - name: workdir
          mountPath: /workdir
```

--------------------------------

### Create a Workflow Template for Terraform Apply

Source: https://docs.pipekit.io/use-cases/infrastructure-management

This WorkflowTemplate defines an 'apply' entrypoint that takes a Terraform path as input and applies changes using `tofu apply -auto-approve`. It includes environment variables for AWS credentials, ensuring consistent and secure deployments.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: terraform-apply
spec:
  entrypoint: apply
  templates:
    - name: apply
      inputs:
        parameters:
          - name: terraform-path
      container:
        image: <your-registry>/terraform
        command:
          - bash
          - -c
          - |
            cd /workdir/{{inputs.parameters.terraform-path}}/
            tofu apply -auto-approve
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-access-key-id
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: <your-secret>
                key: aws-secret-access-key
```

--------------------------------

### Get Run Information by UUID

Source: https://docs.pipekit.io/cli

Retrieve detailed information about a specific workflow run using its unique identifier (UUID).

```bash
$> pipekit get run --run-uuid=6b0b0b1a-1b1a-4b1a-9b1a-1b1a1b1a1b1a
```

--------------------------------

### Add Pipekit Helm Repository and Search Versions

Source: https://docs.pipekit.io/self-hosting-pipekit/helm-chart

Add the Pipekit Helm repository and update it to search for available versions of the Pipekit chart.

```bash
# Add the repo if you don't already have it.
helm repo add pipekit https://helm.pipekit.io
helm repo update
helm search repo pipekit/pipekit --versions
```

--------------------------------

### Extract Current Helm Installation Values

Source: https://docs.pipekit.io/pipekit-agent/helm-chart

Retrieve the currently applied Helm values for the 'pipekit-agent' release in the 'argo' namespace. This is helpful for backing up or comparing configurations.

```bash
helm get values pipekit-agent -n argo
```

--------------------------------

### Remote Cluster Configuration (External URLs)

Source: https://docs.pipekit.io/self-hosting-pipekit/pipekit-agent-configuration

Use this configuration when the Pipekit Agent is on a different cluster from your Pipekit installation. It utilizes external ingress URLs for secure communication via HTTPS.

```yaml
configMap:
  messengerBaseUri: "https://pipekit.example.com"
  usersBaseUri: "https://pipekit.example.com"
```

--------------------------------

### Get Hera token for Python SDK

Source: https://docs.pipekit.io/cli

Use the `pipekit hera` command to obtain an authentication token for running Hera workflows and CronWorkflows through Pipekit using the Pipekit Python SDK.

```bash
$> pipekit hera
Token:   eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
```

--------------------------------

### Search for Pipekit Agent Helm Chart Versions

Source: https://docs.pipekit.io/pipekit-agent/helm-chart

Add the Pipekit Helm repository and search for available versions of the pipekit-agent chart. This helps in identifying the latest version or specific versions for upgrades.

```bash
# Add the repo if you don't already have it.
helm repo add pipekit https://helm.pipekit.io
helm repo update
helm search repo pipekit/pipekit-agent --versions
```
Tokens:
Raw