Tinybird Forward

https://www.tinybird.co/docs/forward/
Admin
Tinybird Forward is a new CLI-based user experience for Tinybird, designed to help developers build,...
Tokens:353,089
Snippets:3,107
Trust Score:9.4
Update:1 week ago
Show doc for...
Tokens:
Raw
### Install and start Tinybird CLI for local development

Source: https://www.tinybird.co/docs/forward/guides/migrate-from-clickhouse

Commands to install the Tinybird CLI, authenticate, and start the local development environment. Run these in order during the local development setup step.

```bash
# Install Tinybird CLI
curl https://tinybird.co | sh

# Authenticate
tb login

# Start local development environment
tb local start

```

--------------------------------

### Tinybird Local startup logs

Source: https://www.tinybird.co/docs/forward/development-workflow/local-development

Example output from 'tb local start' showing the status of the container and services, ending with 'Tinybird Local is ready!'.

```bash
» Starting Tinybird Local...
* Tinybird Local container status: healthy
» Checking services...
✓ Tinybird Local container
✓ Clickhouse
✓ Redis
✓ Server
✓ Events
✓ Tinybird Local authentication
✓ Tinybird Local is ready!
```

--------------------------------

### Build output example

Source: https://www.tinybird.co/docs/forward/development-workflow/local-development

Example output from a successful build, showing created datasources and endpoints.

```bash
» Building project...
✓ datasources/user_actions.datasource created
✓ endpoints/user_actions_line_chart.pipe created
✓ Build completed in 0.2s
```

--------------------------------

### tb command examples

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/tb

Examples of common `tb` commands including help, version, init, and deploy.

```bash
tb --help
tb --version
tb init --type cli --dev-mode manual
tb --cloud deploy

```

--------------------------------

### Install tinybird-sdk with pip

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli

Install the Python SDK package using pip. The `tinybird` command becomes available after installation.

```bash
pip install tinybird-sdk
```

--------------------------------

### Run a SQL query with GET

Source: https://www.tinybird.co/docs/forward/query-data/sql-api

Use GET for short queries, passing the SQL in the q query parameter. Requires a Token with read permissions; tokens without a read scope get a 403 error.

```bash
curl \
  -H "Authorization: Bearer <read token or JWT>" \
  --get \
  --data-urlencode "q=SELECT * FROM <pipe_or_datasource> FORMAT JSON" \
  https://api.tinybird.co/v0/sql
```

--------------------------------

### Start Docker containers

Source: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka

Starts the Docker containers defined in the docker-compose.yml file.

```bash
docker compose up

```

--------------------------------

### Start branch dev with connections

Source: https://www.tinybird.co/docs/forward/core-concepts/branches

Starts a branch development server with connector data sources enabled using the `--with-connections` flag.

```bash
tb --branch=preview_1 dev --with-connections

```

--------------------------------

### tinybird.config.json example

Source: https://www.tinybird.co/docs/forward/dev-reference/config-files

Example tinybird.config.json file showing dev_mode, branch_data_mode, folder, and include settings.

```json
{
  "dev_mode": "branch",
  "branch_data_mode": "last_partition",
  "folder": "tinybird",
  "include": [
    "tinybird"
  ]
}
```

--------------------------------

### Data source example

Source: https://www.tinybird.co/docs/forward/dev-reference/datafiles

Example of a .datasource file defining a schema with columns and engine settings.

```plaintext
SCHEMA >
    `day` Date `json:$.day`,
    `station` String `json:$.station`,
    `abslevel` Nullable(Float32) `json:$.abslevel`,
    `percentagevolume` Float32 `json:$.percentagevolume`,
    `volume` Float32 `json:$.volume`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "station"
ENGINE_SORTING_KEY "day, station"
```

--------------------------------

### Start local server and rebuild with Tinybird CLI

Source: https://www.tinybird.co/docs/forward/development-workflow/examples

Start the local server in daemon mode and rebuild the project after editing resources.

```bash
tb local start --daemon
tb build

```

--------------------------------

### Run the demo app

Source: https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs

Starts the Next.js development server. Open localhost:3000 in your browser.

```bash
npm run dev

```

--------------------------------

### tinybird preview examples

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli

Create a preview environment and deploy project resources. Options include `--dry-run`, `--check`, and `--name`.

```bash
tinybird preview
```

```bash
tinybird preview --dry-run
```

```bash
tinybird preview --check
```

```bash
tinybird preview --name pr_482
```

--------------------------------

### error function example

Source: https://www.tinybird.co/docs/forward/guides/advanced-dynamic-endpoints-functions

Returns a custom error message when a parameter is not defined. Place at the start of a node after optional set lines.

```sql
%
{% if not defined(event_types) %}
  {{error('You need to provide a value for event_types')}}
{% end %}
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE event IN {{Array(event_types, 'String')}}

```

--------------------------------

### Initialize with options

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-init

Run `tb init` with options to set project type to CLI, development mode to manual, folder to current directory, and generate GitHub CI/CD templates.

```bash
tb init --type cli --dev-mode manual --folder . --cicd github
```

--------------------------------

### tinybird init examples

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli

Initialize a project with different options: `--dev-mode`, `--cicd`, `--folder`, and `--skip-login`.

```bash
tinybird init --dev-mode branch --cicd skip --folder tb_project
```

```bash
tinybird init --dev-mode local --cicd github --folder tb_project
```

```bash
tinybird init --dev-mode branch --cicd skip --folder tb_project --skip-login
```

--------------------------------

### Get workspace ID with tb workspace current

Source: https://www.tinybird.co/docs/forward/core-concepts/workspaces

Run `tb workspace current` to find the workspace ID. Example output shows the current workspace details.

```bash
tb workspace current
```

```text
** Current workspace:
--------------------------------------------------------------------------------------------
| name                   | id                                   | role  | plan   | current |
--------------------------------------------------------------------------------------------
| analytics_prod         | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | admin | Custom | True    |
--------------------------------------------------------------------------------------------
```

--------------------------------

### Copy .env.example to .env.local

Source: https://www.tinybird.co/docs/forward/guides/consume-apis-nextjs

Creates a new .env.local file from the example file. Run this in the cloned repo directory.

```bash
cp .env.example .env.local

```

--------------------------------

### Start Kafka datasource in branch

Source: https://www.tinybird.co/docs/forward/development-workflow/using-cloud-branches

Kafka ingestion starts stopped in branches. Start it only when ready to test. Each start creates a new consumer group and starts from the latest offset.

```bash
tb --branch=add-plan-to-events datasource start my_kafka_datasource

```

--------------------------------

### Calculate average storage for billing period

Source: https://www.tinybird.co/docs/forward/monitoring/organization-consumption

Query to calculate the average of daily maximum storage across a billing period, using the organization.datasources_storage table. It computes daily maximums per datasource, then aggregates by date to get daily organization totals, and finally averages those. Replace the date placeholders ('2025-04-XX') with your actual billing period start and end dates.

```sql
SELECT 
    greatest(avg(daily_max_org_storage_gb), 0) as avg_storage_gb,
    greatest(avg(daily_max_org_storage_rows), 0) as avg_storage_rows
FROM (
  SELECT 
    sum(floor(max_total_bytes_by_ds/1000000000, 6)) as daily_max_org_storage_gb,
    sum(max_total_rows_by_ds) as daily_max_org_storage_rows
  FROM (
    SELECT 
        toDate(timestamp) as date, 
        max(bytes + bytes_quarantine) as max_total_bytes_by_ds,
        max(rows + rows_quarantine) AS max_total_rows_by_ds
    FROM organization.datasources_storage
    WHERE 1= 1
        AND date >= '2025-04-XX' -- beginning of term
        AND date <= '2025-04-xx' -- end of term
    GROUP BY date, datasource_id
  )
  GROUP BY date
)
```

--------------------------------

### tb datasource start

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource

Starts Kafka ingestion for a Data Source. Only available for Kafka-connected Data Sources in branches or Tinybird Local. Once started, data is ingested from the Kafka topic.

```APIDOC
## tb datasource start

### Description
Starts Kafka ingestion for a Data Source. Only available for Kafka-connected Data Sources in branches or Tinybird Local. Once started, data is ingested from the Kafka topic. In branches, this creates a new consumer group and starts ingestion from the latest offset. In Tinybird Local, ingestion resumes from the last committed offset.

### Method
CLI command

### Command
`tb [--branch=BRANCH_NAME] datasource start my_kafka_datasource`

### Parameters
- **branch** (string) - Optional - Branch name to run the command in.
- **datasource** (string) - Required - The name of the Kafka-connected data source.

### Notes
Because each start creates a new consumer group, previous consumer groups become orphaned. Depending on your Kafka cluster's consumer group TTL (`offsets.retention.minutes`), these orphan groups may persist until they expire. This is expected behavior for ephemeral branch environments.

```

--------------------------------

### Develop and deploy with the SDK CLI

Source: https://www.tinybird.co/docs/forward/guides/after-migrating-from-classic

Run the dev server, check the deployment, and deploy to the main Workspace.

```bash
npx tinybird dev
npx tinybird deploy --check
npx tinybird deploy
```

--------------------------------

### Initialize a new project

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-init

Run `tb init` to scaffold a new Tinybird project with default settings.

```bash
tb init
```

--------------------------------

### Start Tinybird Local with generated tokens

Source: https://www.tinybird.co/docs/forward/core-concepts/tinybird-local

Pass generated Token values when starting Tinybird Local using `tb local start` with the `--user-token` and `--workspace-token` flags.

```bash
tb local start --user-token=<USER_TOKEN> --workspace-token=<WORKSPACE_TOKEN>
```

--------------------------------

### Initialize the project with npx tinybird init

Source: https://www.tinybird.co/docs/forward/guides/after-migrating-from-classic

Run the init command to create configuration files and authenticate.

```bash
npx tinybird init
```

--------------------------------

### Start and stop Kafka datasource in preview branch

Source: https://www.tinybird.co/docs/forward/development-workflow/cicd

Commands to start or stop a Kafka datasource in a preview branch. Kafka connections are stopped by default in preview branches; start them when ready to test.

```bash
tb --branch=tmp_ci_my_feature datasource start my_kafka_datasource
tb --branch=tmp_ci_my_feature datasource stop my_kafka_datasource

```

--------------------------------

### Example project structure with fixtures folder

Source: https://www.tinybird.co/docs/forward/guides/test-your-project

Shows the expected layout of a Tinybird project with datasources and fixtures folders.

```text
my-app/
├─ datasources/
│  ├─ user_actions.datasource
│  └─ ...
├─ fixtures/
│  ├─ user_actions.ndjson
│  └─ ...
```

--------------------------------

### Start and stop Kafka ingestion in branch

Source: https://www.tinybird.co/docs/forward/core-concepts/branches

Starts and stops Kafka ingestion for a branch. Each start creates a new consumer group with a unique ID, so ingestion starts from the latest offset. Previous consumer groups become orphaned and may persist until they expire based on the Kafka cluster's consumer group TTL.

```bash
tb --branch=preview_1 datasource start my_kafka_datasource
# ... observe data flowing in, test your pipelines ...
tb --branch=preview_1 datasource stop my_kafka_datasource

```

--------------------------------

### Install ClickHouse Go client

Source: https://www.tinybird.co/docs/forward/guides/connect-clickhouse-go

Initialize a Go module and add the ClickHouse Go client dependency.

```bash
go mod init your-project
go get github.com/ClickHouse/clickhouse-go/v2
```

--------------------------------

### Start Docker Compose services

Source: https://www.tinybird.co/docs/forward/core-concepts/tinybird-local

Start the Tinybird Local container defined in the docker-compose.yaml file.

```bash
docker compose up -d
```

--------------------------------

### tinybird info examples

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli

Show project information. Options include `--skip-local`.

```bash
tinybird info
```

```bash
tinybird info --skip-local
```

--------------------------------

### Install ClickHouse Python client

Source: https://www.tinybird.co/docs/forward/guides/connect-clickhouse-python

Install the official ClickHouse Python client using pip.

```bash
pip install clickhouse-connect

```

--------------------------------

### Initialize Python project and add SDK

Source: https://www.tinybird.co/docs/forward/quickstarts/python-sdk

Create a new directory, initialize a Python project with uv, add the Tinybird SDK and python-dotenv, and initialize the Tinybird project with the CLI. This creates tinybird.config.json and .env.local.

```bash
mkdir tinybird-python-quickstart
cd tinybird-python-quickstart
git init

uv init --name tinybird-sdk-python-sample --python 3.13
uv add tinybird-sdk python-dotenv

uv run tinybird init --folder src/tinybird/
```

--------------------------------

### tinybird build examples

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli

Validate and build the current project. Options include `--watch` and `--with-connections`.

```bash
tinybird build
```

```bash
tinybird build --watch
```

```bash
tinybird build --with-connections
```

--------------------------------

### API endpoint wiki_stream_example SQL with parameters

Source: https://www.tinybird.co/docs/forward/guides/consume-apis-in-a-notebook

SQL for the 'wiki_stream_example' endpoint. It selects data from the 'wiki' table within a date range, then applies batch boundaries using the batch_no and batch_size parameters, and optionally filters on bot and server_name columns. The parameters are defined with defaults and descriptions.

```sql
%
SELECT * from wiki
--DATE RANGE
WHERE timestamp BETWEEN toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}}))
AND toUInt64(toDateTime({{String(date_end, '2022-02-13 10:35:00', description="end")}}))
--BATCH BEGIN
AND timestamp BETWEEN toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}})
              + interval {{Int16(batch_no, 1, description="batch number")}}
              * {{Int16(batch_size, 10, description="size of the batch")}} second)
--BATCH END
AND toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}})
              + interval ({{Int16(batch_no, 1, description="batch number")}} + 1)
              * {{Int16(batch_size, 10, description="size of the batch")}} second)
--FILTERS
{% if defined(bot) %}
 AND bot = {{String(bot, description="is a bot")}}
{% end %}
{% if defined(server_name) %}
 AND server_name = {{String(server_name, description="server")}}
{% end %}
```

--------------------------------

### Example error query output

Source: https://www.tinybird.co/docs/forward/monitoring/analyze-endpoints-performance

Example output from the error query showing pipe_name, status_code, and error_count.

```text
Pipe_a | 404 | 127
Pipe_b | 403 | 32

```

--------------------------------

### Initialize Tinybird TypeScript project

Source: https://www.tinybird.co/docs/forward/quickstarts/typescript-sdk

Create a new directory, initialize a git repository, install the SDK, and run tinybird init to set up the project.

```bash
mkdir tinybird-typescript-quickstart
cd tinybird-typescript-quickstart
git init
pnpm add @tinybirdco/sdk

npx tinybird init
```

--------------------------------

### Install @clickhouse/client

Source: https://www.tinybird.co/docs/forward/guides/connect-clickhouse-js

Install the official ClickHouse JavaScript client via npm. Requires Node.js 16.0 or later.

```bash
npm install @clickhouse/client

```

--------------------------------

### .tinyb file structure example

Source: https://www.tinybird.co/docs/forward/dev-reference/config-files

Sample .tinyb file containing host, workspace id, name, scope, token, tokens, cwd, user_email, user_id, user_token, and version fields.

```json
{
  "host": "<tinybird-host>",
  "id": "<workspace-id>",
  "name": "<workspace-name>",
  "scope": "user",
  "token": "<authentication-token>",
  "tokens": {
    "<tinybird-host>": "<authentication-token>",
    "<another-tinybird-host>": "<authentication-token>"
  },
  "cwd": "<optional-path-to-your-tinybird-folder>",
  "user_email": "<user-email>",
  "user_id": "<user-id>",
  "user_token": "<authentication-token>",
  "version": "<tinybird-version>"
}
```

--------------------------------

### Install TypeScript SDK globally with pnpm

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/typescript-sdk-cli

Installs the @tinybirdco/sdk package globally so the tinybird CLI can be run directly.

```bash
pnpm add -g @tinybirdco/sdk
```

--------------------------------

### Create test topic and send sample messages

Source: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka/guides/local-development

Commands to create a Kafka topic and send sample JSON messages using kafka-topics.sh and kafka-console-producer.sh inside the broker container.

```bash
# Create a test topic
docker exec -it broker /opt/kafka/bin/kafka-topics.sh --create \
  --topic test-events \
  --bootstrap-server localhost:9092

# Send sample messages
cat <<EOF | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic test-events \
  --bootstrap-server localhost:9092
{"user_id": "123", "event": "click", "timestamp": "2024-01-01T00:00:00Z"}
{"user_id": "456", "event": "view", "timestamp": "2024-01-01T00:01:00Z"}
{"user_id": "123", "event": "purchase", "timestamp": "2024-01-01T00:02:00Z"}
EOF
```

--------------------------------

### tinybird dev examples

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/python-sdk-cli

Build server-side and watch for changes. Options include `--with-connections` and `--no-connections`.

```bash
tinybird dev
```

```bash
tinybird dev --with-connections
```

```bash
tinybird dev --no-connections
```

--------------------------------

### Start Kafka ingestion for a data source

Source: https://www.tinybird.co/docs/forward/dev-reference/commands/tb-datasource

Starts Kafka ingestion for a Kafka-connected Data Source in a branch or Tinybird Local. In branches, creates a new consumer group and starts from the latest offset; in Tinybird Local, resumes from last committed offset. Each start creates a new consumer group, leaving previous groups orphaned until they expire based on the cluster's offsets.retention.minutes.

```bash
tb [--branch=BRANCH_NAME] datasource start my_kafka_datasource
```

--------------------------------

### Create sample-topic

Source: https://www.tinybird.co/docs/forward/ingest-data/connectors/kafka

Creates the Kafka topic 'sample-topic' using the kafka-topics.sh script inside the broker container.

```bash
docker exec -it broker /opt/kafka/bin/kafka-topics.sh --create --topic sample-topic --bootstrap-server localhost:9092

# Created topic sample-topic.

```

--------------------------------

### Example query with LIMIT

Source: https://www.tinybird.co/docs/forward/query-data/query-parameters

Example query that adds a LIMIT clause, which automatically adds the rows_before_limit_at_least field to the response metadata.

```SQL
SELECT * FROM users WHERE active = true LIMIT 10
```
Tokens:
Raw