StreamNative

https://docs.streamnative.io/
Admin
StreamNative is a cloud-native, real-time data platform powered by Apache Pulsar and Apache Kafka...
Tokens:549,148
Snippets:6,384
Trust Score:9
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install Kafka Go client

Source: https://docs.streamnative.io/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-go

Use go get to install the confluent-kafka-go library.

```bash
go get -u github.com/confluentinc/confluent-kafka-go/v2/kafka
```

--------------------------------

### Start ClickHouse Server

Source: https://docs.streamnative.io/connect/connectors/jdbc-clickhouse-sink/current/jdbc-sink

Commands to download and start a single-node ClickHouse server instance.

```bash
curl https://clickhouse.com/ | sh
./clickhouse server
```

--------------------------------

### Install Prometheus Adapter Chart

Source: https://docs.streamnative.io/private-cloud/v2/configure-private-cloud/advanced/private-cloud-autoscaling

Installs the Prometheus Adapter using the provided configuration file.

```bash
helm install prometheus-adapter -f sample-config.yaml prometheus-community/prometheus-adapter -n monitor
```

--------------------------------

### Install KafkaJS Client

Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-create-project

Installs the kafkajs library via npm.

```bash
npm install kafkajs
```

--------------------------------

### Full Provider Configuration Example

Source: https://docs.streamnative.io/tools/terraform/configure/configure-streamnative-provider

A complete example combining provider definition, authentication, and data source validation.

```hcl
terraform {
  required_providers {
    streamnative = {
      source  = "streamnative/streamnative"
    }
  }
}

provider "streamnative" {
  client_id     = "<client-id>"
  client_secret = "<client-secret>"
}

data "streamnative_service_account" "test-sa" {
  organization = "<organization>"
  name         = "<service-account-name>"
}

output "service_account_id" {
  value = data.streamnative_service_account.test-sa
}
```

--------------------------------

### Namespace Topic Examples

Source: https://docs.streamnative.io/cloud/manage-data-streams/data-streams-overview

Examples of topic paths demonstrating how different applications are grouped under distinct namespaces within a tenant.

```bash
persistent://tenant/app1/topic-1
persistent://tenant/app2/topic-2
persistent://tenant/app3/topic-3
```

--------------------------------

### Install ADK and Orca dependencies

Source: https://docs.streamnative.io/agent-engine/develop-agents/agents-develop-adk

Set up a virtual environment and install the necessary libraries for ADK agent development.

```bash
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools
pip install google-adk
pip install orca-agent-engine
```

--------------------------------

### Install pfSQL CLI via Homebrew

Source: https://docs.streamnative.io/cloud/process/pfsql/pfsql-work-with-cli

Use these commands to tap the StreamNative repository and install the pfSQL CLI tool.

```bash
brew tap streamnative/streamnative
brew install pfsql-cli
```

--------------------------------

### Create User Response Example

Source: https://docs.streamnative.io/api-references/cloudapi/cloud-api

Example JSON response returned after a successful user creation request.

```json
{
  "kind": "User",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
		"name": "<your-instance-name>",
		"namespace": "<your-organization-name>",
    "uid": "ff7fbc4f-4037-4d6d-afb4-4304d57b33e8",
    "resourceVersion": "3110657",
    "generation": 1,
    "creationTimestamp": "2024-05-11T12:06:02Z",
  },
  "spec": {
    "email": "<user-email>"
  },
  "status": {}
}
```

--------------------------------

### Install Agent Dependencies

Source: https://docs.streamnative.io/agent-engine/develop-agents/agents-develop-openai

Commands to set up a virtual environment and install the necessary SDKs for developing Orca-compatible agents.

```bash
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools
pip install openai-agents
pip install openai
pip install orca-agent-engine
```

--------------------------------

### Create and enter tutorial directory

Source: https://docs.streamnative.io/tools/cli/streamnative-cli-tutorial

Initializes a workspace for the tutorial by creating and navigating into a new directory.

```bash
mkdir snctl-getting-started && cd snctl-getting-started
```

--------------------------------

### Start KSQL CLI

Source: https://docs.streamnative.io/cloud/process/kafka-streams-and-ksql/cloud-connect-ksql

Initialize the KSQL CLI tool pointing to the local server instance.

```shell
LOG_DIR=./ksql_logs bin/ksql http://localhost:8088
```

--------------------------------

### Example CRD List Output

Source: https://docs.streamnative.io/tools/operator/quickstart-guides/pulsar-operator-install-olm

Expected output showing the installed Pulsar-related CRDs.

```bash
NAME                                            CREATED AT
bookkeeperclusters.bookkeeper.streamnative.io   2023-03-02T08:52:52Z
pulsarbrokers.pulsar.streamnative.io            2023-03-02T08:52:52Z
pulsarproxies.pulsar.streamnative.io            2023-03-02T08:52:52Z
zookeeperclusters.zookeeper.streamnative.io     2023-03-02T08:52:53Z
```

--------------------------------

### Start KSQL Server

Source: https://docs.streamnative.io/cloud/process/kafka-streams-and-ksql/cloud-connect-ksql

Launch the KSQL server using the configured properties file.

```shell
bin/ksql-server-start etc/ksqldb/ksql-server.properties
```

--------------------------------

### Input messages for producer

Source: https://docs.streamnative.io/tools/cli/other-tools/use-kafka-tools-with-streamnative-cloud

Example of message input strings provided at the command prompt after starting the producer.

```bash
>hello world
>another message
>3rd message
```

--------------------------------

### Create Project Directory

Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c%2B%2B/tutorial/kafka-c-create-project

Initializes a new project directory and navigates into it.

```bash
mkdir kafka-c-getting-started && cd kafka-c-getting-started
```

--------------------------------

### Install and run Lakehouse sink in standalone mode

Source: https://docs.streamnative.io/connect/connectors/lakehouse-sink/current/lakehouse-sink

Commands to deploy the NAR package, start Pulsar, run the connector, and produce test messages.

```bash
cp pulsar-io-lakehouse-4.0.3.1.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-4.0.3.1.nar
```

```bash
PULSAR_HOME/bin/pulsar standalone
```

```bash
PULSAR_HOME/bin/pulsar-admin sink localrun \
--sink-config-file <lakehouse-sink-config.yaml>
```

```bash
PULSAR_HOME/bin/pulsar-client produce public/default/test-lakehouse-pulsar --messages hello -n 10
```

--------------------------------

### Install and run Lakehouse connector in standalone mode

Source: https://docs.streamnative.io/connect/connectors/lakehouse-source/current/lakehouse-source

Commands to deploy the NAR package, start Pulsar, run the connector, and consume output topics.

```bash
cp pulsar-io-lakehouse-4.0.3.1.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-4.0.3.1.nar
```

```bash
PULSAR_HOME/bin/pulsar standalone
```

```bash
PULSAR_HOME/bin/pulsar-admin sources localrun \
--source-config-file <lakehouse-source-config.yaml>
```

```bash
PULSAR_HOME/bin/pulsar-client consume -s test-sub -n 0 <topic-name>
```

--------------------------------

### Elasticsearch startup logs

Source: https://docs.streamnative.io/cloud/connect/self-host-connectors/cloud-connect-elasticsearch

Example output showing successful initialization of the Elasticsearch service.

```bash
...output omitted...
{"@timestamp":"2022-04-15T15:13:17.067Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[efb7b3360ba3][generic][T#7]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ocHgh5mAQROAlUofYHE3Cg","elasticsearch.node.id":"0aWiWmaBTgC0vdp6Zw_ZnQ","elasticsearch.node.name":"efb7b3360ba3","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-04-15T15:13:17.118Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[efb7b3360ba3][generic][T#13]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ocHgh5mAQROAlUofYHE3Cg","elasticsearch.node.id":"0aWiWmaBTgC0vdp6Zw_ZnQ","elasticsearch.node.name":"efb7b3360ba3","elasticsearch.cluster.name":"docker-cluster"}
```

--------------------------------

### Initialize Go Module and Install Dependencies

Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-create-project

Initializes the Go module and fetches the required Pulsar client packages.

```bash
go mod init pulsar-go-getting-started
go get github.com/apache/pulsar-client-go
go get github.com/apache/pulsar-client-go/pulsar/auth
go get github.com/apache/pulsar-client-go/pulsar
```

--------------------------------

### Verify pulsarctl installation

Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-overview

Check the installed version of pulsarctl to confirm successful installation.

```bash
pulsarctl --version
```

--------------------------------

### Setup Virtual Environment

Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-create-project

Create and activate a virtual environment to isolate project dependencies.

```bash
virtualenv venv
source venv/bin/activate
```

--------------------------------

### Verify snctl installation

Source: https://docs.streamnative.io/streamnative-cli/snctl-overview

Check the installed version of the snctl client.

```bash
snctl version --client
```

--------------------------------

### Install OLM

Source: https://docs.streamnative.io/private-cloud/v2/operate-private-cloud/deploy/private-cloud-olm

Execute the installation script for the Operator Lifecycle Manager.

```bash
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.25.0/install.sh | bash -s v0.25.0
```

--------------------------------

### Initialize Go module and install dependencies

Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-create-project

Initializes the Go module and downloads the required Confluent Kafka Go client library.

```bash
go mod init kafka-go-getting-started
go get github.com/confluentinc/confluent-kafka-go/v2/kafka
```

--------------------------------

### MQTT Topic Example

Source: https://docs.streamnative.io/cloud/build/mqtt-clients/mqtt-on-cloud-overview

An example of a multi-level MQTT topic name.

```text
/a/b/c/d/e/f
```

--------------------------------

### Install OLM

Source: https://docs.streamnative.io/tools/operator/quickstart-guides/pulsar-operator-install-olm

Executes the OLM installation script to set up the necessary namespaces.

```bash
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.23.1/install.sh | bash -s v0.23.1
```

--------------------------------

### Create a Basic Cloud Environment

Source: https://docs.streamnative.io/cloud/clusters/byoc/create-cloud-environment

Commands to initialize a cloud environment for AWS, GCP, or Azure using the snctl CLI.

```bash
snctl create cloudenvionment --cloud-connection-name CLOUD_CONNECTION_NAME \
  --region REGION \
  -n ORG_NAME
```

```bash
snctl create cloudenvironment --cloud-connection-name CLOUD_CONNECTION_NAME \
  --region REGION \
  -n ORG_NAME
```

```bash
snctl create cloudenvionment --cloud-connection-name CLOUD_CONNECTION_NAME \
  --region REGION \
  -n ORG_NAME
```

--------------------------------

### Provision the Instance

Source: https://docs.streamnative.io/tools/cli/streamnative-cli-tutorial

Command to apply the configuration file to the cluster.

```bash
snctl create -f 001-instance.yaml
```

--------------------------------

### Install Prometheus Chart

Source: https://docs.streamnative.io/private-cloud/v2/configure-private-cloud/advanced/private-cloud-autoscaling

Installs the Prometheus chart with specific components disabled.

```bash
helm install prometheus prometheus-community/prometheus -n monitor --set alertmanager.enabled=false --set kube-state-metrics.enabled=false --set prometheus-node-exporter.enabled=false --set prometheus-pushgateway.enabled=false
```

--------------------------------

### Start Pulsar Consumer

Source: https://docs.streamnative.io/connect/connectors/file-source/current/file-source

Command to start a consumer to verify the connector output.

```bash
./bin/pulsar-client consume -s file-test -n 0 pulsar-file-test
```

--------------------------------

### Start Canal Connector

Source: https://docs.streamnative.io/connect/connectors/canal-source/current/canal-source

Commands to download the connector and start it using pulsar-admin.

```bash
$ docker exec -it pulsar-standalone /bin/bash
$ wget https://archive.apache.org/dist/pulsar/pulsar-2.3.0/connectors/pulsar-io-canal-2.3.0.nar -P connectors
$ ./bin/pulsar-admin source localrun \
--archive ./connectors/pulsar-io-canal-2.3.0.nar \
--classname org.apache.pulsar.io.canal.CanalStringSource \
--tenant public \
--namespace default \
--name canal \
--destination-topic-name my-topic \
--source-config-file /pulsar/conf/canal-mysql-source-config.yaml \
--parallelism 1
```

--------------------------------

### Maven and JShell Environment Setup

Source: https://docs.streamnative.io/connect/connectors/pinecone-sink/current/pinecone-sink

Scripts for setting up the classpath and verifying the Java version for development.

```bash
mvn dependency:build-classpath -DincludeTypes=jar -Dmdep.outputFile=.cp.txt
jshell --class-path `cat .cp.txt`:target/classes
```

```bash
mvn --version # should be java 8
jenv exec mvn # if using jenv you can exec the local version using
              # this
```

--------------------------------

### KSQL Server Startup Output

Source: https://docs.streamnative.io/cloud/process/kafka-streams-and-ksql/cloud-connect-ksql

Expected console output after successfully starting the KSQL server.

```text
[2023-04-06 14:51:02,811] INFO ksqlDB API server listening on http://0.0.0.0:8088 (io.confluent.ksql.rest.server.KsqlRestApplication:382)

                  ===========================================
                  =       _              _ ____  ____       =
                  =      | | _____  __ _| |  _ \| __ )      =
                  =      | |/ / __|/ _` | | | | |  _ \      =
                  =      |   <\__ \ (_| | | |_| | |_) |     =
                  =      |_|\_\___/\__, |_|____/|____/      =
                  =                   |_|                   =
                  =        The Database purpose-built       =
                  =        for stream processing apps       =
                  ===========================================

Copyright 2017-2022 Confluent Inc.

Server 7.3.2 listening on http://0.0.0.0:8088

To access the KSQL CLI, run:
ksql http://0.0.0.0:8088

[2023-04-06 14:51:02,814] INFO Server up and running (io.confluent.ksql.rest.server.KsqlServerMain:153)
[2023-04-06 14:51:04,117] INFO Successfully submitted metrics to Confluent via secure endpoint (io.confluent.support.metrics.submitters.ConfluentSubmitter:146)
```

--------------------------------

### Verify kcctl setup

Source: https://docs.streamnative.io/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-setup

Check the status and configuration of the kcctl tool after setup.

```shell
> kcctl info
URL:               https://${KAFKA-SERVICE-URL}/admin/kafkaconnect/
Version:           3.7.0
Commit:            839b886f9b732b15
Kafka Cluster ID:  connect
```

--------------------------------

### Provision Serverless Instance with snctl

Source: https://docs.streamnative.io/cloud/clusters/manage-instances/manage-serverless-instances

Command to apply the instance configuration file.

```bash
snctl create -f instance.yaml
```

--------------------------------

### Install snctl with Homebrew

Source: https://docs.streamnative.io/streamnative-cli/snctl-overview

Add the StreamNative repository and install snctl using Homebrew.

```bash
brew tap streamnative/streamnative
```

```bash
brew install snctl
```

--------------------------------

### View Installed CRDs

Source: https://docs.streamnative.io/tools/operator/quickstart-guides/pulsar-operator-install-helm-chart

List the Custom Resource Definitions installed by the Pulsar Operators.

```bash
kubectl get crds | grep streamnative
```

```bash
NAME                                            CREATED AT
bookkeeperclusters.bookkeeper.streamnative.io   2023-03-02T08:52:52Z
pulsarbrokers.pulsar.streamnative.io            2023-03-02T08:52:52Z
pulsarproxies.pulsar.streamnative.io            2023-03-02T08:52:52Z
zookeeperclusters.zookeeper.streamnative.io     2023-03-02T08:52:53Z
```

--------------------------------

### Install bash-completion for Bash

Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-quick-reference

Commands to install bash-completion v2 and configure the environment variables.

```bash
brew install bash-completion@2
```

```bash
export BASH_COMPLETION_COMPAT_DIR="/usr/local/etc/bash_completion.d"
[[ -r "/usr/local/etc/profile.d/bash_completion.sh" ]] && . "/usr/local/etc/profile.d/bash_completion.sh"
```

--------------------------------

### Prepare and execute the plugin

Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-plugins

Commands to make the plugin executable, move it to the system PATH, and invoke it via the pulsarctl CLI.

```bash
chmod +x ./pulsarctl-foo
```

```bash
mv pulsarctl-foo /usr/local/bin
```

```bash
pulsarctl foo
```

```bash
pulsarctl foo args
```

--------------------------------

### Install StreamNative Operator via Helm

Source: https://docs.streamnative.io/private-cloud/v2/operate-private-cloud/upgrade/upgrade-from-pulsar-operators

Install the operator using the Helm chart.

```bash
helm install sn-operator streamnative/sn-operator -n sn-system
```

--------------------------------

### Create a sample bash plugin

Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-plugins

A basic example of a bash script plugin named 'pulsarctl-foo' that handles arguments.

```bash
#!/bin/bash

if [[ $1 == "args" ]]
then
    echo "I am the args of the pulsarctl-foo"
    exit 0
fi

echo "I am a plugin named pulsarctl-foo"
```

--------------------------------

### Setup MySQL Database via Helm

Source: https://docs.streamnative.io/connect/connectors/kafka-connect-jdbc-source/current/kafka-connect-jdbc-source

Deploys a MySQL instance in a Kubernetes cluster and initializes a test table with sample data.

```shell
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install mysql bitnami/mysql \
  --set auth.rootPassword=secretpassword \
  --set auth.database=test
kubectl wait -l app.kubernetes.io/instance=mysql --for=condition=Ready pod --timeout=5m
# Create a table and insert some data
kubectl run -i --rm --tty mysql-client --image=mysql:8.0 --restart=Never -- mysql -h mysql.default.svc.cluster.local -uroot -psecretpassword -Dtest -e "CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(255)); INSERT INTO test_table (id, name) VALUES (1, 'test-user');"
```

--------------------------------

### Install pulsarctl on macOS with Homebrew

Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-overview

Use Homebrew to tap the repository and install the pulsarctl package.

```bash
brew tap streamnative/streamnative
```

```bash
brew install pulsarctl
```

--------------------------------

### Apply Manifest File

Source: https://docs.streamnative.io/cloud/clusters/byoc/create-cloud-connection

Apply the prepared YAML manifest file to create the cloud connection.

```bash
snctl create -f cloudconnection.yaml
```

--------------------------------

### Install pulsarctl on Linux

Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-overview

Use the provided shell script to install the tool on Linux systems.

```bash
sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/streamnative/pulsarctl/master/install.sh)"
```

--------------------------------

### Install Grafana chart

Source: https://docs.streamnative.io/private-cloud/v2/operate-private-cloud/observability/private-cloud-monitor

Installs the Grafana chart using the StreamNative private cloud image.

```bash
helm install grafana grafana/grafana -n monitor --set image.repository=streamnative/private-cloud-grafana --set image.tag=0.1.1
```

--------------------------------

### Create Kafka Connect configuration file

Source: https://docs.streamnative.io/cloud/connect/self-host-connectors/cloud-connect-elasticsearch

Commands to navigate to the Kafka directory and create the configuration file.

```bash
# switch to your kafka folder
cd kafka_2.13-3.1.0
# create a connect configuration file
# which contains information of Kafka server (StreamNative KoP cluster)
vim conf/connect-sn-kop.properties
```

--------------------------------

### KSQL CLI Startup Output

Source: https://docs.streamnative.io/cloud/process/kafka-streams-and-ksql/cloud-connect-ksql

Expected console output after successfully starting the KSQL CLI.

```text
CLI v7.3.2, Server v7.3.2 located at http://localhost:8088
Server Status: RUNNING
```

--------------------------------

### Install virtualenv

Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-prerequisites

Installs the virtualenv package using pip3 to manage Python project environments.

```bash
pip3 install virtualenv
```
Tokens:
Raw