Hevo

https://docs.hevodata.com/
Admin
Hevo is a data integration platform that enables users to create and manage data pipelines,...
Tokens:487,115
Snippets:4,880
Trust Score:10
Update:11 months ago
Show doc for...
Tokens:
Raw
### Install JDK 17 using SDKMAN

Source: https://docs.hevodata.com/edge/custom-connectors/creating-your-first-connector

This snippet demonstrates how to install Java Development Kit (JDK) version 17 using the SDKMAN version manager. It first installs SDKMAN, sources its initialization script, checks the SDKMAN version, installs JDK 17, and then installs the environment.

```bash
curl -s "https://get.sdkman.io" | bash 
source "$HOME/.sdkman/bin/sdkman-init.sh" 
 sdk version 
 sdk install java 17.0.9-tem 
 sdk env install
```

--------------------------------

### Example: Start ngrok TCP Tunnel for MySQL

Source: https://docs.hevodata.com/sources/databases/mssql/generic-mssql

This is an example command demonstrating how to start an ngrok TCP tunnel specifically for a MySQL database, which typically uses port 3306.

```bash
./ngrok tcp 3306
```

--------------------------------

### Install PostgreSQL on Ubuntu

Source: https://docs.hevodata.com/destinations/databases/postgresql

These commands install the PostgreSQL server on an Ubuntu system by adding the PostgreSQL repository, importing its GPG key, updating the package list, and then performing the installation. This process requires root privileges for adding the repository.

```bash
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql
```

--------------------------------

### MongoDB Configuration File Example

Source: https://docs.hevodata.com/sources/databases/mongodb/configuring-generic-mongodb

Example configuration for the MongoDB server, specifying network binding and replica set details. Ensure 'replSetName' is set and 'oplogSizeMB' is adequately sized for replication.

```yaml
net:
  bindIp: 0.0.0.0
replication:
  replSetName: "repSet0"
  oplogSizeMB: 2048

```

--------------------------------

### Authentication Method Setup Issues

Source: https://docs.hevodata.com/sources/engg-analytics/streaming/rest-api/troubleshooting-rest-api-setup

Resolves issues related to access tokens, authorization headers, and authentication methods (No Auth, Basic Auth, OAuth 2.0).

```APIDOC
## Authentication Method Setup Issues

### Access Token Expired

**HTTP Status Code:** 401 Unauthorized

**Error Message(s):**
- The access token is expired. Please reauthorise the account.
- No Authorization Header was found.
- Invalid API key or access token (unrecognized login or wrong password).

**For No Auth or Basic Auth:**

**Potential Causes:**
  * No access token was specified.
  * An invalid access token was specified.
  * No or incorrect authorization key in the Authorization header.

**Suggested Action(s):**
  * Check access token validity.
  * Provide correct keys and key names.
  * Recreate the access token.

**For OAuth 2.0:**

**Potential Causes:**
  * Invalid authorization (_Auth URL_) or authentication (_Token URL_) URLs.
  * Invalid client ID or client secret.

**Suggested Action(s):**
  * Refer to Source API documentation for correct URLs or contact the API developer.
  * Obtain correct client credentials.

### Necessary Scopes Are Missing

**HTTP Status Code:** 403 Forbidden

**Error Message(s):** The necessary scopes and permissions are not available to complete this request.

**For No Auth or Basic Auth:**

**Potential Causes:**
  * Missing scopes in headers.
  * Missing authorization headers for bearer token.
  * Authorization token lacks correct permissions.

**Suggested Action(s):**
  * Configure supported authentication method. Use _No Auth_ for custom headers.
  * Provide _User-Agent_ header for bearer token authorization.
  * Ensure authorization token has correct permissions.

**For OAuth 2.0:**

**Potential Causes:**
  * No scopes granted to the access token.
  * Source does not grant default scopes when none are specified.

**Suggested Action(s):**
  * Specify all required scopes while generating the access token.
  * Check Source API documentation for default behavior when no scopes are specified.
```

--------------------------------

### Example MongoDB Connection String (Compass v1.11 or earlier)

Source: https://docs.hevodata.com/sources/databases/mongodb/configuring-mongodb-atlas

This is an example of a MongoDB connection string for use with MongoDB Compass version 1.11 or earlier. It includes placeholders for username and password that need to be replaced with actual credentials.

```text
mongodb://jerome:Hevo123@cluster0-shard-00-00.t7l5k.mongodb.net:27017,cluster0-shard-00-01.t7l5k.mongodb.net:27017,cluster0-shard-00-02.t7l5k.mongodb.net:27017/test?replicaSet=atlas-uax6f3-shard-0&ssl=true&authSource=admin
```

--------------------------------

### SQL SELECT Statement Example

Source: https://docs.hevodata.com/activate/working-with-activate/viewing-activation-run-details

An example of a SQL SELECT statement used in an Hevo Data Activation. This demonstrates how to retrieve data from a table for synchronization.

```sql
SELECT
  name,
  email,
  phone
FROM
  demo_table;

```

--------------------------------

### REST API Connector Setup

Source: https://docs.hevodata.com/sources/streaming/rest-api

This section details the steps for configuring the REST API as a source in HevoData, including setting up the connection, authentication, and data extraction parameters.

```APIDOC
## REST API Connector Setup

### Description
This endpoint allows you to configure the REST API as a source in HevoData to ingest data from external REST APIs. It involves setting up the connection details, authentication methods, and data extraction parameters.

### Method
POST

### Endpoint
/pipelines/create

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **Pipeline Name** (string) - Required - A unique name for the pipeline, not exceeding 255 characters.
- **Source Configuration** (object) - Required - Configuration for the REST API Source.
  - **Method** (string) - Required - The HTTP method for making API requests (e.g., GET, POST). Default: GET.
  - **URL** (string) - Required - The complete URL of your REST API endpoint.
  - **Request Body** (object/string) - Optional - The data to be sent to the API endpoint if the method is POST. Must be a valid JSON or Form Data in a key-value pair format.
  - **Authentication** (object) - Optional - Configuration for the authentication method.
    - **Type** (string) - Required - The type of authentication (e.g., No Auth, Basic Auth, OAuth 2.0).
    - **Credentials** (object) - Required - Authentication credentials based on the selected type.
  - **Headers** (object) - Optional - Key-value pairs for request headers.
  - **Query Parameters** (object) - Optional - Key-value pairs for query parameters.
  - **Data Root** (string) - Required - The path from where Hevo should replicate the data.
  - **Pagination Method** (string) - Optional - The method to read through the API response. Default: No Pagination.

### Request Example
```json
{
  "pipelineName": "MyRestAPIPipeline",
  "sourceConfiguration": {
    "method": "GET",
    "url": "https://api.example.com/data",
    "authentication": {
      "type": "No Auth"
    },
    "headers": {
      "Accept": "application/json"
    },
    "queryParameters": {
      "limit": "100"
    },
    "dataRoot": "$.items",
    "paginationMethod": "No Pagination"
  }
}
```

### Response
#### Success Response (200)
- **pipelineId** (string) - The unique identifier for the created pipeline.
- **status** (string) - The status of the pipeline creation.

#### Response Example
```json
{
  "pipelineId": "pipeline-12345",
  "status": "created"
}
```
```

--------------------------------

### Configuring ActiveCampaign as a Source

Source: https://docs.hevodata.com/sources/marketing/activecampaign

A step-by-step guide on how to set up ActiveCampaign as a data source within a Hevo Pipeline.

```APIDOC
## Configuring ActiveCampaign as a Source

Follow these steps to configure ActiveCampaign as a source in your Hevo Pipeline:

1.  Click **PIPELINES** in the Hevo navigation bar.
2.  Click **+ CREATE PIPELINE** on the Pipelines List View.
3.  Select **ActiveCampaign** as the Source Type.
4.  Choose your desired Destination Type.
5.  On the **Configure your ActiveCampaign Source** page, provide the following details:
    *   **Pipeline Name**: A unique name for your pipeline (max 255 characters).
    *   **API Key**: Your ActiveCampaign private API key.
    *   **Account Name**: Your unique ActiveCampaign account identifier.
6.  Click **TEST & CONTINUE**.
7.  Proceed with configuring data ingestion and destination settings.
```

--------------------------------

### Configure Shopify Source

Source: https://docs.hevodata.com/sources/e-commerce/shopify/shopify-app

Step-by-step guide to configuring the Shopify source in Hevo, including creating a custom app in Shopify and assigning necessary permissions.

```APIDOC
## Configure Shopify Source

### Description
This section provides a detailed procedure for setting up your Shopify source within Hevo. It involves creating a custom application in your Shopify store to grant Hevo the necessary API access for data ingestion.

### Method
N/A (Procedural Guide)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (Informational)
- **Prerequisites**: An active Shopify account, matching data types in Source and Destination, and appropriate Hevo user roles (Team Administrator, Team Collaborator, or Pipeline Administrator).

#### Response Example
**Step 1: Create an App in Shopify**
1. Log in to your Shopify Store Admin account.
2. Navigate to **Apps** in the left navigation pane.
3. Click **App and sales channel settings**.
4. On the **Apps and sales channel** page, click **Develop apps**.
5. Click **Allow custom app development** (you may need to do this twice).
6. On the **App Development** page, click **Create an app**.
7. In the **Create an app** dialog:
   - **App name**: Provide a name for your custom app.
   - **App developer**: Select your email address.
   - Click **Create app**.
8. Click **Configure Admin API scopes**.
9. Refer to the "Configure API Permissions in Shopify" section for detailed steps on assigning permissions to fetch all data from Shopify.
```

--------------------------------

### Initialize dbt Project

Source: https://docs.hevodata.com/transform/dbt-projects/setting-up-dbt-in-git

This command initializes a new dbt project, creating a default project structure and sample models. It requires dbt Core to be installed. The output is the creation of a new directory for the dbt project.

```bash
$ dbt init first_dbt_project

```

--------------------------------

### Verify PostgreSQL Installation on Linux

Source: https://docs.hevodata.com/destinations/databases/postgresql

This command checks the status of the PostgreSQL service to determine if it is installed and running on a Linux system. It helps in deciding whether to proceed with installation or setup.

```bash
systemctl status postgresql
```

--------------------------------

### Example ngrok TCP Tunnel for MySQL

Source: https://docs.hevodata.com/sources/databases/microsoft-sql-server

Example command to start a TCP tunnel for a MySQL database, which typically uses port 3306.

```shell
./ngrok tcp 3306

```

--------------------------------

### Link dbt Project to GitHub Repository

Source: https://docs.hevodata.com/transform/dbt-projects/setting-up-dbt-in-git

These Git commands initialize a Git repository in your dbt project directory, stage all project files, create an initial commit, and then link it to a remote GitHub repository before pushing the changes. This ensures your dbt project is version-controlled on GitHub. Requires Git to be installed and a GitHub repository to be created.

```bash
git init
git branch -M main
git add .
git commit -m "Create a dbt project"
git remote add origin https://github.com/<username>/dbt-sample-repository.git
git push -u origin main

```

--------------------------------

### Clone Connector Repository using Git

Source: https://docs.hevodata.com/edge/custom-connectors/creating-your-first-connector

This command clones the 'test_connector' repository from GitHub to your local machine. Ensure you have Git installed and necessary permissions to clone the repository. This is the first step in accessing your custom connector code.

```bash
git clone https://github.com/hevoio/test_connector.git
```

--------------------------------

### Run dbt Models

Source: https://docs.hevodata.com/transform/dbt-projects/setting-up-dbt-in-git

This command executes all the data transformation models defined within your dbt project. It requires a successful connection to the data platform. The output will show the progress and results of each model run.

```bash
dbt run

```

--------------------------------

### Start ngrok TCP Tunnel

Source: https://docs.hevodata.com/sources/databases/microsoft-sql-server

Command to start a TCP tunnel with ngrok, forwarding to a specified database port.

```shell
./ngrok tcp <your_database_port>

```

--------------------------------

### Configuration Issues

Source: https://docs.hevodata.com/sources/engg-analytics/streaming/rest-api/troubleshooting-rest-api-setup

Resolves problems related to generating access tokens and accessing resources.

```APIDOC
## Configuration Issues

### Error While Generating a New Access Token

**HTTP Status Code:** 400 Bad Request

**Error Message(s):** Error while generating new access token from user’s login service.

**Potential Causes:**
  * Invalid or changed client credentials (client ID/secret) or login credentials (username/password).
  * Incorrect or unsupported grant type for OAuth 2.0 access token generation.
  * App registered for Hevo lacks permission to request OAuth 2.0 access token.
  * Redirect URL mismatch.
  * Expired refresh token for OAuth 2.0 access token.

**Suggested Action(s):**
  * Provide correct client or login credentials.
  * Ascertain validity of credentials.
  * Specify grant type as _Authorization Code_ for OAuth 2.0.
  * Correct the redirect URL.
  * Provide any additional required scopes to obtain a refresh token and create the OAuth 2.0 access token again.

### The Requested Resource Is Not Available

**HTTP Status Code:** 404 Not Found

**Error Message(s):**
- The requested resource is not available.
- Failed to get a response.

**Potential Causes:**
  * Incorrect REST API endpoint URL.
  * Private API configured, but Hevo IPs lack access.
```

--------------------------------

### Start ngrok TCP Tunnel

Source: https://docs.hevodata.com/destinations/databases/connecting-to-a-local-db-dest

Command to start a TCP tunnel using ngrok, forwarding traffic to your local database port. This allows Hevo to connect to your database remotely.

```shell
./ngrok tcp <your_database_port>
```

```shell
./ngrok tcp 3306
```

--------------------------------

### Google Maps API URL Example

Source: https://docs.hevodata.com/glossary

An example URL demonstrating how to use the Google Maps API to display a map without markers or directions. This illustrates a basic API endpoint for a specific action.

```url
https://www.google.com/maps/@api=1&map_action=map&parameters
```

--------------------------------

### Edit MySQL Configuration File

Source: https://docs.hevodata.com/sources/databases/mysql/generic-mysql

This command opens the MySQL server's main configuration file using the nano text editor. You will modify this file to enable and configure binary logging.

```bash
sudo nano /etc/mysql/my.cnf
```

--------------------------------

### REST API Request Body Examples (JSON/Form Data) - Hevo

Source: https://docs.hevodata.com/sources/sdk-%26-streaming/rest-api

This snippet provides examples of how to format the request body for POST requests when configuring a REST API source in Hevo. It highlights that the request body must be valid JSON or Form Data in a key-value pair format. This is essential for sending data to the API endpoint, such as creating a PagerDuty incident as shown in the example.

```Hevo Configuration
3. If you selected the _POST_ method, specify the **Request Body** , which contains the data to be sent to your API endpoint. Else, skip to setting up the authentication method.
The request body must be a valid _JSON_ or _Form Data_ in a key-value pair format. For example, the request body in the image below is in JSON format and contains the data for creating a PagerDuty incident.
```

--------------------------------

### Fetch and Publish SaaS Data using Hevo SDK

Source: https://docs.hevodata.com/edge/custom-connectors/creating-your-first-connector/creating-the-connection-logic

Demonstrates fetching data from a SaaS source using an API client, converting the response to an HStruct object, and publishing it. It includes steps for refining API responses and converting them to a structured format compatible with Hevo.

```java
Response apiResponse = apiClient.request(request);
Map<String, Object> refinedData = DataConverterUtils.convertResponse(apiResponse, objectMapper, new TypeReference<>() {});
HStruct row = DataConverterUtils.convertToHStruct(refinedData, connectorContext.schema());
connectorProcessor.publish(row, ConnectorMeta.builder().opType(OpType.READ).build(), connectorContext.schema().objectDetail());
```

--------------------------------

### CSV Data Example without Column Headers

Source: https://docs.hevodata.com/sources/dbfs/file-storage/google-cloud-storage-%28gcs%29

This example demonstrates CSV data where the first row does not contain column headers. When the 'Treat first row as column headers' option is disabled, Hevo auto-generates column headers.

```CSV
CLAY COUNTY,32003,11973623
CLAY COUNTY,32003,46448094
CLAY COUNTY,32003,55206893
CLAY COUNTY,32003,15333743
SUWANNEE COUNTY,32003,85751490
SUWANNEE COUNTY,32062,50972562
ST JOHNS COUNTY,846636,32033,
NASSAU COUNTY,32025,88310177
NASSAU COUNTY,32041,34865452
```

--------------------------------

### Example: Delete Deleted Customer Events

Source: https://docs.hevodata.com/destinations/destination-faqs/filter-deleted-events

An example SQL query to permanently delete events marked as deleted from the 'customer' table. It targets records where the '__hevo__marked_deleted' column is TRUE. This action is irreversible.

```sql
DELETE from customer
WHERE __hevo__marked_deleted = TRUE;
```

--------------------------------

### Configuring Pendo as a Source in Hevo

Source: https://docs.hevodata.com/sources/prod-analytics/pendo

Instructions for setting up Pendo as a data source within your Hevo pipeline, including required parameters.

```APIDOC
## Configuring Pendo as a Source

Perform the following steps to configure Pendo as the Source in your Pipeline:
  1. Click **PIPELINES** in the Navigation Bar.
  2. Click **+ CREATE PIPELINE** in the Pipelines List View.
  3. On the **Select Source Type** page, select _Pendo_.
  4. On the **Select Destination Type** page, select the type of Destination you want to use.
  5. On the **Configure your Pendo Source** page, specify the following:
     * **Pipeline Name** : A unique name for your Pipeline, not exceeding 255 characters.
     * **Region** : The subscription region of your Pendo account. Default value: _Region - US_.
     * **Integration Key** : A secret value with read-write access to your Pendo data via v1 APIs.
**Note** : This key is specific to the subscription region of your account.
     * **Historical Sync Duration** : The duration for which you want to ingest the existing data from the Source. Default duration: _3 Months_.
**Note** : If you select _All Available Data_ , Hevo ingests all the data available in your Pendo account since January 01, 2013.
  6. Click **TEST & CONTINUE**.
  7. Proceed to configuring the data ingestion and setting up the Destination.
```

--------------------------------

### Start TCP Tunnel with ngrok

Source: https://docs.hevodata.com/sources/databases/connecting-to-a-local-db

This command starts a TCP tunnel using ngrok to forward traffic to your local database port. Replace '<your_database_port>' with the actual port number of your database (e.g., 3306 for MySQL).

```bash
./ngrok tcp <your_database_port>
```

--------------------------------

### JSON Event Data Example

Source: https://docs.hevodata.com/destinations/data-warehouses/snowflake/snowflake-data-structure

This snippet shows an example of a JSON event ingested from a webhook source, with a nested metadata field of type string. This demonstrates the structure of semi-structured data that Snowflake can handle.

```json
{
    "event_name": "agents",
    "properties": {
        "agent_name": "James",
        "agent_id": "Bond 007",
        "metadata": {
          "message": "This is a test message", 
          "rand_id": 3523
        }
    }
}

```

--------------------------------

### Edit MySQL Configuration File

Source: https://docs.hevodata.com/edge/sources/mysql/generic-mysql

This command opens the MySQL configuration file using the nano text editor with sudo privileges.  The configuration file is modified to enable and configure binary logging for replication purposes with HevoData.

```Shell
sudo nano /etc/mysql/my.cnf
```

```Shell
sudo nano /etc/my.cnf
```

--------------------------------

### Example: Filter Deleted Customer Events

Source: https://docs.hevodata.com/destinations/destination-faqs/filter-deleted-events

An example SQL query demonstrating how to filter deleted customer events from a 'customer' table using the '__hevo__marked_deleted' metadata column. This query selects only the records where the deletion flag is FALSE.

```sql
SELECT *
FROM customer
WHERE __hevo__marked_deleted = FALSE;
```

--------------------------------

### Azure Synapse Analytics Connection String Example

Source: https://docs.hevodata.com/destinations/data-warehouses/azure-synapse

An example of a JDBC connection string for Azure Synapse Analytics. It demonstrates the format and parameters required, including server details, user credentials, and security settings. Note that the 'user' parameter may need to be updated if a specific login user was added.

```SQL
jdbc:sqlserver://asea-workspace-doc....;user=sqladminuser@...;password={your_password_here};encrypt=true;trustServerCertificate=false; hostNameInCertificate=*.sql.azuresynapse.net;loginTimeout=30;
```

--------------------------------

### Hevo Connector Logic Skeleton - Java

Source: https://docs.hevodata.com/edge/custom-connectors/creating-your-first-connector/creating-the-connection-logic

Skeleton Java code demonstrating the implementation of core connection logic methods for a custom Hevo connector. This includes initializing connection, fetching objects and schemas, retrieving data, and closing the connection.

```java
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;

// Assuming these are placeholder classes for Hevo SDK
class ObjectDetails {}
class ObjectSchema {}
class ConnectorContext {
    Schema schema() { return new Schema(); } // Placeholder for schema retrieval
}
class Schema { ObjectDetails objectDetail() { return new ObjectDetails(); } } // Placeholder
class ConnectorProcessor {}
class ExecutionResultStats { public ExecutionResultStats(long l) {} } // Placeholder
class ExecutionResult { public ExecutionResult(Map<ObjectDetails, ExecutionResultStats> map, Offset empty) {} } // Placeholder
class Offset { public static Offset empty() { return new Offset(); } } // Placeholder

public class CustomConnector {

    @Override
    public void initializeConnection() {
        // Implement sample test connection
        System.out.println("Initializing connection...");
    }

    @Override
    public List<ObjectDetails> getObjects() {
        return new LinkedList<>();
    }

    @Override
    public List<ObjectSchema> fetchSchemaFromSource(List<ObjectDetails> objectDetails) {
        return Collections.emptyList();
    }

    @Override
    public ExecutionResult fetchDataFromSource(
        ConnectorContext context, ConnectorProcessor processor) {
        return new ExecutionResult(
            Map.of(context.schema().objectDetail(), new ExecutionResultStats(0L)), Offset.empty());
    }

    @Override
    public void close() {
        // Do nothing
        System.out.println("Closing connection.");
    }

}
```

--------------------------------

### Create MySQL Database User

Source: https://docs.hevodata.com/sources/databases/mysql/generic-mysql

These commands demonstrate how to create a new database user in MySQL. Two variations are provided for different MySQL versions (5.6-8.0 and 8.0-8.4), specifying the username, password, and authentication method.

```sql
CREATE USER <database_username>@'%' IDENTIFIED BY '<password>';
```

```sql
CREATE USER <database_username>@'%' IDENTIFIED WITH mysql_native_password BY '<password>';
```

--------------------------------

### Example SQL Queries for Quoted and Unquoted Table Names in Snowflake

Source: https://docs.hevodata.com/edge/destinations/snowflake

Demonstrates how table and column names are represented in SQL queries based on the 'Always quote table names or entity names' setting in Hevo Edge. The first example shows quoted identifiers, preserving case and special characters, while the second shows sanitized, unquoted identifiers with special characters replaced by underscores.

```sql
SELECT "Column 1", "name" from RON.DARK_ARTS."test1_Table namE 05";

```

```sql
SELECT COLUMN_1, NAME from RON.DARK_ARTS.TEST1_TABLE_NAME_05;

```

--------------------------------

### Initialize TestConnector Class

Source: https://docs.hevodata.com/edge/custom-connectors/creating-your-first-connector

This Java code snippet shows the basic structure for initializing a custom connector class named 'TestConnector'. It implements the 'GenericConnector' interface and sets up a logger using SLF4j. This serves as the entry point for your connector's logic.

```java
public class TestConnector implements GenericConnector { 
    private static final Logger log = LoggerFactory.getLogger(TestConnector.class);

```
Tokens:
Raw