Zingg

https://github.com/zinggai/zingg
Admin
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
Tokens:13,343
Snippets:111
Trust Score:8.2
License:AGPL-3.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Installing Apache Maven on Ubuntu

Source: https://github.com/zinggai/zingg/blob/main/docs/settingUpZingg.md

Commands to download and install Apache Maven 3.8.8 on Ubuntu. This includes downloading the binary, extracting it, and verifying the installation.

```bash
wget https://dlcdn.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
tar -xvf apache-maven-3.8.8-bin.tar.gz 
rm -rf apache-maven-3.8.8-bin.tar.gz 
cd apache-maven-3.8.8/
cd bin
./mvn --version
```

--------------------------------

### Configuring Environment Variables for Spark and Java

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/single-machine-setup.md

Essential environment variable exports needed in ~/.bash_aliases to run Zingg with local Spark. Sets up JAVA_HOME, SPARK_HOME, and configures Spark to run in local mode.

```bash
export JAVA_HOME=path to jdk
export SPARK_HOME=path to location of Apache Spark
export SPARK_MASTER=local[*]
```

--------------------------------

### Running Zingg Sample Program

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/verification.md

Commands to navigate to the Zingg directory and run a sample program that builds Zingg models and finds duplicates in a test dataset, which validates the installation.

```bash
cd zingg
```

```bash
./scripts/zingg.sh --phase trainMatch --conf examples/febrl/config.json
```

--------------------------------

### Updating Ubuntu and Installing WSL2

Source: https://github.com/zinggai/zingg/blob/main/docs/settingUpZingg.md

Commands to update Ubuntu and install Ubuntu on WSL2 for Windows users. This includes installing WSL, downloading Ubuntu from Microsoft Store, and initial configuration.

```bash
sudo apt update
```

```bash
wsl --install
```

```bash
sudo apt update
```

--------------------------------

### Installing Apache Spark on Ubuntu

Source: https://github.com/zinggai/zingg/blob/main/docs/settingUpZingg.md

Commands to download and install Apache Spark 3.5.0 on Ubuntu. This includes downloading the tarball, extracting it, and moving it to the /opt directory.

```bash
wget https://www.apache.org/dyn/closer.lua/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvf spark-3.5.0-bin-hadoop3.tgz
rm -rf spark-3.5.0-bin-hadoop3.tgz
sudo mv spark-3.5.0-bin-hadoop3 /opt/spark
```

--------------------------------

### Installing and Configuring Zingg Enterprise Snowflake

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-zingg-enterprise-snowflake/setting-up-zingg-enterprise-for-snowflake.md

Series of commands to extract the Zingg Enterprise Snowflake package, set up environment variables, and move the license file to the correct location. Includes archive extraction, directory navigation, and environment variable configuration.

```bash
gzip -d zingg-enterprise-snowflake-0.4.1-SNAPSHOT.tar.gz
```

```bash
tar xvf zingg-enterprise-snowflake-0.4.1-SNAPSHOT.tar
```

```bash
cd zingg-enterprise-snowflake-0.4.1-SNAPSHOT
```

```bash
export ZINGG_SNOW_JAR=~/zingg-enterprise-snowflake-0.4.1-SNAPSHOT
```

```bash
export ZINGG_SNOW_HOME=~/zingg-enterprise-snowflake-0.4.1-SNAPSHOT
```

```bash
mv ~/zingg.license .
```

--------------------------------

### Installing JDK 11 on Ubuntu

Source: https://github.com/zinggai/zingg/blob/main/docs/settingUpZingg.md

Commands to install Java Development Kit 11 (JDK 11) on Ubuntu. This includes both the JDK and JRE, and verifies the installation.

```bash
sudo apt install openjdk-11-jdk openjdk-11-jre
javac -version
java -version
```

--------------------------------

### Setting Environment Variables for Spark Cluster

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/spark-cluster-checklist.md

Defines required environment variables for connecting Zingg with a Spark cluster. These include paths to Java and Spark installations, along with the Spark master connection URL.

```bash
export JAVA_HOME=path to jdk
```

```bash
export SPARK_HOME=path to Apache Spark
```

```bash
export SPARK_MASTER=spark://master-host:master-port
```

--------------------------------

### Run Zingg Sample Verification Job (Shell)

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-zingg-enterprise-snowflake/verifying-the-installation.md

This command executes a sample Zingg job using the provided properties and configuration files. It runs the 'findTrainingData' phase to build models and find duplicates in the test data. Successful execution indicates a correct Zingg installation.

```Shell
./scripts/zingg.sh --properties-file snowEnv.txt --phase findTrainingData --conf  examples/febrl/configSnow.json
```

--------------------------------

### Setting the ZINGG_HOME environment variable

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/installing-zingg.md

Command to set the ZINGG_HOME environment variable to the path where Zingg is installed.

```bash
export ZINGG_HOME=path to zingg
```

--------------------------------

### Extracting Zingg package using gzip and tar

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/installing-zingg.md

Command to decompress and extract the Zingg package from the downloaded tar.gz file.

```bash
gzip -d zingg-0.5.0.tar.gz ; tar xvf zingg-0.5.0.tar
```

--------------------------------

### Moving the Zingg directory

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/installing-zingg.md

Command to move the extracted Zingg directory to a designated location in the home folder.

```bash
mv zingg-0.5.0 ~/zingg
```

--------------------------------

### Setting AWS Credentials via Environment Variables

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/amazonS3.md

Commands to export AWS access credentials as environment variables for S3 access. Includes optional MFA token setup.

```bash
export AWS_ACCESS_KEY_ID=<access key id>\nexport AWS_SECRET_ACCESS_KEY=<access key>
```

--------------------------------

### Checking Environment Variables in Bash

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/verification.md

Commands to verify that essential environment variables for Zingg are set correctly, including SPARK_HOME, JAVA_HOME, ZINGG_HOME, and Java version.

```bash
bash
```

```bash
echo $SPARK_HOME
```

```bash
echo $JAVA_HOME
```

```bash
java --version
```

```bash
echo $ZINGG_HOME
```

--------------------------------

### Updating PATH environment variable

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-from-release/installing-zingg.md

Command to update the PATH environment variable to include Java, Spark, and Zingg executable paths for easier access.

```bash
export PATH=$PATH:$JAVA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:ZINGG_HOME/scripts
```

--------------------------------

### Installing Zingg Python Package

Source: https://github.com/zinggai/zingg/blob/main/python/README.md

This code snippet shows how to install the Zingg package using pip. The package provides APIs for entity resolution and related tasks.

```bash
pip install zingg
```

--------------------------------

### Installing the Zingg Python Package

Source: https://github.com/zinggai/zingg/blob/main/docs/working-with-python.md

Command to install the Zingg Python package using pip. This is required before using the Python API to build and run Zingg entity resolution programs.

```bash
python -m pip install zingg
```

--------------------------------

### Configuring Lookup Settings in JSON

Source: https://github.com/zinggai/zingg/blob/main/docs/runApproval.md

Example configuration file (lookupConf.json) for the Zingg lookup feature. It defines the configuration, lookup data source, and output destination with specific formatting properties.

```json
{
  "config" : "config.json",
  "lookupData": [{
    "name":"lookup-test-data",
    "format":"inMemory"
  }
  ],
  "lookupOutput": [
    {
      "name":"lookup-output",
      "format":"csv",
      "props": {
        "location": "/tmp/zinggOutput/lookup",
        "delimiter": ",",
        "header":true
      }
    }
  ]
}
```

--------------------------------

### Specifying Training Samples in Zingg Configuration (JSON)

Source: https://github.com/zinggai/zingg/blob/main/docs/setup/training/addOwnTrainingData.md

This JSON snippet shows how to include the trainingSamples attribute in the Zingg configuration file to specify pre-existing training data. It refers to an example configuration file that demonstrates the correct format for including training samples.

```json
"trainingSamples": [...]
```

--------------------------------

### Implementing Deterministic Matching in Python

Source: https://github.com/zinggai/zingg/blob/main/docs/deterministicMatching.md

Python code example showing how to create and configure deterministic matching conditions using the Zingg API. Sets up three matching rules corresponding to the JSON configuration.

```python
detMatchNameAdd = DeterministicMatching('fname','stNo','add1')  
detMatchNameDobSsn = DeterministicMatching('fname','dob','ssn')  
detMatchNameEmail = DeterministicMatching('fname','email')  
args.setDeterministicMatchingCondition(detMatchNameAdd,detMatchNameDobSsn,detMatchNameEmail)  
```

--------------------------------

### Viewing Zingg Analytics Event Example

Source: https://github.com/zinggai/zingg/blob/main/docs/security.md

This snippet shows an example of an analytics event captured while running Zingg. It includes various metadata about the execution environment and performance metrics, but no actual user data.

```json
event is {"client_id":"localhost","events":[{"name":"match","params":{"executionTime":"1.743246748E9","country":"IN","zingg_version":"0.5.0","modelId":"100","domain":"localhost","java_version":"11.0.18","os_arch":"aarch64","os_name":"Mac OS X","DB_INSTANCE_TYPE":null,"DATABRICKS_RUNTIME_VERSION":null,"dataCount":"65.0","ZINGG_HOME":"true"}}],"user_id":"zingg"}
```

--------------------------------

### Executing Zingg Pipeline with S3 Integration

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/amazonS3.md

Complete sequence of Zingg commands for running the data processing pipeline using S3 storage, including findTrainingData, label, train, and match phases.

```bash
./scripts/zingg.sh --phase findTrainingData --properties-file config/zingg.conf  --conf examples/febrl/config.json --zinggDir  s3a://zingg28032023/zingg\n./scripts/zingg.sh --phase label --properties-file config/zingg.conf  --conf examples/febrl/config.json --zinggDir  s3a://zingg28032023/zingg\n./scripts/zingg.sh --phase train --properties-file config/zingg.conf  --conf examples/febrl/config.json --zinggDir  s3a://zingg28032023/zingg\n./scripts/zingg.sh --phase match --properties-file config/zingg.conf  --conf examples/febrl/config.json --zinggDir  s3a://zingg28032023/zingg
```

--------------------------------

### Monitor nohup Output (tail)

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-zingg-enterprise-snowflake/running-asynchronously.md

Displays the end of the nohup.out file and follows new content as it is written. This command is used to monitor the progress and output of the background process started with nohup.

```Shell
tail -f nohup.out
```

--------------------------------

### Configuring Spark Dependencies for AWS S3

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/amazonS3.md

Configuration entry for spark.jars property in zingg.conf to include required AWS S3 dependencies.

```bash
spark.jars=/<location>/hadoop-aws-3.1.0.jar,/<location>/aws-java-sdk-bundle-1.11.271.jar
```

--------------------------------

### Implementing Entity Resolution with Zingg Python API

Source: https://github.com/zinggai/zingg/blob/main/python/docs/index.rst

Example demonstrating how to use the Zingg Python API for entity resolution. The code shows how to configure field definitions with different match types, set up model parameters, define input/output data pipes, and execute the matching process using Zingg client.

```python
from zingg.client import *
from zingg.pipes import *

#build the arguments for zingg
args = Arguments()
#set field definitions
fname = FieldDefinition("fname", "string", MatchType.FUZZY)
lname = FieldDefinition("lname", "string", MatchType.FUZZY)
stNo = FieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = FieldDefinition("add1","string", MatchType.FUZZY)
add2 = FieldDefinition("add2", "string", MatchType.FUZZY)
city = FieldDefinition("city", "string", MatchType.FUZZY)
areacode = FieldDefinition("areacode", "string", MatchType.FUZZY)
state = FieldDefinition("state", "string", MatchType.FUZZY)
dob = FieldDefinition("dob", "string", MatchType.FUZZY)
ssn = FieldDefinition("ssn", "string", MatchType.FUZZY)

fieldDefs = [fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]

args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)

#reading dataset into inputPipe and settint it up in 'args'
#below line should not be required if you are reading from in memory dataset
#in that case, replace df with input df
schema = "id string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn  string"
inputPipe = CsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
outputPipe = CsvPipe("resultFebrl", "/tmp/febrlOutput")

args.setOutput(outputPipe)

options = ClientOptions([ClientOptions.PHASE,"match"])

#Zingg execution for the given phase
zingg = Zingg(args, options)
zingg.initAndExecute()
```

--------------------------------

### Sample Google Service Account Key Format

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/bigquery.md

Example of a Google service account key JSON file structure used for authentication. Contains fields like project_id, private_key, client_id, and various URIs for OAuth2 authentication.

```json
{
 "type": "service_account",
 "project_id": "mynotification-46566",
 "private_key_id": "905cbfd273ff9205d1cabfe06fa6908e54534",
 "private_key": "-----BEGIN PRIVATE KEY-----CERT.....",
 "client_id": "11143646541283115487",
 "auth_uri": "https://accounts.google.com/o/oauth2/auth",
 "token_uri": "https://oauth2.googleapis.com/token",
 "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
 "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/zingtest%44mynotification-46566.iam.gserviceaccount.com"
}
```

--------------------------------

### Copying Files Between Host and Docker Container

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/docker/copying-files-to-and-from-the-container.md

Commands demonstrating how to copy files to and from a Docker container using docker cp. These examples show copying a file named foo.txt in both directions between the host and container.

```bash
$ docker cp foo.txt <container_id>:/foo.txt
$ docker cp <container_id>:/foo.txt foo.txt
```

--------------------------------

### Adding Stopwords to Field Definition in Zingg Configuration

Source: https://github.com/zinggai/zingg/blob/main/docs/accuracy/stopWordsRemoval.md

JSON configuration showing how to integrate a stopwords file into a field definition in Zingg. The example demonstrates configuring a fuzzy match field with a reference to a CSV file containing stopwords.

```json
"fieldDefinition":[
   	{
   		"fieldName" : "fname",
   		"matchType" : "fuzzy",
   		"fields" : "fname",
   		"dataType": "string",
   		"stopWords": "models/100/stopWords/fname.csv"
   	},
```

--------------------------------

### Field Definition JSON Structure

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/configuration/field-definitions.md

JSON structure showing required attributes for defining fields in Zingg data matching. Each field requires fieldName, fields, dataType, and matchType properties.

```json
{
  "fieldName": "<column_name>",
  "fields": "<same_as_fieldName>",
  "dataType": "string|integer|double",
  "matchType": "FUZZY,NUMERIC"
}
```

--------------------------------

### Running Zingg Docker Container with Custom User ID

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/docker/file-read-write-permissions.md

This command demonstrates how to run the Zingg Docker container with a specified user ID. It allows the container to have the necessary permissions for file operations in shared locations. The example shows checking the current user's ID and then running the container with that ID.

```bash
$ id 
uid=1000(abc) gid=1000(abc) groups=1000(abc)
$ docker run -u <uid> -it zingg/zingg:0.4.1-SNAPSHOT bash
```

--------------------------------

### Configuring Zingg JSON with Environment Variables

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/configuration/configuring-through-environment-variables.md

Example configuration showing how to reference environment variables in a Zingg JSON config file. Environment variables must be enclosed in $variable$ syntax, with strings requiring quotes and numeric/boolean values without quotes. The config filename must end with .env extension.

```json
"output" : [{
  "name":"unifiedCustomers", 
  "format":"net.snowflake.spark.snowflake",
  "props": {
    "location": "$location$",
    "delimiter": ",",
    "header": false,				
    "password": "$passwd",					
  }
}],

"labelDataSampleSize" : 0.5,
"numPartitions":4,
"modelId": $modelId$,
"zinggDir": "models",
"collectMetrics": $collectMetrics$
```

--------------------------------

### Configuring Pass Through Data Expression in JSON

Source: https://github.com/zinggai/zingg/blob/main/docs/passthru.md

This JSON snippet shows how to define a pass-through expression in Zingg configuration to identify records that should be excluded from matching. In this example, records where 'is_deceased' equals true will be treated as pass-through records.

```json
     "passthroughExpr": "is_deceased = true"
```

--------------------------------

### Running Zingg to Find Training Data

Source: https://github.com/zinggai/zingg/blob/main/docs/settingUpZingg.md

Command to run Zingg for finding training data using a specific configuration file.

```bash
./scripts/zingg.sh --phase findTrainingData --conf examples/febrl/config.json
```

--------------------------------

### Compiling the Zingg Repository

Source: https://github.com/zinggai/zingg/blob/main/docs/settingUpZingg.md

Commands to compile the Zingg repository using Maven. This includes initializing, cleaning, compiling, and packaging the project with specific Spark version.

```bash
git branch
```

```bash
mvn initialize
mvn clean compile package -Dspark=sparkVer
```

```bash
mvn initialize
mvn clean compile package -Dspark=sparkVer -Dmaven.test.skip=true
```

--------------------------------

### Executing Maven Commands for Zingg Compilation

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/compiling-from-source.md

This snippet shows the Maven commands needed to initialize the project and then compile and package it. The first command initializes the project dependencies, and the second command cleans any previous builds, compiles the source code, and packages it into deployable artifacts.

```bash
mvn initialize
```

```bash
mvn clean compile package
```

--------------------------------

### Running Zingg Explain Phase in Bash

Source: https://github.com/zinggai/zingg/blob/main/docs/modelexplain.md

This command executes the explain phase in Zingg. It requires specifying the phase for explanation, the path to the configuration file, the mode as 'explain', and a suffix for the explanation output.

```bash
./scripts/zingg.sh --phase <phase for explanation> --conf <path to config> --mode explain --suffix <suffix for explain>
```

--------------------------------

### Initializing ClientOptions Class - Python

Source: https://github.com/zinggai/zingg/blob/main/docs/python/markdown/zingg.md

Class for configuring Zingg client options including phase selection and command line parameters. Handles core configuration for the matching process.

```python
class ClientOptions(argsSent=None):
    Bases: `object`
```

--------------------------------

### Running Zingg Docker Container for Entity Matching

Source: https://github.com/zinggai/zingg/blob/main/README.md

This snippet demonstrates how to pull the Zingg Docker image, run a container, and execute the match phase using a prebuilt model with a sample configuration file.

```bash
docker pull zingg/zingg:0.4.0
docker run -it zingg/zingg:0.4.0 bash
./scripts/zingg.sh --phase match --conf examples/febrl/config.json
```

--------------------------------

### Setting Environment Variables for Zingg Development

Source: https://github.com/zinggai/zingg/blob/main/docs/settingUpZingg.md

Commands to set up environment variables in the .bashrc file for Zingg development. This includes paths for Spark, Maven, Zingg, and Java.

```bash
vim ~/.bashrc
export SPARK_HOME=/opt/spark
export SPARK_MASTER=local[*]
export MAVEN_HOME=/home/ubuntu/apache-maven-3.8.8
export ZINGG_HOME=<path_to_zingg>/assembly/target
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin
```

```bash
source ~/.bashrc
```

```bash
echo $PATH
mvn --version
```

--------------------------------

### Executing Zingg Match Phase

Source: https://github.com/zinggai/zingg/blob/main/docs/setup/match.md

Command to run Zingg's match phase using a configuration file. The match phase assigns cluster IDs and match scores to identify matching records in the dataset.

```bash
./zingg.sh --phase match --conf config.json
```

--------------------------------

### Training Zingg Model via Command Line

Source: https://github.com/zinggai/zingg/blob/main/docs/setup/train.md

Command to execute Zingg's model training phase using a configuration file. The trained model will be saved to zinggDir/modelId directory as specified in the config.json file.

```bash
./zingg.sh --phase train --conf config.json
```

--------------------------------

### Initializing Zingg Client in Python

Source: https://github.com/zinggai/zingg/blob/main/docs/python/markdown/zingg.md

The Zingg class is the main interface for the Zingg matching product. It initializes a client with provided arguments and options, and includes methods for executing various phases of the matching process.

```Python
class Zingg(object):
    def __init__(self, args, options):
        # Initialize Zingg client
        pass

    def execute(self):
        # Execute matching process
        pass

    def executeLabel(self):
        # Run label phase
        pass

    def executeLabelUpdate(self):
        # Run label update phase
        pass
```

--------------------------------

### Running Zingg Label Phase via Command Line

Source: https://github.com/zinggai/zingg/blob/main/docs/setup/training/label.md

This command executes the label phase in Zingg, allowing users to interactively mark pairs as matches or non-matches. The optional showConcise flag can be used to display only non-DONT_USE fields.

```bash
./zingg.sh --phase label --conf config.json <optional --showConcise=true|false>
```

--------------------------------

### Configuring Exasol as a Data Source in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/exasol.md

JSON configuration for using Exasol as a data input source in Zingg. This snippet defines connection parameters including host, port, credentials, and the SQL query to retrieve data from an Exasol table.

```json
...\n \"data\": [\n    {\n        \"name\": \"input\",\n        \"format\": \"com.exasol.spark\",\n        \"props\": {\n            \"host\": \"10.11.0.2\",\n            \"port\": \"8563\",\n            \"username\": \"sys\",\n            \"password\": \"exasol\",\n            \"query\": \"SELECT * FROM DB_SCHEMA.CUSTOMERS\"\n        }\n    }\n ],\n ...
```

--------------------------------

### Executing Zingg with JSON Configuration

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/zingg-command-line.md

Command to run Zingg with a JSON configuration file, specifying an optional properties file and the desired execution phase.

```shell
./scripts/zingg.sh <optional --properties-file path to zingg.conf> --phase <phase> --conf <path to json>
```

--------------------------------

### Initializing Arguments Class - Python

Source: https://github.com/zinggai/zingg/blob/main/docs/python/markdown/zingg.md

Main class for configuring match arguments with support for defining data location, training, and matching phases. Provides methods for JSON serialization and configuration management.

```python
class Arguments:
    Bases: `object`
```

--------------------------------

### Executing Find and Label Phase in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/setup/training/findAndLabel.md

A shell command to run the findAndLabel phase in Zingg with a configuration file. This command initiates both the finding of training data and subsequent labeling process in one step.

```bash
./zingg.sh --phase findAndLabel --conf config.json
```

--------------------------------

### Creating Snowflake Configuration for Zingg Enterprise in JSON

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-zingg-enterprise-snowflake/match-configuration.md

This snippet outlines the structure of a Snowflake configuration file for Zingg Enterprise. It includes placeholders for modelId, input table name, and output table name. Users need to replace these placeholders with their specific values.

```json
{
  "modelId": "28NovDev",
  "data": {
    "table": "INPUT_TABLE_NAME"
  },
  "output": {
    "table": "OUTPUT_TABLE_NAME"
  }
}
```

--------------------------------

### Creating Snowflake Connection Properties File for Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-zingg-enterprise-snowflake/snowflake-properties.md

This snippet shows the format of the snowEnv.txt file, which contains the necessary Snowflake connection properties for Zingg. It includes parameters for URL, user credentials, role, warehouse, database, schema, and a session keep-alive setting.

```plaintext
URL={snowflake_url}   
USER={snowflake_user_name} 
PASSWORD={snowflake_password}  
ROLE={role} 
WAREHOUSE={warehouse}  
DB={database_name}  
SCHEMA={schema}
CLIENT_SESSION_KEEP_ALIVE_HEARTBEAT_FREQUENCY=900
```

--------------------------------

### Running Zingg Documentation Generation Command

Source: https://github.com/zinggai/zingg/blob/main/docs/generatingdocumentation.md

Shell command to generate documentation for Zingg model training data. Outputs documentation to zinggDir/modelId folder with options to show concise or detailed views.

```bash
./scripts/zingg.sh --phase generateDocs --conf <location to conf.json> <optional --showConcise=true|false>
```

--------------------------------

### Running Zingg's verifyBlocking Phase using Shell Script

Source: https://github.com/zinggai/zingg/blob/main/docs/verifyBlocking.md

This command executes the verifyBlocking phase in Zingg to analyze blocking performance. It requires a configuration path and optionally a zinggDir parameter to specify the model location.

```shell
./scripts/zingg.sh --phase verifyBlocking --conf <path to conf> <optional --zinggDir <location of model>>
```

--------------------------------

### Running Zingg with Python Programs

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/zingg-command-line.md

Command to execute Zingg with Python programs, including an optional properties file configuration.

```shell
./scripts/zingg.sh <optional --properties-file path-to-zingg.conf> --run <path-to-python-program>
```

--------------------------------

### Executing Zingg Link Phase via Bash Command

Source: https://github.com/zinggai/zingg/blob/main/docs/setup/link.md

Command to run the link phase in Zingg, which matches records across different datasets. This requires a configuration file that defines the data sources and matching parameters.

```bash
./zingg.sh --phase link --conf config.json
```

--------------------------------

### Implementing Incremental Data Processing with Zingg Python Client

Source: https://github.com/zinggai/zingg/blob/main/docs/runIncremental.md

Python code that demonstrates how to set up and execute incremental data processing using Zingg's Python client. It configures field definitions, initializes the model, runs the findAndLabel and trainMatch phases, and then processes incremental data.

```python
#import the packages

from zingg.client import *
from zingg.pipes import *
from zinggEC.enterprise.common.ApproverArguments import *
from zinggEC.enterprise.common.IncrementalArguments import *
from zinggEC.enterprise.common.epipes import *
from zinggEC.enterprise.common.EArguments import *
from zinggEC.enterprise.common.EFieldDefinition import EFieldDefinition
from zinggES.enterprise.spark.ESparkClient import EZingg
import os

#build the arguments for zingg
args = EArguments()
#set field definitions
recId = EFieldDefinition("recId", "string", MatchType.DONT_USE)
recId.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = EFieldDefinition("add1","string", MatchType.FUZZY)
add2 = EFieldDefinition("add2", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.FUZZY)
areacode = EFieldDefinition("areacode", "string", MatchType.FUZZY)
state = EFieldDefinition("state", "string", MatchType.FUZZY)
dob = EFieldDefinition("dob", "string", MatchType.FUZZY)
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)

fieldDefs = [recId, fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]
args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("/tmp/models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)

#reading dataset into inputPipe and settint it up in 'args'
schema = "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn  string"
inputPipe = ECsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)

outputPipe = ECsvPipe("resultFebrl", "/tmp/febrlOutput")
outputPipe.setHeader("true")

args.setOutput(outputPipe)

#Run findAndLabel
options = ClientOptions([ClientOptions.PHASE,"findAndLabel"])
zingg = EZingg(args, options)
zingg.initAndExecute()

#Run trainMatch after above completes
options = ClientOptions([ClientOptions.PHASE,"trainMatch"])
zingg = EZingg(args, options)
zingg.initAndExecute()

#Now run incremental on output generated above
incrArgs = IncrementalArguments()
incrArgs.setParentArgs(args)
incrPipe = ECsvPipe("testFebrlIncr", "examples/febrl/test-incr.csv", schema)
incrArgs.setIncrementalData(incrPipe)

options = ClientOptions([ClientOptions.PHASE,"runIncremental"])
zingg = EZingg(incrArgs, options)
zingg.initAndExecute()
```

--------------------------------

### Running findTrainingData Phase in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/setup/training/findTrainingData.md

Command to execute Zingg's findTrainingData phase using a configuration file. This command instructs Zingg to search for edge cases in the data that can be used for training purposes, writing the results to the directory specified in the config file.

```bash
./zingg.sh --phase findTrainingData --conf config.json
```

--------------------------------

### Running Zingg Docker Image with Volume Mapping for Permission Issues

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/docker/README.md

Command to run the Zingg Docker container with volume mapping to solve permission denied issues by mapping the host's /tmp directory to the container's /tmp directory.

```bash
docker run -v /tmp:/tmp -it zingg/zingg:0.5.0 bash
```

--------------------------------

### Pulling and Running Zingg Docker Image

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/docker/README.md

Commands to pull the Zingg Docker image from Docker Hub and run it in interactive mode with a bash shell. The image contains all the dependencies required to run Zingg version 0.5.0.

```bash
docker pull zingg/zingg:0.5.0
docker run -it zingg/zingg:0.5.0 bash
```

--------------------------------

### Configuring and Executing Zingg for Entity Matching in Python

Source: https://github.com/zinggai/zingg/blob/main/docs/python/markdown/index.md

This snippet demonstrates the complete workflow for using Zingg's entity matching capabilities. It shows how to define fields with match types, configure Zingg arguments including model parameters, set up input/output data pipes for CSV files, and execute the matching process.

```python
from zingg.client import *
from zingg.pipes import *

#build the arguments for zingg
args = Arguments()
#set field definitions
fname = FieldDefinition("fname", "string", MatchType.FUZZY)
lname = FieldDefinition("lname", "string", MatchType.FUZZY)
stNo = FieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = FieldDefinition("add1","string", MatchType.FUZZY)
add2 = FieldDefinition("add2", "string", MatchType.FUZZY)
city = FieldDefinition("city", "string", MatchType.FUZZY)
areacode = FieldDefinition("areacode", "string", MatchType.FUZZY)
state = FieldDefinition("state", "string", MatchType.FUZZY)
dob = FieldDefinition("dob", "string", MatchType.FUZZY)
ssn = FieldDefinition("ssn", "string", MatchType.FUZZY)

fieldDefs = [fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]

args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)

#reading dataset into inputPipe and settint it up in 'args'
#below line should not be required if you are reading from in memory dataset
#in that case, replace df with input df
schema = "id string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn  string"
inputPipe = CsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
outputPipe = CsvPipe("resultFebrl", "/tmp/febrlOutput")

args.setOutput(outputPipe)

options = ClientOptions([ClientOptions.PHASE,"match"])

#Zingg execution for the given phase
zingg = Zingg(args, options)
zingg.initAndExecute()
```

--------------------------------

### Setting Google Application Credentials Environment Variable

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/bigquery.md

Sets the environment variable that points to the Google service account key file for authentication when running Zingg outside Google cloud.

```bash
export GOOGLE_APPLICATION_CREDENTIALS=path to google service account key file
```

--------------------------------

### Configuring CSV Data Source in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/configuration/data-input-and-output/data.md

This snippet demonstrates how to configure a CSV data source in Zingg using the data array and Zingg Pipes. It includes the definition of the source name, format, properties like delimiter and header information, and a detailed schema definition for the febrl test dataset. The schema defines field names and types for all columns in the CSV file.

```json
 "data" : [ {
    "name" : "test",
    "format" : "csv",
    "props" : {
      "delimiter" : ",",
      "header" : "true",
      "location" : "examples/febrl/test.csv"
    },
    "schema" : "{
      \"type\":\"struct\",
      \"fields\":[
      {\"name\":\"id\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"firstName\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"lastName\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"streetnumber\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"street\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"address1\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"address2\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"areacode\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"stateCode\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"dateOfbirth\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
      {\"name\":\"ssn\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}
      ]
      }"
  }
```

--------------------------------

### Running Zingg Lookup Phase via Shell Script

Source: https://github.com/zinggai/zingg/blob/main/docs/runApproval.md

Command to execute the lookup phase in Zingg Enterprise using the provided shell script and referencing a lookup configuration file.

```bash
./scripts/zingg.sh --phase runLookup --conf <location to lookupConf.json>
```

--------------------------------

### Generating Stopword Recommendations with Zingg CLI

Source: https://github.com/zinggai/zingg/blob/main/docs/accuracy/stopWordsRemoval.md

Command to invoke Zingg's stopword recommendation feature which analyzes a specific column and suggests words to ignore during matching.

```bash
./scripts/zingg.sh --phase recommend --conf <conf.json> --column <name of column to generate stopword recommendations>
```

--------------------------------

### Configuring Exasol as a Data Sink in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/exasol.md

JSON configuration for using Exasol as an output destination in Zingg. This defines connection parameters and output settings, including the option to create a new table for storing entity resolution results.

```json
...\n\"output\": [\n   {\n       \"name\": \"output\",\n       \"format\": \"com.exasol.spark\",\n       \"props\": {\n           \"host\": \"10.11.0.2\",\n           \"port\": \"8563\",\n           \"username\": \"sys\",\n           \"password\": \"exasol\",\n           \"create_table\": \"true\",\n           \"table\": \"DB_SCHEMA.ENTITY_RESOLUTION\",\n       },\n       \"mode\": \"Append\"\n   }\n],\n...
```

--------------------------------

### Configuring Spark Jars for BigQuery Integration

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/bigquery.md

Sets the required BigQuery driver jars in the Spark configuration. Two drivers are needed: spark-bigquery-with-dependencies and gcs-connector-hadoop2 for working with BigQuery.

```bash
spark.jars=./spark-bigquery-with-dependencies_2.12-0.24.2.jar,./gcs-connector-hadoop2-latest.jar
```

--------------------------------

### Documenting Zingg Client Module

Source: https://github.com/zinggai/zingg/blob/main/python/docs/zingg.rst

Sphinx autodoc directive to generate documentation for the zingg.client module including all members, undocumented members and inheritance hierarchy.

```rst
.. automodule:: zingg.client
   :members:
   :undoc-members:
   :show-inheritance:
```

--------------------------------

### Setting Google Hadoop FileSystem Implementation for BigQuery

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/bigquery.md

Configures the Hadoop FileSystem implementation for Google Cloud Storage which is required for BigQuery connectivity.

```bash
spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem                                                      
```

--------------------------------

### Executing Zingg with Shared Configuration Location

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/docker/shared-locations.md

This command demonstrates how to run Zingg's label phase using a configuration file and specifying a shared location for the zinggDir. The zinggDir is where model information and other editable files like config.json should be stored.

```bash
zingg.sh --phase label --conf config.json --zinggDir /location
```

--------------------------------

### Running Zingg with Custom Properties File

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/zingg-runtime-properties.md

Command to run Zingg with a custom properties file that defines JVM settings and other runtime configurations. The properties file allows specification of memory allocation, external dependencies, and other Spark configuration options.

```shell
./scripts/zingg.sh --properties-file <location to file> --conf conf.json
```

--------------------------------

### Initializing FieldDefinition Class - Python

Source: https://github.com/zinggai/zingg/blob/main/docs/python/markdown/zingg.md

Class definition for configuring field properties used in data matching. Takes name, dataType and matchType as parameters to define how each field should be processed during matching.

```python
class FieldDefinition(name, dataType, *matchType):
    Bases: `object`
```

--------------------------------

### Configuring BigQuery as a Data Source in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/bigquery.md

JSON configuration for reading data from BigQuery in Zingg. Specifies credentials file, table location, and enables views for reading data from BigQuery.

```json
    "data" : [{
        "name":"test", 
         "format":"bigquery", 
        "props": {
            "credentialsFile": "/home/work/product/final/zingg-1/mynotification-46566-905cbfd2723f.json",
            "table": "mynotification-46566.zinggdataset.zinggtest",
            "viewsEnabled": true
        }
    }],
```

--------------------------------

### Configuring Spark Jars for Exasol in Zingg Properties

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/exasol.md

Updates the spark.jars parameter in Zingg's runtime properties to include the Exasol Spark connector. This configuration enables Zingg to utilize Exasol's dependencies for database operations.

```properties
spark.jars=spark-connector_2.12-1.3.0-spark-3.3.2-assembly.jar
```

--------------------------------

### Generating Stopword Recommendations in Zingg Enterprise Snowflake

Source: https://github.com/zinggai/zingg/blob/main/docs/accuracy/stopWordsRemoval.md

Command to invoke Zingg's stopword recommendation feature specifically for Snowflake Enterprise edition, which includes additional parameters for Snowflake properties.

```bash
./scripts/zingg.sh --phase recommend --conf <conf.json> --properties-file <path to Snowflake properties file> --column <name of column to generate stopword recommendations>
```

--------------------------------

### Documenting Zingg Pipes Module

Source: https://github.com/zinggai/zingg/blob/main/python/docs/zingg.rst

Sphinx autodoc directive to generate documentation for the zingg.pipes module including all members, undocumented members and inheritance hierarchy.

```rst
.. automodule:: zingg.pipes
   :members:
   :undoc-members:
   :show-inheritance:
```

--------------------------------

### Configuring Cassandra Output in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/cassandra.md

JSON configuration for setting up Cassandra as an output data sink in Zingg. This snippet defines the connection properties including table name, keyspace, cluster name, and connection host. It also specifies Spark-specific connection properties and the write mode.

```json
"output" : [
			{
			"name":"sampleTest", 
			"format":"CASSANDRA" ,
			"props": {
				"table":"dataschematest",
				"keyspace":"zingg",
				"cluster":"zingg",
				"spark.cassandra.connection.host":"192.168.0.6"
			},
			"sparkProps": {
				"spark.cassandra.connection.host":"127.0.0.1"
			},
			"mode":"Append"
		}
		]
```

--------------------------------

### Implementing Snowflake Pipe in Python

Source: https://github.com/zinggai/zingg/blob/main/docs/python/markdown/zingg.md

The SnowflakePipe class is used for working with Snowflake pipelines. It provides methods for setting various Snowflake-specific parameters such as database, schema, and credentials.

```Python
class SnowflakePipe(Pipe):
    DATABASE = 'sfDatabase'
    DBTABLE = 'dbtable'
    PASSWORD = 'sfPassword'
    SCHEMA = 'sfSchema'
    URL = 'sfUrl'
    USER = 'sfUser'
    WAREHOUSE = 'sfWarehouse'

    def __init__(self, name):
        super().__init__(name, 'snowflake')

    def setDatabase(self, db):
        self.addProperty(self.DATABASE, db)

    def setDbTable(self, dbtable):
        self.addProperty(self.DBTABLE, dbtable)

    def setPassword(self, passwd):
        self.addProperty(self.PASSWORD, passwd)

    def setSFSchema(self, schema):
        self.addProperty(self.SCHEMA, schema)

    def setURL(self, url):
        self.addProperty(self.URL, url)
```

--------------------------------

### Documenting Zingg Base Module

Source: https://github.com/zinggai/zingg/blob/main/python/docs/zingg.rst

Sphinx autodoc directive to generate documentation for the base zingg module including all members, undocumented members and inheritance hierarchy.

```rst
.. automodule:: zingg
   :members:
   :undoc-members:
   :show-inheritance:
```

--------------------------------

### Run Zingg Script in Background (nohup)

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/installing-zingg-enterprise-snowflake/running-asynchronously.md

Executes the Zingg script in the background using nohup. This allows the process to continue running even after the user logs out or the SSH connection is terminated. The output is redirected to nohup.out.

```Shell
nohup ./scripts/zingg.sh --properties-file ~/zingg/snowEnv.txt --phase findTrainingData --conf ~/zingg/snowConfigFile.json &
```

--------------------------------

### Configuring Combined Match Models in Zingg Enterprise (JSON)

Source: https://github.com/zinggai/zingg/blob/main/docs/relations.md

This JSON configuration demonstrates how to set up multiple vertices (spouse and household) with different data sources and matching strategies. It includes a Snowflake query for spouse data and a separate matching model for household data, combining them into a unified graph structure.

```json
{ 
    "vertices" : 
    [ 
        { 
            "name" : "spouse",  
            "vertexType" : "zingg_pipe", 
            "data" : [
                {
                "name" : "spouse", 
                "format" : "snowflake", 
                "props": {
                        "query": "select a.id as id, a.FNAME, a.LNAME, a.STNO, a.ADD1, a.CITY, a.STATE, a.ZINGG_ID_PERSON, b.id as z_id, b.fname as Z_FNAME,b.lname as Z_LNAME,b.stno as Z_STNO,b.add1 as Z_ADD1, b.city as Z_CITY,b.state as Z_STATE, b.ZINGG_ID_PERSON as Z_ZINGG_ID_PERSON from CUSTOMER_RELATE_PARTIAL a, CUSTOMER_RELATE_PARTIAL b where a.familyId = b.familyId"
                        }
                }
                ],
            "edges" :  
            {   "edgeType" : "same_edge",
                "edges":[
                    {
                        "dataColumn" : "zingg_personId",
                        "column" : "zingg_personId",
                        "name" : "zingg_personId1"
                    },
                    {
                        "dataColumn" : "zingg_personId",
                        "column" : "z_zingg_personId",
                        "name" : "zingg_personId2"
                    }
                ]
            }
        },
        { 
            "name" : "household",
            "config" : "$ZINGG_ENTERPRISE_HOME$/zinggEnterprise/configHousehold.json", 
            "strategy" : {
                "vDataStrategy" : "unique_edge",
                "props" : {
                        "column" : "zingg_personId",
                        "edge" : "zingg_personId,z_zingg_personId"
                    }
            },
            "vertexType" : "zingg_match", 
             "edges" :  
            {   "edgeType" : "same_edge",
                "edges":[
                    {
                        "dataColumn" : "zingg_personId",
                        "column" : "zingg_personId",
                        "name" : "zingg_personId1"
                    },
                    {
                        "dataColumn" : "zingg_personId",
                        "column" : "z_zingg_personId",
                        "name" : "zingg_personId2"
                    }
                ]
            }
        }
    ],
    "output" : [{
        "name":"relatedCustomers", 
        "format":"snowflake", 
        "props": {
            "table": "RELATED_CUSTOMERS_PARTIAL"
            }
    }],
    "strategy":"pairs_and_vertices"
}
```

--------------------------------

### Configuring Neo4j as a Data Source in Zingg

Source: https://github.com/zinggai/zingg/blob/main/docs/dataSourcesAndSinks/neo4j.md

JSON configuration for connecting to a Neo4j database as a data source in Zingg. This specifies the connection URL, labels to query, and uses the Neo4j Spark connector format.

```json
"data" : [{
		"name":"neo", 
		"format":"org.neo4j.spark.DataSource", 
		"props": {
			"url": "bolt://localhost:7687",
            "labels":"Person"		
			}	
		}]
```

--------------------------------

### Mounting Volumes with Docker for Zingg Data Persistence

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/docker/sharing-custom-data-and-config-files.md

Docker command for creating a volume mount between a local directory on the host machine and a location inside the Zingg container. This enables data persistence after the container stops, allowing generated models and data files to be saved and reused.

```bash
docker run -v <local-location>:<container-location> -it zingg/zingg:0.5.0 bash
```

--------------------------------

### Finding Docker Container ID

Source: https://github.com/zinggai/zingg/blob/main/docs/stepbystep/installation/docker/copying-files-to-and-from-the-container.md

Command to list all running Docker containers, which displays container IDs needed for the docker cp operation.

```bash
$ docker ps
```
Tokens:
Raw