Project Nessie

https://projectnessie.org/nessie-latest/
Admin
Project Nessie is a transactional catalog for data lakes that provides Git-like semantics for...
Tokens:46,676
Snippets:356
Trust Score:8.8
Update:1 year ago
Show doc for...
Tokens:
Raw
### Nessie CLI Usage Example

Source: https://projectnessie.org/nessie-latest/cli

Demonstrates how to connect to a locally running Nessie instance using its native REST API or Iceberg REST.

```Shell
CONNECT TO http://127.0.0.1:19120/iceberg
CONNECT TO http://127.0.0.1:19120/api/v2
```

--------------------------------

### Get Nessie Repository Information (MongoDB Example)

Source: https://projectnessie.org/nessie-latest/export_import

Retrieves and displays information about a Nessie repository, including commit IDs and counts. This example specifically shows how to connect to a MongoDB instance and configure the necessary Nessie properties.

```bash
java \
  -Dnessie.version.store.type=MONGODB \
  -Dquarkus.mongodb.database=nessie \
  -Dquarkus.mongodb.connection-string=mongodb://<user>:<password>@nessie.example.com:27017 \
  -jar nessie-server-admin-tool-runner.jar \
  info
```

--------------------------------

### Install Nessie CLI (Python)

Source: https://projectnessie.org/nessie-latest/pynessie

Installs the Nessie CLI using pip, requiring Python 3. This command fetches and installs the `pynessie` package.

```Shell
# python 3 required
pip install pynessie

```

--------------------------------

### Get Trino Static Catalog Config (wget)

Source: https://projectnessie.org/nessie-latest/trino

Downloads the starter configuration SQL for a Trino 'static' catalog configuration to a 'nessie.properties' file using wget. This is used for Nessie Catalog integration.

```bash
NESSIE_BASE_URL="http://127.0.0.1:19120/"
wget --content-disposition "${NESSIE_CATALOG_URL}/iceberg-ext/v1/client-template/trino?format=static"
```

--------------------------------

### Example Trino Dynamic Catalog SQL

Source: https://projectnessie.org/nessie-latest/trino

An example SQL statement to create a Trino catalog with dynamic configuration for Nessie Catalog. It specifies the Iceberg connector, S3 details, and REST catalog settings.

```sql
/*
 * Example Trino starter configuration properties for warehouse s3://bucket-yzeq68
 * generated by Nessie to be placed for example in
 * /etc/trino/catalogs/nessie.properties within a Trino container/pod when.
 * using Trino 'static' configurations.
 * 
 * This starter configuration must be inspected and verified to validate that
 * all options and values match your specific needs and no mandatory options
 * are missing or superfluous options are present.
 * 
 * When using OAuth2, you have to supply the 'iceberg.rest-catalog.oauth2.token'
 * configuration.
 * 
 * WARNING! Trino lacks functionality to configure the oauth endpoint and is therefore
 * unable to work with any Iceberg REST catalog implementation and demands a standard
 * OAuth2 server like Keycloak or Authelia. If you feel you need client-ID/secret flow,
 * please report an issue against Trino.
 * 
 * No guarantees that this configuration works for your specific needs.
 * Use at your own risk! 
 * Do not distribute the contents as those may contain sensitive information!
 */
CREATE CATALOG nessie
  USING iceberg
  WITH (
    "connector.name" = 'iceberg',
    "fs.native-s3.enabled" = 'true',
    "iceberg.catalog.type" = 'rest',
    "iceberg.rest-catalog.uri" = 'http://localhost:33405/iceberg/',
    "iceberg.rest-catalog.vended-credentials-enabled" = 'true',
    "s3.endpoint" = 'http://localhost:37029/',
    "s3.path-style-access" = 'true',
    "s3.region" = 'us-east-1'
  );

```

--------------------------------

### Example Trino Static Catalog Properties

Source: https://projectnessie.org/nessie-latest/trino

An example Trino catalog properties file for a 'static' configuration when using Nessie Catalog. It includes settings for the Iceberg connector, S3 endpoint, and OAuth2 considerations.

```properties
#
# Example Trino starter configuration properties for warehouse s3://bucket-yzeq68
# generated by Nessie to be placed for example in
# /etc/trino/catalogs/nessie.properties within a Trino container/pod when.
# using Trino 'static' configurations.
# 
# This starter configuration must be inspected and verified to validate that
# all options and values match your specific needs and no mandatory options
# are missing or superfluous options are present.
# 
# When using OAuth2, you have to supply the 'iceberg.rest-catalog.oauth2.token'
# configuration.
# 
# WARNING! Trino lacks functionality to configure the oauth endpoint and is therefore
# unable to work with any Iceberg REST catalog implementation and demands a standard
# OAuth2 server like Keycloak or Authelia. If you feel you need client-ID/secret flow,
# please report an issue against Trino.
# 
# No guarantees that this configuration works for your specific needs.
# Use at your own risk! 
# Do not distribute the contents as those may contain sensitive information!
#

connector.name=iceberg
fs.native-s3.enabled=true
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http\://localhost\:33405/iceberg/
iceberg.rest-catalog.vended-credentials-enabled=true
s3.endpoint=http\://localhost\:37029/
s3.path-style-access=true
s3.region=us-east-1

```

--------------------------------

### Add Nessie Helm Repository and Install Chart

Source: https://projectnessie.org/nessie-latest/nessie-0-104-3

Instructions for adding the Nessie Helm repository, updating it, and installing the Nessie Helm chart version 0.104.3.

```bash
helm repo add nessie https://charts.projectnessie.org/
helm repo update
helm install my-nessie nessie/nessie --version "0.104.3"
```

--------------------------------

### Add Nessie Helm Repository and Install Nessie

Source: https://projectnessie.org/nessie-latest/index

Adds the Nessie Helm repository, updates it, and installs the Nessie Helm chart version 0.105.0.

```Shell
helm repo add nessie https://charts.projectnessie.org/
helm repo update
helm install my-nessie nessie/nessie --version "0.105.0"
```

--------------------------------

### Display Nessie Version

Source: https://projectnessie.org/nessie-latest/gc

Prints the current version information of the Nessie CLI and then exits. This is useful for verifying the installed version.

```bash
-V, --version
```

--------------------------------

### Run PostgreSQL with Docker

Source: https://projectnessie.org/nessie-latest/gc

Starts a PostgreSQL container for testing purposes, exposing port 5432. This container is configured with user, password, and database name for the Nessie GC tool.

```bash
docker run --rm -e POSTGRES_USER=pguser -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_DB=nessie_gc -p 5432:5432 postgres:16.2
```

--------------------------------

### Get Trino Dynamic Catalog Config (wget)

Source: https://projectnessie.org/nessie-latest/trino

Downloads the SQL for a Trino 'dynamic' catalog configuration to a 'create-catalog-nessie.sql' file using wget. This is for Nessie Catalog integration.

```bash
NESSIE_CATALOG_URL="http://127.0.0.1:19120/iceberg"
wget --content-disposition "${NESSIE_CATALOG_URL}/v1-clients/trino?format=dynamic"
```

--------------------------------

### Nessie CLI Default Configuration (YAML)

Source: https://projectnessie.org/nessie-latest/pynessie

Example of the default configuration file for the Nessie CLI. It specifies authentication type, timeout, Nessie endpoint, and SSL verification settings.

```YAML
auth:
    # Authentication type can be: none, bearer or aws
    type: none

    # OpenID token for the "bearer" authentication type
    # token: <OpenID token>

    timeout: 10

# Nessie endpoint
endpoint: http://localhost/api/v1

# whether to skip SSL cert verification
verify: true 

```

--------------------------------

### Get Trino Static Catalog Config (curl)

Source: https://projectnessie.org/nessie-latest/trino

Fetches the starter configuration template for a Trino 'static' catalog configuration using curl. This configuration is for integrating with Nessie Catalog.

```bash
NESSIE_BASE_URL="http://127.0.0.1:19120/"
curl "${NESSIE_BASE_URL}/iceberg-ext/v1/client-template/trino?format=static"
```

--------------------------------

### Configure Nessie Client with Apache HTTP

Source: https://projectnessie.org/nessie-latest/client_config

This example shows how to configure the Nessie client when using the Apache HTTP client. It assumes the Apache HTTP client is available on the classpath and demonstrates setting the client builder name.

```properties
nessie.client-builder-name=ApacheHttp
nessie.uri=http://localhost:1999
```

--------------------------------

### Nessie GC Hadoop Configuration Options

Source: https://projectnessie.org/nessie-latest/gc

Specifies Hadoop configuration options, which are necessary when using an Iceberg FileIO that is not S3FileIO. Examples for S3, GCS, and ADLS configurations are provided.

```bash
-H, --hadoop=<String=String>[,<String=String>...] 
                             Hadoop configuration option, required when using an Iceberg FileIO
                               that is not S3FileIO.
                             The following configuration settings might be required.

                             For S3:
                             - fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
                             - fs.s3a.access.key
                             - fs.s3a.secret.key
                             - fs.s3a.endpoint, if you use an S3 compatible object store like MinIO

                             For GCS:
                             - fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
                             - fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.
                               GoogleHadoopFS
                             - fs.gs.project.id
                             - fs.gs.auth.type=USER_CREDENTIALS
                             - fs.gs.auth.client.id
                             - fs.gs.auth.client.secret
                             - fs.gs.auth.refresh.token

                             For ADLS:
                             - fs.azure.impl=org.apache.hadoop.fs.azure.AzureNativeFileSystemStore
```

--------------------------------

### List References in Nessie CLI

Source: https://projectnessie.org/nessie-latest/cli

Lists all named references in the repository. Supports filtering references by value, starting with a prefix, or containing a substring using CEL expressions.

```Nessie CLI
LIST REFERENCES [ FILTER Value | [ STARTING WITH Value ] [ CONTAINING Value ] ]
```

--------------------------------

### Store Expiring Token in Hashicorp Vault

Source: https://projectnessie.org/nessie-latest/configuration

Provides an example of storing an expiring token in Hashicorp Vault, including the expiration timestamp in ISO format. This is crucial for time-sensitive access.

```bash
vault kv put secret/nessie-secrets/... token=value_of_the_token expiresAt=2024-12-24T18:00:00Z
```

--------------------------------

### Configure Nessie Commit Metadata

Source: https://projectnessie.org/nessie-latest/client_config

This example demonstrates how to override commit messages and authors using Nessie client configuration properties. It shows the properties for setting the commit message and author.

```properties
nessie.commit-meta.message=My custom commit message
nessie.commit-meta.authors=John Doe,Jane Smith
```

--------------------------------

### List Contents in Nessie CLI

Source: https://projectnessie.org/nessie-latest/cli

Lists all tables, views, and namespaces in a specified branch or tag. Supports filtering by value, starting with a prefix, or containing a substring. Can also specify a commit timestamp or ID.

```Nessie CLI
LIST CONTENTS [ ON [ ReferenceType ] ExistingReference ] [ AT [ TIMESTAMP | COMMIT ] TimestampOrCommit ] [ FILTER Value | [ STARTING WITH Value ] [ CONTAINING Value ] ]
```

--------------------------------

### Configure Nessie SSL Settings (Java 11+)

Source: https://projectnessie.org/nessie-latest/client_config

This example details SSL configuration options for Nessie clients using the Java 11+ HTTP client. It covers disabling certificate verification, specifying cipher suites, protocols, and SNI hosts.

```properties
nessie.ssl.no-certificate-verification=true
nessie.ssl.cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384
nessie.ssl.protocols=TLSv1.3
nessie.ssl.sni-hosts=example.com,api.example.com
```

--------------------------------

### Configure Nessie Catalog in Spark (Python)

Source: https://projectnessie.org/nessie-latest/client_config

Configures a Nessie catalog within a Spark session using Python. It sets the catalog type to Iceberg and specifies 'NONE' for authentication. This example assumes a local Spark instance.

```Python
spark = SparkSession.builder \
        .config("spark.sql.catalog.<catalog_name>", "org.apache.iceberg.spark.SparkCatalog") \
        .config("spark.sql.catalog.<catalog_name>.authentication.type", "NONE") \
        .config(...) 
        .getOrCreate()
```

--------------------------------

### Configure Nessie Catalog in Spark (Java)

Source: https://projectnessie.org/nessie-latest/client_config

Configures a Nessie catalog within a Spark session using Java. It sets the catalog type to Iceberg and specifies 'NONE' for authentication. This example assumes a local Spark instance.

```Java
conf.set("spark.sql.catalog.<catalog_name>", "org.apache.iceberg.spark.SparkCatalog")
    .set("spark.sql.catalog.<catalog_name>.authentication.type", "NONE")
    .set(...);
spark = SparkSession.builder()
                    .master("local[2]")
                    .config(conf)
                    .getOrCreate();
```

--------------------------------

### Nessie GC mark-live Command Options

Source: https://projectnessie.org/nessie-latest/gc

Details the various command-line options for the `mark-live` command, including default cutoff policies, cutoff policies per reference, help flags, parallelism settings, storage flags (in-memory, JDBC), JDBC connection details, Nessie client configuration, and Nessie options.

```bash
-c, --default-cutoff=<defaultCutoffPolicy>
                           Default cutoff policy. Policies can be one of:
                           - number of commits as an integer value
                           - a duration (see java.time.Duration)
                           - an ISO instant
                           - 'NONE', means everything's considered as live
-C, --cutoff[=<String=String>[,<String=String>...]...]
                           Cutoff policies per reference names. Supplied as a
                             ref-name-pattern=policy tuple.
                           Reference name patterns are regular expressions.
                           Policies can be one of:
                           - number of commits as an integer value
                           - a duration (see java.time.Duration)
                           - an ISO instant
                           - 'NONE', means everything's considered as live
-h, --help                 Show this help message and exit.
    --identify-parallelism=<parallelism>
                           Number of Nessie references that can be walked in parallel.
    --inmemory             Flag whether to use the in-memory contents storage. Prefer a JDBC
                               storage.
    --jdbc                 Flag whether to use the JDBC contents storage.
    --jdbc-fetch-size[=<fetchSize>]
                           JDBC fetch size, defaults to 100.
    --jdbc-password=<password>
                           JDBC password used to authenticate the database access.
    --jdbc-properties[=<String=String>[,<String=String>...]...]
                           JDBC parameters.
    --jdbc-schema=<schemaCreateStrategy>
                           How to create the database schema. Possible values: CREATE,
                             DROP_AND_CREATE, CREATE_IF_NOT_EXISTS.
    --jdbc-url=<url>       JDBC URL of the database to connect to.
    --jdbc-user=<user>     JDBC user name used to authenticate the database access.
    --nessie-api=<nessieApi>
                           Class name of the NessieClientBuilder implementation to use, defaults
                             to HttpClientBuilder suitable for REST. Using this parameter is not
                             recommended. Prefer the --nessie-client parameter instead.
    --nessie-client=<nessieClientName>
                           Name of the Nessie client to use, defaults to HTTP suitable for REST.
-o, --nessie-option[=<String=String>[,<String=String>...]...]
                           Parameters to configure the NessieClientBuilder.
-R, --cutoff-ref-time=<cutoffPolicyRefTime>
                           Reference timestamp for durations specified for --cutoff. Defaults to
                             'now'.
    --time-zone=<zoneId>   Time zone ID used to show timestamps.
                           Defaults to system time zone.
```

--------------------------------

### Show Nessie GC Help

Source: https://projectnessie.org/nessie-latest/gc

Displays all available commands and options for the Nessie GC tool when run as a standalone uber-jar. This is useful for understanding the tool's capabilities and usage.

```Shell
java -jar nessie-gc.jar --help
```

--------------------------------

### List Live-Sets with Nessie CLI

Source: https://projectnessie.org/nessie-latest/gc

This snippet shows the usage of the `list` command in the Nessie CLI to display live-sets. It details the options for configuring storage (in-memory or JDBC), including JDBC connection parameters like URL, user, password, properties, fetch size, and schema creation strategy. It also includes options for help, version, and time zone settings.

```bash
Usage: nessie-gc.jar list [-hV] [--time-zone=<zoneId>] ([--inmemory] | [[--jdbc] --jdbc-url=<url>
                          [--jdbc-properties[=<String=String>[,<String=String>...]...]]
                          [--jdbc-user=<user>] [--jdbc-password=<password>] [--jdbc-fetch-size
                          [=<fetchSize>]] [--jdbc-schema=<schemaCreateStrategy>]])
List existing live-sets, must not be used with the in-memory contents-storage.
  -h, --help                 Show this help message and exit.
      --inmemory             Flag whether to use the in-memory contents storage. Prefer a JDBC
                               storage.
      --jdbc                 Flag whether to use the JDBC contents storage.
      --jdbc-fetch-size[=<fetchSize>]
                             JDBC fetch size, defaults to 100.
      --jdbc-password=<password>
                             JDBC password used to authenticate the database access.
      --jdbc-properties[=<String=String>[,<String=String>...]...]
                             JDBC parameters.
      --jdbc-schema=<schemaCreateStrategy>
                             How to create the database schema. Possible values: CREATE,
                               DROP_AND_CREATE, CREATE_IF_NOT_EXISTS.
      --jdbc-url=<url>       JDBC URL of the database to connect to.
      --jdbc-user=<user>     JDBC user name used to authenticate the database access.
      --time-zone=<zoneId>   Time zone ID used to show timestamps.
                             Defaults to system time zone.
  -V, --version              Print version information and exit.
```

--------------------------------

### Allow Branch Listing by Role and Branch Name

Source: https://projectnessie.org/nessie-latest/authorization

This rule allows viewing references (branches/tags) if the role starts with 'test_' and the reference name starts with 'allowedBranch'. It uses string prefix matching.

```CEL
op=='VIEW_REFERENCE' && role.startsWith('test_') && ref.startsWith('allowedBranch')
```

--------------------------------

### Nessie Client Configuration Options

Source: https://projectnessie.org/nessie-latest/gc

Provides options for configuring Nessie client behavior, including parallelism, storage types, and API endpoints.

```bash
--identify-parallelism=<parallelism>
--inmemory
--jdbc
--jdbc-fetch-size[=<fetchSize>]
--jdbc-password=<password>
--jdbc-properties[=<String=String>[,<String=String>...]...]
--jdbc-schema=<schemaCreateStrategy>
--jdbc-url=<url>
--jdbc-user=<user>
--max-file-modification=<maxFileModificationTime>
--nessie-api=<nessieApi>
--nessie-client=<nessieClientName>
-o, --nessie-option[=<String=String>[,<String=String>...]...]
-R, --cutoff-ref-time=<cutoffPolicyRefTime>
--time-zone=<zoneId>
-u, --uri=<nessieUri>
-V, --version
--write-live-set-id-to=<liveSetIdFile>
```

--------------------------------

### Download and Run Nessie Server Admin Tool Docker Image (GitHub Container Registry)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie Server Admin Tool Docker image from GitHub Container Registry and runs it with help command.

```Shell
docker pull ghcr.io/projectnessie/nessie-server-admin:0.105.0
docker run ghcr.io/projectnessie/nessie-server-admin:0.105.0 --help
```

--------------------------------

### Allow Listing Commit Log for Branches Starting with 'dev'

Source: https://projectnessie.org/nessie-latest/authorization

This rule grants permission to list the commit log for any reference (branch or tag) whose name starts with 'dev'. It allows both 'VIEW_REFERENCE' and 'LIST_COMMIT_LOG' operations.

```CEL
op in ['VIEW_REFERENCE', 'LIST_COMMIT_LOG'] && ref.startsWith('dev')
```

--------------------------------

### Download and Run Nessie Server Docker Image (GitHub Container Registry)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie Server Docker image from GitHub Container Registry and runs it, exposing necessary ports.

```Shell
docker pull ghcr.io/projectnessie/nessie:0.105.0
docker run -p 19120:19120 -p 9000:9000 ghcr.io/projectnessie/nessie:0.105.0
```

--------------------------------

### Nessie GC Tool Usage and Commands

Source: https://projectnessie.org/nessie-latest/gc

Displays the usage information and available commands for the Nessie GC tool. This includes commands for schema creation, garbage collection, managing live-sets, and utility functions like showing licenses.

```shell
Usage: nessie-gc.jar [-hV] [COMMAND]
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  help                           Display help information about the specified command.
  mark-live, identify, mark      Run identify-live-content phase of Nessie GC, must not be used
                                   with the in-memory contents-storage.
  sweep, expire                  Run expire-files + delete-orphan-files phase of Nessie GC using a
                                   live-contents-set stored by a previous run of the mark-live
                                   command, must not be used with the in-memory contents-storage.
  gc                             Run identify-live-content and expire-files + delete-orphan-files.
  list                           List existing live-sets, must not be used with the in-memory
                                   contents-storage.
  delete                         Delete a live-set, must not be used with the in-memory
                                   contents-storage.
  list-deferred                  List files collected as deferred deletes, must not be used with the
                                   in-memory contents-storage.
  deferred-deletes               Delete files collected as deferred deletes, must not be used with the
                                   in-memory contents-storage.
  show                           Show information of a live-content-set, must not be used with the
                                   in-memory contents-storage.
  show-sql-create-schema-script  Print DDL statements to create the schema.
  create-sql-schema              JDBC schema creation.
  completion-script              Extracts the command-line completion script.
  show-licenses                  Show 3rd party license information.
```

--------------------------------

### Configure Nessie Bearer Authentication Token

Source: https://projectnessie.org/nessie-latest/client_config

This example demonstrates how to configure the token for Bearer authentication in Nessie. This is used when the `nessie.authentication.type` is set to `BEARER`.

```properties
nessie.authentication.token=your_bearer_token_here
```

--------------------------------

### Download and Run Nessie GC Tool Docker Image (GitHub Container Registry)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie GC Tool Docker image from GitHub Container Registry and runs it with help command.

```Shell
docker pull ghcr.io/projectnessie/nessie-gc:0.105.0
docker run ghcr.io/projectnessie/nessie-gc:0.105.0 --help
```

--------------------------------

### Show Nessie CLI Help (Shell)

Source: https://projectnessie.org/nessie-latest/pynessie

Displays the help message for the Nessie CLI, listing all available commands and options. This is a fundamental command for understanding the CLI's capabilities.

```Shell
$ nessie --help

```

--------------------------------

### Configure Nessie OAuth2 Authorization Endpoint

Source: https://projectnessie.org/nessie-latest/client_config

Sets the URL for the OAuth2 authorization endpoint, used in the Authorization Code grant flow. Example for Keycloak.

```properties
nessie.authentication.oauth2.auth-endpoint = https://<keycloak-server>/realms/<realm-name>/protocol/openid-connect/auth
```

--------------------------------

### Download and Run Nessie Server Admin Tool Docker Image (GitHub Container Registry)

Source: https://projectnessie.org/nessie-latest/nessie-0-104-3

Provides commands to pull the Nessie Server Admin Tool Docker image from GitHub Container Registry and run it with the --help flag.

```docker
docker pull ghcr.io/projectnessie/nessie-server-admin:0.104.3
docker run ghcr.io/projectnessie/nessie-server-admin:0.104.3 --help
```

--------------------------------

### Configure Nessie OAuth2 Token Endpoint

Source: https://projectnessie.org/nessie-latest/client_config

Sets the direct URL for the OAuth2 token endpoint. This can be used instead of or in conjunction with the issuer URL. Example for Keycloak.

```properties
nessie.authentication.oauth2.token-endpoint = https://<keycloak-server>/realms/<realm-name>/protocol/openid-connect/token
```

--------------------------------

### Download and Run Nessie GC Tool Docker Image (Quay.io)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie GC Tool Docker image from Quay.io and runs it with help command.

```Shell
docker pull quay.io/projectnessie/nessie-gc:0.105.0
docker run quay.io/projectnessie/nessie-gc:0.105.0 --help
```

--------------------------------

### Run Nessie Server Admin Tool Help Command

Source: https://projectnessie.org/nessie-latest/export_import

Displays all available commands and options for the Nessie Server Admin Tool. This is useful for discovering the full range of functionalities and their specific arguments.

```bash
java -jar nessie-server-admin-tool-x.y.z-runner.jar help
```

--------------------------------

### Download and Run Nessie GC Tool Docker Image (Quay.io)

Source: https://projectnessie.org/nessie-latest/nessie-0-104-3

Provides commands to pull the Nessie GC Tool Docker image from Quay.io and run it with the --help flag.

```docker
docker pull quay.io/projectnessie/nessie-gc:0.104.3
docker run quay.io/projectnessie/nessie-gc:0.104.3 --help
```

--------------------------------

### Allow Deleting Entity by Path Prefix for All Roles

Source: https://projectnessie.org/nessie-latest/authorization

This rule permits deleting entities where the path starts with 'dev.' for any role. It includes 'VIEW_REFERENCE' and 'DELETE_ENTITY' as allowed operations.

```CEL
op in ['VIEW_REFERENCE', 'DELETE_ENTITY'] && path.startsWith('dev.')
```

--------------------------------

### Download and Run Nessie GC Tool Docker Image (GitHub Container Registry)

Source: https://projectnessie.org/nessie-latest/nessie-0-104-3

Provides commands to pull the Nessie GC Tool Docker image from GitHub Container Registry and run it with the --help flag.

```docker
docker pull ghcr.io/projectnessie/nessie-gc:0.104.3
docker run ghcr.io/projectnessie/nessie-gc:0.104.3 --help
```

--------------------------------

### Nessie GC mark-live Command Usage

Source: https://projectnessie.org/nessie-latest/gc

Provides the command-line usage for the `mark-live` command in Nessie GC. It outlines the available options for configuring cutoff policies, parallelism, Nessie API/client details, storage options (in-memory or JDBC), and JDBC connection parameters.

```bash
Usage: nessie-gc.jar mark-live [-hV] [-c=<defaultCutoffPolicy>]
                               [--identify-parallelism=<parallelism>] [--nessie-api=<nessieApi>]
                               [--nessie-client=<nessieClientName>] [-R=<cutoffPolicyRefTime>]
                               [--time-zone=<zoneId>] [-u=<nessieUri>]
                               [--write-live-set-id-to=<liveSetIdFile>] [-C[=<String=String>[, 
                               <String=String>...]...]]... [-o[=<String=String>[, 
                               <String=String>...]...]]... ([--inmemory] | [[--jdbc] 
                               --jdbc-url=<url> [--jdbc-properties[=<String=String>[, 
                               <String=String>...]...]]... [--jdbc-user=<user>] 
                               [--jdbc-password=<password>] [--jdbc-fetch-size[=<fetchSize>]] 
                               [--jdbc-schema=<schemaCreateStrategy>]])
Run identify-live-content phase of Nessie GC, must not be used with the in-memory contents-storage.
```

--------------------------------

### Configure Nessie OAuth2 Issuer URL

Source: https://projectnessie.org/nessie-latest/client_config

Sets the issuer URL for the OAuth2 provider, used for discovering endpoints. This is an alternative to setting the token endpoint directly. Example for Keycloak.

```properties
nessie.authentication.oauth2.issuer-url = https://<keycloak-server>/realms/<realm-name>
```

--------------------------------

### Nessie CLI Command Line Options

Source: https://projectnessie.org/nessie-latest/cli

Provides a comprehensive list of command-line options for the Nessie CLI, including help, history, quiet mode, version, script execution, and connection parameters.

```Shell
Usage: nessie-cli.jar [-hHqV] [--no-up-to-date-check] [--non-ansi] [-j=<historyFile>] [[-K] [-E]
                      [-s=<scriptFile> | -c[=<commands>...] [-c[=<commands>...]]...] [-u=<uri>
                      [--client-name=<clientName>] [-o[=<String=String>[,<String=String>...]...] [-r=<initialReference>]]

The Nessie CLI
See https://projectnessie.org/nessie-latest/cli/ for documentation.

  -h, --help                Show this help message and exit.
  -H, --[no-]history        Allows disabling the command history file in the REPL.
                            Default is to save the command history.
  -j, --history-file=<historyFile>
                            Specify an alternative history file for the REPL.
                              Default: ~/.nessie/nessie-cli.history
      --no-up-to-date-check Optionally disable the up-to-date check.
                            Only effective if --quiet is not specified.
      --non-ansi            Allows disabling the (default) ANSI mode. Disabling ANSI support can be
                              useful in non-interactive scripts.
  -q, --quiet               Quiet option - omit the welcome and exit output.
  -V, --version             Print version information and exit.

Statements to execute before or without running the REPL
========================================================

  -c, --command[=<commands>...]
                            Nessie CLI commands to run. Each value represents one command.
                            The process will exit once all specified commands have been executed.
                              To keep the REPL running in case of errors, specify the
                              --keep-running option.
  -E, --continue-on-error   When running commands via the --command or --run-script option the
                              process will stop/exit when a command could not be parsed or ran into
                              an error.
                            Specifying this option lets the REPL continue executing the remaining
                              commands after parse or runtime errors.
  -K, --keep-running        When running commands via the --command or --run-script option the
                              process will exit once the commands have been executed.
                            To keep the REPL running, specify this option.See the
                              --continue-on-error option.
  -s, --run-script=<scriptFile>
                            Run the commands in the Nessie CLI script referenced by this option.
                            Possible values are either a file path or use the minus character ('-')
                              to read the script from stdin.

Connect options
===============

      --client-name=<clientName>
                            Name of the client implementation to use, defaults to HTTP suitable for
                              Nessie REST API.
                            See https://projectnessie.org/nessie-latest/client_config/ for the
                              'nessie.client-builder-name' option.
  -o, --client-option[=<String=String>[,<String=String>...]...]
                            Parameters to configure the REST client.
                            See https://projectnessie.org/nessie-latest/client_config/
  -r, --initial-reference=<initialReference>
                            Name of the Nessie reference to use.
  -u, --uri=<uri>           REST API endpoint URI to connect to.
                            See 'HELP CONNECT' in the REPL.
```

--------------------------------

### Enable Remote Debugging in Docker

Source: https://projectnessie.org/nessie-latest/configuration

This example illustrates how to enable remote debugging for Nessie in a Docker container by setting the `JAVA_DEBUG` environment variable to 'true'. The default debugging port is 5005.

```Shell
docker run ... -e JAVA_DEBUG=true -e JAVA_DEBUG_PORT=5005 ...
```

--------------------------------

### Download and Run Nessie Server Standalone JAR

Source: https://projectnessie.org/nessie-latest/index

Downloads the Nessie Server standalone JAR file and runs it using Java.

```Shell
curl -L -o nessie-quarkus-0.105.0-runner.jar \
  https://github.com/projectnessie/nessie/releases/download/nessie-0.105.0/nessie-quarkus-0.105.0-runner.jar
java -jar nessie-quarkus-0.105.0-runner.jar
```

--------------------------------

### Download and Run Nessie CLI Docker Image (GitHub Container Registry)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie CLI Docker image from GitHub Container Registry and runs it interactively.

```Shell
docker pull ghcr.io/projectnessie/nessie-cli:0.105.0
docker run -it ghcr.io/projectnessie/nessie-cli:0.105.0
```

--------------------------------

### Download and Run Nessie Server Admin Tool Docker Image (Quay.io)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie Server Admin Tool Docker image from Quay.io and runs it with help command.

```Shell
docker pull quay.io/projectnessie/nessie-server-admin:0.105.0
docker run quay.io/projectnessie/nessie-server-admin:0.105.0 --help
```

--------------------------------

### Get Trino Dynamic Catalog Config (curl)

Source: https://projectnessie.org/nessie-latest/trino

Fetches the SQL for a Trino 'dynamic' catalog configuration using curl. This configuration is intended for use with Nessie Catalog.

```bash
NESSIE_CATALOG_URL="http://127.0.0.1:19120/iceberg"
curl "${NESSIE_CATALOG_URL}/v1-clients/trino?format=dynamic"
```

--------------------------------

### Download and Run Nessie Server Admin Tool Docker Image (Quay.io)

Source: https://projectnessie.org/nessie-latest/nessie-0-104-3

Provides commands to pull the Nessie Server Admin Tool Docker image from Quay.io and run it with the --help flag.

```docker
docker pull quay.io/projectnessie/nessie-server-admin:0.104.3
docker run quay.io/projectnessie/nessie-server-admin:0.104.3 --help
```

--------------------------------

### Download and Run Nessie Server Admin Tool Standalone JAR

Source: https://projectnessie.org/nessie-latest/index

Downloads the Nessie Server Admin Tool standalone JAR file and runs it using Java.

```Shell
curl -L -o nessie-server-admin-tool-0.105.0-runner.jar \
  https://github.com/projectnessie/nessie/releases/download/nessie-0.105.0/nessie-server-admin-tool-0.105.0-runner.jar
java -jar nessie-server-admin-tool-0.105.0-runner.jar
```

--------------------------------

### Allow Reading Entity Value by Role and Path Prefix

Source: https://projectnessie.org/nessie-latest/authorization

This rule allows a specific role ('test_user') to read entity values where the path starts with 'allowed.'. It permits both 'VIEW_REFERENCE' and 'READ_ENTITY_VALUE' operations.

```CEL
op in ['VIEW_REFERENCE', 'READ_ENTITY_VALUE'] && role=='test_user' && path.startsWith('allowed.')
```

--------------------------------

### Nessie CLI Help Command

Source: https://projectnessie.org/nessie-latest/cli

Displays available commands or detailed information about a specific Nessie CLI command. Can be used with command names for specific help, including multi-keyword commands.

```bash
HELP [USE | CONNECT | CREATE [(BRANCH | TAG | NAMESPACE)] | ALTER [NAMESPACE] | DROP [BRANCH | TAG | NAMESPACE | TABLE | VIEW] | LIST [CONTENTS | REFERENCES] | SHOW [LOG | TABLE | VIEW | NAMESPACE | REFERENCE] | ASSIGN [BRANCH | TAG] | MERGE | REVERT | HELP | EXIT | LICENSE]
```

--------------------------------

### List Nessie References (Spark SQL)

Source: https://projectnessie.org/nessie-latest/spark-sql

Lists all named references in Nessie. Supports filtering by value, starting with a specific prefix, or containing a substring. References can be filtered within a specific catalog.

```SQL
LIST REFERENCES [ FILTER <Value> | [ STARTING WITH <Value> ] [ CONTAINING <Value> ] ] [ IN <CatalogName> ]
```

--------------------------------

### Nessie GC Command Usage

Source: https://projectnessie.org/nessie-latest/gc

Provides the command-line usage and available options for the `nessie-gc.jar gc` command. This includes parameters for managing live content identification, file expiry, deletion policies, and integration with various storage backends.

```bash
Usage: nessie-gc.jar gc [-hV] [--[no-]defer-deletes]
                        [--allowed-fpp=<allowedFalsePositiveProbability>]
                        [-c=<defaultCutoffPolicy>] [--expected-file-count=<expectedFileCount>]
                        [--expiry-parallelism=<parallelism>] [--fpp=<falsePositiveProbability>]
                        [--identify-parallelism=<parallelism>]
                        [--max-file-modification=<maxFileModificationTime>]
                        [--nessie-api=<nessieApi>] [--nessie-client=<nessieClientName>]
                        [-R=<cutoffPolicyRefTime>] [--time-zone=<zoneId>] [-u=<nessieUri>]
                        [--write-live-set-id-to=<liveSetIdFile>] [-H=<String=String>[,<String=String>...]...
] [-I=<String=String>[,<String=String>...]...] [-C[=<String=String>[,<String=String>...]...]...]
[-o[=<String=String>[,<String=String>...]...]...] ([--inmemory] | [[--jdbc] --jdbc-url=<url>
[--jdbc-properties[=<String=String>[,<String=String>...]...]...]
[--jdbc-user=<user>] [--jdbc-password=<password>] [--jdbc-fetch-size[=<fetchSize>]] [--jdbc-schema=<schemaCreateStrategy>]])
```

--------------------------------

### Set Java Heap Size in Docker

Source: https://projectnessie.org/nessie-latest/configuration

This example shows how to configure the Java heap size for Nessie in a Docker container using the `JAVA_OPTS_APPEND` environment variable. It sets both the initial and maximum heap size to 8 gigabytes.

```Shell
docker run ... -e JAVA_OPTS_APPEND="-Xms8g -Xmx8g" ...
```

--------------------------------

### Configure Garbage Collection in Docker

Source: https://projectnessie.org/nessie-latest/configuration

This example demonstrates how to set a specific Garbage Collector (GC) for Nessie within a Docker container by using the `GC_CONTAINER_OPTIONS` environment variable. It overrides the default parallel GC with Shenandoah GC.

```Shell
docker run ... -e GC_CONTAINER_OPTIONS="-XX:+UseShenandoahGC" ...
```

--------------------------------

### Download and Run Nessie Server Docker Image (Quay.io)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie Server Docker image from Quay.io and runs it, exposing necessary ports.

```Shell
docker pull quay.io/projectnessie/nessie:0.105.0
docker run -p 19120:19120 -p 9000:9000 quay.io/projectnessie/nessie:0.105.0
```

--------------------------------

### Download and Run Nessie CLI Docker Image (GitHub Container Registry)

Source: https://projectnessie.org/nessie-latest/nessie-0-104-3

Provides commands to pull the Nessie CLI Docker image from GitHub Container Registry and run it interactively.

```docker
docker pull ghcr.io/projectnessie/nessie-cli:0.104.3
docker run -it ghcr.io/projectnessie/nessie-cli:0.104.3 
```

--------------------------------

### Download and Run Nessie CLI Standalone JAR

Source: https://projectnessie.org/nessie-latest/index

Downloads the Nessie CLI standalone JAR file and runs it using Java.

```Shell
curl -L -o nessie-cli-0.105.0.jar \
  https://github.com/projectnessie/nessie/releases/download/nessie-0.105.0/nessie-cli-0.105.0.jar
java -jar nessie-cli-0.105.0.jar
```

--------------------------------

### Download and Run Nessie GC Tool Standalone JAR

Source: https://projectnessie.org/nessie-latest/index

Downloads the Nessie GC Tool standalone JAR file and runs it using Java.

```Shell
curl -L -o nessie-gc-0.105.0.jar \
  https://github.com/projectnessie/nessie/releases/download/nessie-0.105.0/nessie-gc-0.105.0.jar
java -jar nessie-gc-0.105.0.jar
```

--------------------------------

### Show Nessie Table, View, or Namespace Content

Source: https://projectnessie.org/nessie-latest/cli

Retrieves and displays the content of a table, view, or namespace stored in Nessie. Allows specifying the content key and optionally a reference or timestamp/commit for lookup.

```bash
SHOW ContentKind [ON [ReferenceType] ExistingReference] [AT [TIMESTAMP | COMMIT] TimestampOrCommit] ContentKey
```

--------------------------------

### Download and Run Nessie CLI Docker Image (Quay.io)

Source: https://projectnessie.org/nessie-latest/index

Pulls the Nessie CLI Docker image from Quay.io and runs it interactively.

```Shell
docker pull quay.io/projectnessie/nessie-cli:0.105.0
docker run -it quay.io/projectnessie/nessie-cli:0.105.0
```

--------------------------------

### Nessie Server Admin Tool Command Reference

Source: https://projectnessie.org/nessie-latest/export_import

Provides a reference for all available commands in the Nessie Server Admin Tool, including their usage and brief descriptions. This helps users understand the capabilities of the tool for managing a Nessie repository.

```bash
Usage: nessie-server-admin-tool-runner.jar [-hV] [COMMAND]
Nessie Server Admin Tool
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  info                  Nessie repository information
  help                  Display help information about the specified command.
  cleanup-repository    Cleanup unreferenced data from Nessie's repository.
  cut-history           Advanced commit log manipulation command that removes
                          parents from the specified commit. Read the full help
                          message before using!
  check-content         Check content readability of active keys.
  delete-catalog-tasks  Delete persisted state of Iceberg snapshot loading
                          tasks previously executed by the Nessie Catalog.
  erase-repository      Erase current Nessie repository (all data will be lost)
                          and optionally re-initialize it.
  export                Exports a Nessie repository to the local file system.
  import                Imports a Nessie repository from the local file system.
  show-licenses         Show 3rd party license information.
```

--------------------------------

### Show Live Content Set Information (Nessie GC)

Source: https://projectnessie.org/nessie-latest/gc

Displays information about a live content set. It supports both in-memory and JDBC storage configurations. For JDBC, it requires connection details like URL, user, and optionally password, fetch size, and properties. It also allows specifying the live set ID directly or via a file, and can show base locations, content references, and deferred deletes.

```bash
Usage: nessie-gc.jar show [-BCDhV] [--time-zone=<zoneId>] ([--inmemory] | [[--jdbc]
                          --jdbc-url=<url> [--jdbc-properties[=<String=String>[,<String=String>...]...]] [--jdbc-user=<user>]
                          [--jdbc-password=<password>] [--jdbc-fetch-size[=<fetchSize>]]
                          [--jdbc-schema=<schemaCreateStrategy>]]) (-l=<liveSetId> |
                          -L=<liveSetIdFile>)
Show information of a live-content-set, must not be used with the in-memory contents-storage.
  -B, --with-base-locations  Show base locations.
  -C, --with-content-references
                             Show content references.
  -D, --with-deferred-deletes
                             Show deferred deletes.
  -h, --help                 Show this help message and exit.
      --inmemory             Flag whether to use the in-memory contents storage. Prefer a JDBC
                               storage.
      --jdbc                 Flag whether to use the JDBC contents storage.
      --jdbc-fetch-size[=<fetchSize>]
                             JDBC fetch size, defaults to 100.
      --jdbc-password=<password>
                             JDBC password used to authenticate the database access.
      --jdbc-properties[=<String=String>[,<String=String>...]...]
                             JDBC parameters.
      --jdbc-schema=<schemaCreateStrategy>
                             How to create the database schema. Possible values: CREATE,
                               DROP_AND_CREATE, CREATE_IF_NOT_EXISTS.
      --jdbc-url=<url>       JDBC URL of the database to connect to.
      --jdbc-user=<user>     JDBC user name used to authenticate the database access.
  -l, --live-set-id=<liveSetId>
                             ID of the live content set.
  -L, --read-live-set-id-from=<liveSetIdFile>
                             The file to read the live-set-id from.
      --time-zone=<zoneId>   Time zone ID used to show timestamps.
                             Defaults to system time zone.
  -V, --version              Print version information and exit.
```

--------------------------------

### Nessie GC Deferred Deletes Storage Options

Source: https://projectnessie.org/nessie-latest/gc

Explains the options for specifying the contents storage for the `deferred-deletes` command, including `--inmemory` for in-memory storage and `--jdbc` for JDBC storage with associated URL, user, password, and fetch size configurations.

```bash
--inmemory             Flag whether to use the in-memory contents storage. Prefer a JDBC
                               storage.
--jdbc                 Flag whether to use the JDBC contents storage.
--jdbc-fetch-size[=<fetchSize>]
                             JDBC fetch size, defaults to 100.
--jdbc-password=<password>
                             JDBC password used to authenticate the database access.
--jdbc-properties[=<String=String>[,<String=String>...]...]
                             JDBC parameters.
--jdbc-schema=<schemaCreateStrategy>
                             How to create the database schema. Possible values: CREATE,
                               DROP_AND_CREATE, CREATE_IF_NOT_EXISTS.
```

--------------------------------

### Download and Run Nessie Server Admin Tool Standalone JAR

Source: https://projectnessie.org/nessie-latest/nessie-0-104-3

Downloads the Nessie Server Admin Tool standalone JAR file and provides the command to run it. Requires Java 17 or later.

```bash
curl -L -o nessie-server-admin-tool-0.104.3-runner.jar \
  https://github.com/projectnessie/nessie/releases/download/nessie-0.104.3/nessie-server-admin-tool-0.104.3-runner.jar
java -jar nessie-server-admin-tool-0.104.3-runner.jar
```
Tokens:
Raw