lakeFS

https://docs.lakefs.io/v1.70/
Admin
lakeFS is a version control system for data lakes that applies Git-like semantics including...
Tokens:305,962
Snippets:1,597
Trust Score:10
Update:11 months ago
Show doc for...
Tokens:
Raw
### Installation Configuration

Source: https://docs.lakefs.io/v1.70/reference/configuration

Settings for initial server setup and user creation.

```APIDOC
## Installation Configuration

### Description
Settings for initial server setup and user creation.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Configuration Fields
- **installation.user_name** (string) - Initial admin username (requires `database.type` to be 'local' and `access_key_id`/`secret_access_key` to be set).
- **installation.access_key_id** (string) - Initial admin access key ID.
- **installation.secret_access_key** (string) - Initial admin secret access key.
- **installation.allow_inter_region_storage** (bool : true) - Allow storage in regions different from the server's region.
```

--------------------------------

### Install and Launch lakeFS using Python

Source: https://docs.lakefs.io/v1.70/quickstart/launch

Installs the lakeFS Python package and launches the quickstart environment. This requires Python and pip to be installed on the system. It provides a local instance of lakeFS ready for use.

```python
pip install lakefs
python -m lakefs.quickstart
```

--------------------------------

### GET /setup_lakefs

Source: https://docs.lakefs.io/v1.70/reference/api

Checks if the lakeFS installation is already set up. This is useful for determining whether the initial setup process needs to be run.

```APIDOC
## GET /setup_lakefs

### Description
Checks if lakeFS is already set up.

### Method
GET

### Endpoint
/setup_lakefs

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
{}

```

--------------------------------

### GET /setup_lakefs

Source: https://docs.lakefs.io/v1.70/reference/api/swagger

Checks the setup state of the lakeFS installation. This endpoint allows you to determine if lakeFS has already been configured.

```APIDOC
## GET /setup_lakefs

### Description
Checks if the lakeFS installation is already set up.

### Method
GET

### Endpoint
/setup_lakefs

### Parameters
None

### Request Example
None

### Response
#### Success Response (200)
- **SetupState** (object) - lakeFS setup state

#### Response Example
```json
{
  "example": "lakeFS setup state object"
}
```

#### Error Response
- **429 Too Many Requests**: Rate limit exceeded.
- **default**: Server Error.
```

--------------------------------

### Setup lakeFS Environment: Start Docker Compose

Source: https://docs.lakefs.io/v1.70/understand/use_cases/cicd_for_data

This command starts the necessary components for the lakeFS CI/CD demo using Docker Compose. It brings up services like the lakeFS server, the webhook server, and potentially other dependencies defined in the `docker-compose.yml` file.

```bash
docker compose up

```

--------------------------------

### Setup lakeFS Environment: Clone Samples

Source: https://docs.lakefs.io/v1.70/understand/use_cases/cicd_for_data

This command clones the lakeFS samples repository, which contains example configurations and code for implementing CI/CD pipelines with lakeFS hooks. It's the first step in setting up the demo environment.

```bash
git clone https://github.com/treeverse/lakeFS-samples.git
cd lakeFS-samples

```

--------------------------------

### Start lakeFS and Jupyter Environment with Docker Compose (Bash)

Source: https://docs.lakefs.io/v1.70/understand/use_cases/etl_testing

Initializes and updates Git submodules, then starts Docker containers defined in the `docker-compose.yml` file. This setup includes Python, Spark, Jupyter Notebook, JDK, Hadoop, lakeFS Python SDK, and Airflow.

```bash
git submodule init && git submodule update
docker compose up

```

--------------------------------

### Retrieve Currently Installed lakeFS Enterprise License via API

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/install

Demonstrates how to fetch the currently installed lakeFS Enterprise license token using an HTTP GET request to the lakeFS server API. The response contains the license JWT.

```http
GET https://your-lakefs-server/api/v1/license
```

--------------------------------

### Read Image with lakeFS and Boto3 (S3 API Compatibility)

Source: https://docs.lakefs.io/v1.70/index

This example illustrates reading an image file from a lakeFS repository using Boto3, leveraging lakeFS's compatibility with the S3 API. It showcases how existing S3 interaction methods can be used with lakeFS.

```python
>>> import boto3
>>> 
>>> s3 = boto3.resource('s3')
>>> obj = s3.Object('example-repository', 'main-branch/path/image.png')
>>> image_data = obj.get()['Body'].read()
```

--------------------------------

### Setup State

Source: https://docs.lakefs.io/v1.70/reference/api/swagger

Information about the initial setup state of the system.

```APIDOC
## Setup State

### Description
Indicates whether the system has been initialized and if communication preferences are missing.

### Endpoint
This documentation describes a state object, not a specific endpoint.

### Parameters
#### Request Body
- **state** (string) - Required - The current setup state. Enum: `initialized`, `not_initialized`.
- **comm_prefs_missing** (boolean) - Optional - True if communication preferences are missing.
- **login_config** (object) - Optional - Login configuration details, referencing the `LoginConfig` schema.

### Response
#### Success Response (200)
- **state** (string) - The current setup state.
- **comm_prefs_missing** (boolean) - Indicates if comm prefs are missing.
- **login_config** (object) - Login configuration.
```

--------------------------------

### Run lakeFS Server with Docker

Source: https://docs.lakefs.io/v1.70/howto/deploy/onprem

Docker run command to start a lakeFS instance, configuring it via environment variables. This setup uses a PostgreSQL database and S3-compatible storage like MinIO.

```bash
docker run \
    --name lakefs \
    -p 8000:8000 \
    -e LAKEFS_DATABASE_TYPE="postgres" \
    -e LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING="[DATABASE_CONNECTION_STRING]" \
    -e LAKEFS_AUTH_ENCRYPT_SECRET_KEY="[ENCRYPTION_SECRET_KEY]" \
    -e LAKEFS_BLOCKSTORE_TYPE="s3" \
    -e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE="true" \
    -e LAKEFS_BLOCKSTORE_S3_ENDPOINT="http://<minio_endpoint>" \
    -e LAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION="false" \
    -e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID="<minio_access_key>" \
    -e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY="<minio_secret_key>" \
    treeverse/lakefs:latest run

```

--------------------------------

### Example lakeFS Enterprise License API Response

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/install

Shows the structure of the JSON response when retrieving the lakeFS Enterprise license via the API. The 'token' field contains the license JWT.

```json
{
  "token": "eyJhbGciOiJSUzI1NiIs..."
}
```

--------------------------------

### lakeFS Enterprise Server Won't Start Error

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/install

Troubleshooting step for when the lakeFS Enterprise server fails to start due to a missing or invalid license. This indicates a configuration issue with the license path.

```text
Error: no valid license found for this lakeFS Enterprise Server
```

--------------------------------

### Setup User

Source: https://docs.lakefs.io/v1.70/reference/api/swagger

Defines user setup information including username and access keys.

```APIDOC
## Setup User

### Description
Used to set up a user with their username and associated access key credentials.

### Endpoint
This documentation describes a setup object, not a specific endpoint.

### Parameters
#### Request Body
- **username** (string) - Required - An identifier for the user.
- **key** (object) - Required - The user's access key credentials, referencing the `AccessKeyCredentials` schema.

### Response
#### Success Response (200)
- **username** (string) - The username.
- **key** (object) - The access key credentials.
```

--------------------------------

### Webhook Example Configuration

Source: https://docs.lakefs.io/v1.70/howto/hooks/webhooks

An example demonstrating how to configure a webhook within an Action file.

```APIDOC
```yaml
...
hooks:
- id: prevent_user_columns
    type: webhook
    description: Ensure no user_* columns under public/
    properties:
    url: "http://<host:port>/webhooks/schema"
    timeout: 1m30s
    query_params:
        disallow: ["user_", "private_"]
        prefix: public/
    headers:
        secret_header: "{% raw %}{{{% endraw %} ENV.MY_SECRET {% raw %}}}{% endraw %}"
...
```
```

--------------------------------

### lakeFS Flare Command Example Usage

Source: https://docs.lakefs.io/v1.70/enterprise/troubleshooting

An example demonstrating how to run the lakeFS flare command and redirect its output to a file. This is useful for capturing diagnostic data for later analysis.

```bash
# This will run flare with output set to stdout, which is redirected into a file
$ ./lakefs flare --stdout > lakefs.flare

```

--------------------------------

### Configure PostgreSQL Database via Environment Variables

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/install

This example demonstrates configuring lakeFS Enterprise to use PostgreSQL as its primary database by setting environment variables. It specifies the database type and the connection string, which should be securely managed.

```yaml
extraEnvVars:
  - name: LAKEFS_DATABASE_TYPE
    value: postgres
  - name: LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING
    value: '<postgres connection string>'

```

--------------------------------

### Get Help for lakectl Commands

Source: https://docs.lakefs.io/v1.70/reference/cli

Provides help information for any lakectl command. Users can specify a command path to get detailed usage instructions and available flags.

```bash
lakectl help [command] [flags]
```

--------------------------------

### Install Dependencies with pip

Source: https://docs.lakefs.io/v1.70/howto/backup-and-restore

This command installs the necessary Python dependencies for the lakeFS helper script from a requirements.txt file. Ensure you have pip installed and the requirements.txt file is in the current directory.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install lakefs-spec

Source: https://docs.lakefs.io/v1.70/integrations/huggingface_datasets

Installs the lakefs-spec library, which provides fsspec FileSystem implementations for lakeFS. This is a prerequisite for using lakeFS with HuggingFace Datasets.

```bash
pip install lakefs-spec

```

--------------------------------

### Docker Compose for lakeFS Enterprise (SSO Enabled with OIDC)

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/quickstart

This Docker Compose configuration sets up lakeFS Enterprise with Single Sign-On (SSO) integration using OpenID Connect (OIDC). It requires a separate `.env` file for OIDC provider configurations. This setup is suitable for advanced users who need to integrate with an existing identity provider.

```yaml
# docker-compose.yaml content would be here, similar to the SSO Disabled version but potentially referencing .env variables for OIDC configuration.
# For example:
# environment:
#   - LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_ID=${LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_ID}
#   - LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_SECRET=${LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_SECRET}
#   - LAKEFS_AUTH_PROVIDERS_OIDC_URL=${LAKEFS_AUTH_PROVIDERS_OIDC_URL}
#   - LAKEFS_AUTH_LOGOUT_REDIRECT_URL=${LAKEFS_AUTH_LOGOUT_REDIRECT_URL}

```

--------------------------------

### Install lakectl using Homebrew

Source: https://docs.lakefs.io/v1.70/reference/cli

Installs the lakectl command-line tool using Homebrew on Linux or macOS. This is a straightforward method for users who have Homebrew installed.

```bash
brew tap treeverse/lakefs
brew install lakefs

```

--------------------------------

### Reading a Currently Installed License Token via API

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/install

This endpoint allows you to retrieve the currently installed lakeFS Enterprise license token. This token can be decoded to view license information.

```APIDOC
## GET /api/v1/license

### Description
Retrieves the currently installed lakeFS Enterprise license token.

### Method
GET

### Endpoint
/api/v1/license

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```
GET https://your-lakefs-server/api/v1/license
```

### Response
#### Success Response (200)
- **token** (string) - The lakeFS Enterprise license token.

#### Response Example
```json
{
  "token": "eyJhbGciOiJSUzI1NiIs..."
}
```
```

--------------------------------

### Install Documentation Dependencies (Python)

Source: https://docs.lakefs.io/v1.70/project/docs

Installs all necessary Python packages for building and serving the lakeFS documentation. Requires Python 3.13+ and pip.

```shell
pip install -r docs/requirements-docs.txt

```

--------------------------------

### Get Help for lakectl Local Commands

Source: https://docs.lakefs.io/v1.70/reference/cli

Provides help information for any subcommand within the `lakectl local` command group. Users can specify a command path to get detailed usage instructions and available flags.

```bash
lakectl local help [command] [flags]
```

--------------------------------

### Install lakeFS Python Client

Source: https://docs.lakefs.io/v1.70/integrations/python

Installs the lakeFS Python client library using pip. This is the first step to using the lakeFS SDK in your Python projects.

```bash
pip install lakefs

```

--------------------------------

### Setup lakeFS Environment: Initialize and Update Submodules

Source: https://docs.lakefs.io/v1.70/understand/use_cases/cicd_for_data

These commands initialize and update git submodules within the cloned lakeFS samples repository. Submodules are often used to manage external dependencies or components required for the project setup.

```bash
git submodule init
git submodule update

```

--------------------------------

### GET /auth/policies

Source: https://docs.lakefs.io/v1.70/reference/api

Lists all policies in the system.

--------------------------------

### POST /setup_lakefs

Source: https://docs.lakefs.io/v1.70/reference/api

Sets up lakeFS and creates a first user. This endpoint is used for the initial installation of lakeFS and the creation of an administrative user.

```APIDOC
## POST /setup_lakefs

### Description
Sets up lakeFS and creates the first user.

### Method
POST

### Endpoint
/setup_lakefs

### Parameters
N/A

### Request Example
{}

### Response
#### Success Response (200)
N/A

#### Response Example
{}

```

--------------------------------

### POST /setup_lakefs

Source: https://docs.lakefs.io/v1.70/reference/api/swagger

Sets up lakeFS and creates the first user. This is typically the initial configuration step for a new lakeFS deployment.

```APIDOC
## POST /setup_lakefs

### Description
Sets up lakeFS and creates a first user.

### Method
POST

### Endpoint
/setup_lakefs

### Parameters
#### Request Body
- **Setup** (object) - Required - The schema for setting up lakeFS.

### Request Example
```json
{
  "example": "request body for Setup"
}
```

### Response
#### Success Response (200)
- **CredentialsWithSecret** (object) - user created successfully

#### Response Example
```json
{
  "example": "credentials with secret"
}
```

#### Error Response
- **400 Bad Request**
- **409 Conflict**: setup was already called.
- **429 Too Many Requests**: Rate limit exceeded.
- **default**: Server Error.
```

--------------------------------

### Initialize lakeFS Enterprise Setup

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/migrate-from-oss

Sets up a new lakeFS Enterprise instance with specified user credentials. This command is used after the migration and Helm upgrade to log in to the new pod.

```bash
lakefs setup --user-name <admin> --access-key-id <key> --secret-access-key <secret> --no-check

```

--------------------------------

### Get Group Details

Source: https://docs.lakefs.io/v1.70/security/authorization-yaml/authorization

Retrieves the details of a specific group using its unique ID. This endpoint is crucial for verifying group creation during setup.

```APIDOC
## GET /auth/groups/{groupId}

### Description
Returns the details of a specific group. Takes a unique groupId as a path parameter, which must match an existing group ID. The response includes a Group component. Internally, the name property in the Group component is converted to the id. During the setup phase, this endpoint is called with the "Admins" group ID to verify that the group was created. This endpoint should be implemented in a function named getGroup.

### Method
GET

### Endpoint
/auth/groups/{groupId}

### Parameters
#### Path Parameters
- **groupId** (string) - Required - The unique identifier for the group.

### Response
#### Success Response (200)
- **Group** (object) - Contains the details of the group, with the internal name property mapped to id.

#### Error Responses
- **401** - Unauthorized
- **404** - Not Found
- **default** - Server Error
```

--------------------------------

### Serve Documentation Locally (Makefile)

Source: https://docs.lakefs.io/v1.70/project/docs

Builds and serves the lakeFS documentation locally using the provided Makefile. Assumes dependencies are installed.

```shell
make docs-serve

```

--------------------------------

### S3-Compatible Client Credentials (.aws/credentials)

Source: https://docs.lakefs.io/v1.70/howto/garbage-collection/standalone-gc

Example AWS credentials file setup for an S3-compatible client like MinIO. It defines the access key ID and secret access key for a specific profile.

```ini
[minio]
    aws_access_key_id     = <MinIO access key>
    aws_secret_access_key = <MinIO secret key>
```

--------------------------------

### POST /setup_comm_prefs

Source: https://docs.lakefs.io/v1.70/reference/api

Sets up communication preferences for lakeFS. This endpoint is used during the initial setup process to configure how the system communicates with users.

```APIDOC
## POST /setup_comm_prefs

### Description
Sets up communication preferences.

### Method
POST

### Endpoint
/setup_comm_prefs

### Parameters
N/A

### Request Example
{}

### Response
#### Success Response (200)
N/A

#### Response Example
{}

```

--------------------------------

### Read CSV with lakeFS and Pandas

Source: https://docs.lakefs.io/v1.70/index

This snippet shows how to read a CSV file from a lakeFS repository using the Pandas library. It demonstrates lakeFS acting as a metadata layer to efficiently fetch data from underlying storage.

```python
>>> import pandas as pd
>>> 
>>> df = pd.read_csv('lakefs://example-repository/main-branch/path/to.csv')
```

--------------------------------

### Upgrading to Multi-Storage Backends

Source: https://docs.lakefs.io/v1.70/howto/multiple-storage-backends

Guidelines for transitioning from a single storage backend to a multi-storage setup in lakeFS.

```APIDOC
## Upgrading from a single storage backend to Multiple Storage backends

When upgrading from a single storage backend to a multi-storage setup, follow these guidelines:

- Use the new `blockstores` structure, **replacing** the existing `blockstore` configuration. Note that `blockstore` and `blockstores` configurations are mutually exclusive - lakeFS does not support both simultaneously.
- Define all previously available single-blockstore settings under their respective storage backends.
- The `signing.secret_key` is a required setting global to all connected stores.
- Set `backward_compatible: true` for the existing storage backend to ensure:
  - Existing repositories continue to use the original storage backend.
  - Newly created repositories default to this backend unless explicitly assigned a different one, to ensure a non-breaking upgrade process.
  - **This setting is mandatory** — lakeFS will not function if it is unset.
  - **Do not remove this setting** as long as you need to support repositories created before the upgrade. If removed, lakeFS will fail to start because it will treat existing repositories as disconnected from any configured storage.
```

--------------------------------

### Example GC Run Summary JSON

Source: https://docs.lakefs.io/v1.70/howto/garbage-collection/standalone-gc

A JSON object summarizing a lakefs-sgc garbage collection run. It includes details like the run ID, success status, start and cutoff times, and the number of deleted objects.

```json
{
    "run_id": "gcoca17haabs73f2gtq0",
    "success": true,
    "first_slice": "gcss5tpsrurs73cqi6e0",
    "start_time": "2024-10-27T13:19:26.890099059Z",
    "cutoff_time": "2024-10-27T07:19:26.890099059Z",
    "num_deleted_objects": 33000
}

```

--------------------------------

### Configure PostgreSQL Database via Shared Secret

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/install

This example configures lakeFS Enterprise to use PostgreSQL, storing the database connection string in a Kubernetes Secret. It also sets an encryption key for authentication. The `lakefsConfig` specifies the database type.

```yaml
secrets:
  authEncryptSecretKey: shared-key-hello
  databaseConnectionString: <postgres connection string>
lakefsConfig: |
  database:
    type: postgres

```

--------------------------------

### Docker Compose Configuration for lakeFS Enterprise

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/quickstart

This snippet defines a docker-compose.yaml file to run lakeFS Enterprise. It configures the lakeFS service, its connection to a PostgreSQL database, local block storage, and various authentication settings including OIDC. It also includes environment variables for license and OIDC parameters.

```yaml
version: "3"
services:
  lakefs:
    image: "treeverse/lakefs-enterprise:latest"
    command: "RUN"
    ports:
      - "8080:8080"
    depends_on:
      - "postgres"
    environment:
      - LAKEFS_LISTEN_ADDRESS=0.0.0.0:8000
      - LAKEFS_LOGGING_LEVEL=DEBUG
      - LAKEFS_LOGGING_AUDIT_LOG_LEVEL=INFO
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=shared-secret-key
      - LAKEFS_AUTH_LOGOUT_REDIRECT_URL=${LAKEFS_AUTH_LOGOUT_REDIRECT_URL}
      - LAKEFS_AUTH_UI_CONFIG_LOGIN_URL=http://localhost:8000/oidc/login
      - LAKEFS_AUTH_UI_CONFIG_LOGOUT_URL=http://localhost:8000/oidc/logout
      - LAKEFS_AUTH_UI_CONFIG_RBAC=internal
      - LAKEFS_AUTH_OIDC_FRIENDLY_NAME_CLAIM_NAME=${LAKEFS_AUTH_OIDC_FRIENDLY_NAME_CLAIM_NAME}
      - LAKEFS_AUTH_PROVIDERS_OIDC_ENABLED=true
      - LAKEFS_AUTH_PROVIDERS_OIDC_POST_LOGIN_REDIRECT_URL=http://localhost:8000/
      - LAKEFS_AUTH_PROVIDERS_OIDC_URL=${LAKEFS_AUTH_PROVIDERS_OIDC_URL}
      - LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_ID=${LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_ID}
      - LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_SECRET=${LAKEFS_AUTH_PROVIDERS_OIDC_CLIENT_SECRET}
      - LAKEFS_AUTH_PROVIDERS_OIDC_CALLBACK_BASE_URL=http://localhost:8000
      - LAKEFS_AUTH_PROVIDERS_OIDC_LOGOUT_CLIENT_ID_QUERY_PARAMETER=${LAKEFS_AUTH_OIDC_LOGOUT_CLIENT_ID_QUERY_PARAMETER}
      - LAKEFS_LICENSE_CONTENTS=${LAKEFS_LICENSE_CONTENTS} # for production use, we recommend using LAKEFS_LICENSE_PATH instead
      - LAKEFS_AUTH_PROVIDERS_OIDC_LOGOUT_CLIENT_ID_QUERY_PARAMETER=${LAKEFS_AUTH_OIDC_LOGOUT_CLIENT_ID_QUERY_PARAMETER}
      - LAKEFS_DATABASE_TYPE=postgres
      - LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=postgres://lakefs:lakefs@postgres:5432/postgres?sslmode=disable
      - LAKEFS_BLOCKSTORE_TYPE=local
      - LAKEFS_BLOCKSTORE_LOCAL_PATH=/tmp/lakefs/data
      - LAKEFS_BLOCKSTORE_LOCAL_IMPORT_ENABLED=true
      - LAKEFS_FEATURES_LOCAL_RBAC=true
    entrypoint: ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
    configs:
      - source: lakefs.yaml
        target: /etc/lakefs/config.yaml
  postgres:
    image: "postgres:11"
    ports:
      - "5433:5432"
    environment:
      POSTGRES_USER: lakefs
      POSTGRES_PASSWORD: lakefs

#This tweak is unfortunate but also necessary.
logout_endpoint_query_parameters is a list
of strings which isn't parsed nicely as env vars.
configs:
  lakefs.yaml:
    content: |
      auth:
        ui_config:
          login_cookie_names:
            - internal_auth_session
            - oidc_auth_session
        oidc:
          # friendly_name_claim_name: "name"
          default_initial_groups:
            - Admins
        providers:
          oidc:
            logout_endpoint_query_parameters:
              - returnTo
              - http://localhost:8000/oidc/login

```

--------------------------------

### Configure lakeFS Python Client (Python)

Source: https://docs.lakefs.io/v1.70/understand/use_cases/etl_testing

Sets up the lakeFS access credentials (key, secret, endpoint) and the storage namespace for the lakeFS instance. It then initializes the lakeFS client using these configurations for programmatic access.

```python
import lakefs
from lakefs import Client

lakefs_access_key = 'AKIAIOSFODNN7EXAMPLE'
lakefs_secret_key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
lakefs_endpoint = 'http://lakefs:8000'

# lakeFS credentials and endpoint
client = Client(
    host=lakefs_endpoint,
    username=lakefs_access_key,
    password=lakefs_secret_key
)

storageNamespace = 's3://example/' 

```

--------------------------------

### Query lakeFS Object Metadata with PyIceberg and DuckDB

Source: https://docs.lakefs.io/v1.70/datamanagment/metadata-search

Loads a lakeFS Iceberg table and queries its object metadata using DuckDB. This example demonstrates filtering by user-defined metadata (e.g., 'animal') and system metadata (e.g., 'last_modified'). It requires the duckdb library to be installed.

```python
from pyiceberg.catalog import load_catalog

# Initialize the catalog
catalog = RestCatalog(name = "my_catalog", **{
    'prefix': 'lakefs',
    'uri': 'https://lakefs.example.com/mds/iceberg/api',
    'oauth2-server-uri': 'https://lakefs.example.com/mds/iceberg/api/iceberg/api/v1/oauth/tokens',
    'credential': f'AKIAlakefs12345EXAMPLE:abc/lakefs/1234567bPxRfiCYEXAMPLEKEY',
})

# `repo` is the repository name we would like to search 
con = catalog.load_table('repo.main.system.object_metadata').scan().to_duckdb('object_metadata')

query = f"""
SELECT path   
FROM object_metadata
WHERE user_metadata['animal'] = 'cat' 
  AND last_modified > (now() - INTERVAL '20 days') 
"""

df = con.execute(query).df()

```

--------------------------------

### List Objects in a lakeFS Repository Branch

Source: https://docs.lakefs.io/v1.70/integrations/r

Lists objects within a specific branch of a lakeFS repository using `get_bucket_df`. It's recommended to use a branch prefix for clarity. The example lists objects in the `main` branch of the `quickstart` repository.

```r
get_bucket_df(bucket = "quickstart",
              prefix = "main/",
              region = "",
              use_https = FALSE)
```

--------------------------------

### lakeFS License Error: Installation ID Mismatch

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/install

This error occurs when the provided license token is associated with a different installation ID than the current one. Ensure you have the correct license for this installation or contact support for a new license.

```plaintext
Error: license belongs to installation ID X (current installation ID: Y)

```

--------------------------------

### Get Object Stat with lakectl fs stat

Source: https://docs.lakefs.io/v1.70/understand/data-structure

Retrieves detailed information about an object within a lakeFS repository, including its logical path, modification time, size, and physical address in the object store. The `--pre-sign=false` flag is used to disable pre-signing of the physical address. This command requires the `lakectl` CLI to be installed and configured.

```bash
lakectl fs stat --pre-sign=false lakefs://my-repo/main/allstar_games_stats.csv

```

--------------------------------

### Build Documentation Site (MkDocs)

Source: https://docs.lakefs.io/v1.70/project/docs

Builds the static site for the lakeFS documentation using MkDocs. This is a prerequisite for running the link checker locally.

```shell
mkdocs build

```

--------------------------------

### Configure lakeFS CLI

Source: https://docs.lakefs.io/v1.70/quickstart/branch

Configures the `lakectl` command-line interface with necessary credentials for interacting with a lakeFS instance. This is a prerequisite for most lakeFS operations. Ensure you have your credentials ready.

```bash
lakectl config

```

--------------------------------

### Docker Compose for lakeFS Enterprise (SSO Disabled)

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/quickstart

This Docker Compose configuration sets up a lakeFS Enterprise instance connected to a local blockstore and a PostgreSQL database. It's a recommended option for a fully functional setup without SSO support. Environment variables configure the lakeFS instance, and a separate config file handles authentication details.

```yaml
version: "3"
services:
  lakefs:
    image: "treeverse/lakefs-enterprise:latest"
    command: "RUN"
    ports:
      - "8000:8000"
    depends_on:
      - "postgres"
    environment:
      - LAKEFS_LISTEN_ADDRESS=0.0.0.0:8000
      - LAKEFS_LOGGING_LEVEL=DEBUG
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=random_secret
      - LAKEFS_AUTH_UI_CONFIG_RBAC=internal
      - LAKEFS_DATABASE_TYPE=postgres
      - LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=postgres://lakefs:lakefs@postgres:5432/postgres?sslmode=disable
      - LAKEFS_BLOCKSTORE_TYPE=local
      - LAKEFS_BLOCKSTORE_LOCAL_PATH=/home/lakefs
      - LAKEFS_BLOCKSTORE_LOCAL_IMPORT_ENABLED=true
      - LAKEFS_AUTH_POST_LOGIN_REDIRECT_URL=http://localhost:8000/
      - LAKEFS_FEATURES_LOCAL_RBAC=true
      - LAKEFS_LICENSE_CONTENTS=<license token> # for production use, we recommend using LAKEFS_LICENSE_PATH instead
    configs:
      - source: lakefs.yaml
        target: /etc/lakefs/config.yaml
  postgres:
    image: "postgres:11"
    ports:
      - "5433:5432"
    environment:
      POSTGRES_USER: lakefs
      POSTGRES_PASSWORD: lakefs

configs:
  lakefs.yaml:
    content: |
      auth:
        ui_config:
          login_cookie_names:
            - internal_auth_session

```

--------------------------------

### lakectl usage help: Get Help for Commands

Source: https://docs.lakefs.io/v1.70/reference/cli

Provides help information for any lakeFS command by specifying the command path. This is a plumbing command.

```bash
lakectl usage help [command] [flags]
```

--------------------------------

### Create Repository Python SDK

Source: https://docs.lakefs.io/v1.70/howto/multiple-storage-backends

Example of creating a repository using the High-Level Python SDK with a specified storage ID.

```APIDOC
## High-Level Python SDK - Create Repository

### Description
Demonstrates how to dynamically pass the `storage_id` when creating a repository using the lakeFS Python SDK.

### Method
High-Level Python SDK Call

### Endpoint
N/A (SDK method)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body (Implicit via SDK parameters)
- **name** (string) - Required - The name of the repository.
- **storage_namespace** (string) - Required - The storage namespace for the repository.
- **storage_id** (string) - Required - The ID of the storage backend to use.

### Request Example
```python
import lakefs

repo = lakefs.Repository("example-repo").create(
    storage_namespace="s3://storage-bucket/repos/example-repo", 
    storage_id="my-storage-id"
)
```

### Response
#### Success Response
Returns a Repository object representing the newly created repository.

#### Response Example
(Object representation of the created repository, similar to the API response)
```

--------------------------------

### GET /repositories/{repository}/branch_protection

Source: https://docs.lakefs.io/v1.70/reference/api

Gets the branch protection rules for a repository.

```APIDOC
## GET /repositories/{repository}/branch_protection

### Description
Gets branch protection rules.

### Method
GET

### Endpoint
/repositories/{repository}/branch_protection

#### Path Parameters
- **repository** (string) - Required - The name of the repository.

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
{}

```

--------------------------------

### Kubernetes Helm Chart Configuration for lakeFS Enterprise

Source: https://docs.lakefs.io/v1.70/enterprise/getstarted/quickstart

This snippet shows a sample values.yaml file for the lakeFS Helm chart. It enables enterprise features, configures private registry access with a secret token, and sets lakeFS configurations for logging, blockstore, and authentication. It also includes settings for ingress.

```yaml
enterprise:
  enabled: true

image:
  privateRegistry:
    enabled: true
    secretToken: <lakefs-enterprise-docker-registry-token>

lakefsConfig: |
  logging:
    level: "DEBUG"
  blockstore:
    type: local
  auth:
    ui_config:
      rbac: internal

ingress:
  enabled: true
  ingressClassName: <ingress-class-name>
  annotations: {}
  hosts:
    - host: <lakefs.acme.com>
      paths:
       - /

```

--------------------------------

### lakectl command synopsis

Source: https://docs.lakefs.io/v1.70/reference/cli

Provides the basic synopsis for the `lakectl` command, indicating it accepts flags for various operations. It also mentions extensibility via plugins.

```bash
lakectl [flags]

```

--------------------------------

### Webhook Request Body Example

Source: https://docs.lakefs.io/v1.70/howto/hooks/webhooks

An example of the JSON payload sent by a webhook.

```APIDOC
```json
{
    "event_type": "pre-merge",
    "event_time": "2021-02-28T14:03:31Z",
    "action_name": "test action",
    "hook_id": "prevent_user_columns",
    "repository_id": "repo1",
    "branch_id": "feature-1",
    "source_ref": "feature-1",
    "commit_message": "merge commit message",
    "commit_id": "5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03",
    "committer": "committer",
    "commit_metadata": {
        "key": "value"
    }
}
```
```

--------------------------------

### Get Object API

Source: https://docs.lakefs.io/v1.70/reference/api/swagger

Retrieves the content of an object at a specific reference and path. Supports byte range requests and conditional GETs.

```APIDOC
## GET /repositories/{repository}/refs/{ref}/objects

### Description
Retrieves the content of an object at a specific reference and path. Supports byte range requests and conditional GETs.

### Method
GET

### Endpoint
/repositories/{repository}/refs/{ref}/objects

### Parameters
#### Path Parameters
- **repository** (string) - Required - The name of the repository.
- **ref** (string) - Required - A reference (could be either a branch or a commit ID).

#### Query Parameters
- **path** (string) - Required - The relative path of the object within the repository.
- **presign** (boolean) - Optional - If true, returns a pre-signed URL for the object.

#### Request Headers
- **Range** (string) - Optional - Byte range to retrieve (e.g., `bytes=0-1023`).
- **If-None-Match** (string) - Optional - Returns response only if the object does not have a matching ETag (e.g., `33a64df551425fcc55e4d42a148795d9f25f89d4`).

### Response
#### Success Response (200)
- **object content** (binary) - The content of the object.
  - **Content-Length** (integer) - The size of the object.
  - **Last-Modified** (string) - The last modified timestamp of the object.
  - **ETag** (string) - The ETag of the object.

#### Partial Content Response (206)
- **partial object content** (binary) - The partial content of the object.
  - **Content-Length** (integer) - The size of the partial content.
  - **Content-Range** (string) - The range of bytes returned.
  - **Last-Modified** (string) - The last modified timestamp of the object.
  - **ETag** (string) - The ETag of the object.

#### Redirect Response (302)
- **Location** (string) - The pre-signed URL for the object.

#### Not Modified Response (304)
- No content is returned.

#### Response Example (200)
```json
{
  "example": "[Binary object content]"
}
```

#### Response Example (206)
```json
{
  "example": "[Partial binary object content]"
}
```

#### Response Example (302)
```json
{
  "example": "{\"Location\": \"https://example.com/presigned-url\"}"
}
```
```
Tokens:
Raw