lakeFS

https://docs.lakefs.io
Admin
lakeFS provides version control over data lakes with Git-like semantics for branching, committing,...
Tokens:220,735
Snippets:2,615
Trust Score:10
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install and launch lakeFS

Source: https://docs.lakefs.io/quickstart/launch

Use these commands to install the lakeFS package and start the quickstart environment.

```bash
pip install lakefs
python -m lakefs.quickstart
```

--------------------------------

### Two-Region Setup Configuration

Source: https://docs.lakefs.io/admin/mirroring

Example configuration for source and destination regions in a multi-region setup.

```yaml
replication:
  enabled: true
  config:
    region: "us-east-1"
    organization_id: "my-org"
    regional_endpoint: "http://lakefs-source.mirroring.svc.cluster.local:80"
    dst_endpoints:
      us-west-2: "http://lakefs-dest-internal.example.com:80"
    mirrors_database:
      type: dynamodb
      dynamodb:
        table_name: my-replication-table
        aws_region: us-east-1
    blockstore:
      type: s3
      s3:
        region: us-east-1
    auth:
      encrypt:
        secret_key: "<shared-secret>"
    logging:
      level: INFO
```

```yaml
replication:
  enabled: true
  config:
    region: "us-west-2"
    organization_id: "my-org"
    regional_endpoint: "http://lakefs-dest.mirroring.svc.cluster.local:80"
    dst_endpoints:
      us-east-1: "http://lakefs-source-internal.example.com:80"
    mirrors_database:
      type: dynamodb
      dynamodb:
        table_name: my-replication-table
        aws_region: us-west-2
    blockstore:
      type: s3
      s3:
        region: us-west-2
    auth:
      encrypt:
        secret_key: "<shared-secret>"
    logging:
      level: INFO
```

--------------------------------

### Install Dependencies

Source: https://docs.lakefs.io/admin/backup-and-restore

Install the necessary Python dependencies required to run the helper script.

```bash
pip install -r requirements.txt
```

--------------------------------

### View lakeFS quickstart output

Source: https://docs.lakefs.io/quickstart/launch

The output displayed after running the quickstart command, providing the login URL and credentials.

```text
│
│ lakeFS running in quickstart mode.
│     Login at http://127.0.0.1:8000/
│
│     Access Key ID    : AKIAIOSFOLQUICKSTART
│     Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
│
```

--------------------------------

### Install and load aws.s3

Source: https://docs.lakefs.io/reference/r

Install and load the aws.s3 library to interface with lakeFS.

```R
install.packages(c("aws.s3"))
library(aws.s3)
```

--------------------------------

### Start mount server

Source: https://docs.lakefs.io/mount

Starts the mount server process without performing an OS-level mount.

```bash
everest mount-server <remote_mount_uri> [flags]
```

--------------------------------

### Minimalistic configuration file

Source: https://docs.lakefs.io/admin/garbage-collection/standalone-gc

A basic YAML configuration example.

```yaml
logging:
  level: debug
```

--------------------------------

### Execute a GET request

Source: https://docs.lakefs.io/guides/hooks/lua

Example of performing a basic GET request and handling the response status code.

```lua
local http = require("net/http")
local code, body = http.request("<https://example.com>")
if code == 200 then
    print(body)
else
    print("Failed to get example.com - status code: " .. code)
end
```

--------------------------------

### Install Boto3

Source: https://docs.lakefs.io/reference/python/boto

Commands to install or upgrade the Boto3 library via pip.

```bash
pip install boto3
```

```bash
pip install --upgrade boto3
```

--------------------------------

### Install and load Arrow

Source: https://docs.lakefs.io/reference/r

Install and load the Arrow library for advanced data analysis.

```R
install.packages("arrow")
library(arrow)
```

--------------------------------

### Install and Load HTTPFS Extension

Source: https://docs.lakefs.io/integrations/duckdb

Ensure the HTTPFS extension is installed and loaded to enable S3-compatible connectivity.

```sql
INSTALL httpfs;
LOAD httpfs;
```

--------------------------------

### Get help for repository commands

Source: https://docs.lakefs.io/reference/cli/oss

Displays help information for repository-related commands.

```bash
lakectl repo help [command] [flags]
```

--------------------------------

### Configure cataloger for fresh start

Source: https://docs.lakefs.io/admin/upgrade

Configuration snippet to discard existing data and start fresh, applicable only for version 0.30.0.

```yaml
cataloger:
  type: rocks
```

--------------------------------

### Verify migration output

Source: https://docs.lakefs.io/admin/upgrade

Example of a successful migration log output.

```text
time="2022-08-10T14:46:25Z" level=info msg="KV Migration took 717.629563ms" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_2913402680
```

--------------------------------

### Get help for show commands

Source: https://docs.lakefs.io/reference/cli/oss

Displays help information for show-related commands.

```bash
lakectl show help [command] [flags]
```

--------------------------------

### Verify branch contents

Source: https://docs.lakefs.io/quickstart/work-with-data-locally

Example output showing the contents of a branch after operations.

```text
object          2023-03-21 17:33:51 +0000 UTC    20.9 kB         denmark-lakes.parquet
object          2023-03-21 14:45:38 +0000 UTC    916.4 kB        lakes.parquet
```

--------------------------------

### Initialize AWSIAMAuthProvider

Source: https://docs.lakefs.io/security/external-principals-aws

Install the OkHttp interceptor on the ApiClient to handle automatic token exchange and refreshing.

```java
import io.lakefs.clients.sdk.ApiClient;
import io.lakefs.clients.sdk.RepositoriesApi;
import io.lakefs.clients.sdk.auth.iam.AWSIAMAuthProvider;
import io.lakefs.clients.sdk.auth.iam.IAMAuthParams;

ApiClient apiClient = new ApiClient();
apiClient.setBasePath("https://lakefs.example.com/api/v1");

IAMAuthParams params = IAMAuthParams.builder("lakefs.example.com").build();
AWSIAMAuthProvider.install(apiClient, params);

RepositoriesApi repos = new RepositoriesApi(apiClient);
```

--------------------------------

### Get help for local commands

Source: https://docs.lakefs.io/reference/cli/oss

Displays help information for local synchronization commands.

```bash
lakectl local help [command] [flags]
```

```bash
-h, --help   help for help
```

--------------------------------

### Get help for users

Source: https://docs.lakefs.io/reference/cli/oss

Provides detailed help for specific user commands.

```bash
lakectl auth users help [command] [flags]
```

```bash
  -h, --help   help for help
```

--------------------------------

### Get help for policies commands

Source: https://docs.lakefs.io/reference/cli/oss

Displays help information for policies-related commands.

```bash
lakectl auth policies help [command] [flags]
```

--------------------------------

### Initialize lakeFS setup

Source: https://docs.lakefs.io/admin/install/migrate-from-community

Configure the new lakeFS instance with administrative credentials after the migration is complete.

```bash
lakefs setup --user-name <admin> --access-key-id <key> --secret-access-key <secret> --no-check
```

--------------------------------

### Get Parquet Schema Structure

Source: https://docs.lakefs.io/guides/hooks/lua

Example of the table structure returned by the parquet schema reader.

```lua
{
  { ["name"] = "column_a", ["type"] = "INT32" },
  { ["name"] = "column_b", ["type"] = "BYTE_ARRAY" }
}
```

--------------------------------

### Repository help and listing

Source: https://docs.lakefs.io/reference/cli/enterprise

Commands for accessing help documentation and listing existing repositories.

```bash
lakectl repo help [command] [flags]
```

```bash
lakectl repo list [flags]
```

--------------------------------

### Get help for credentials commands

Source: https://docs.lakefs.io/reference/cli/oss

Displays help information for credentials-related commands.

```bash
lakectl auth users credentials help [command] [flags]
```

--------------------------------

### GET /api/v1/license

Source: https://docs.lakefs.io/admin/install/enterprise

Retrieves the currently installed lakeFS Enterprise license token. The returned token is a JWT that can be decoded to view license details such as organization ID, installation ID, and enabled features.

```APIDOC
## GET /api/v1/license

### Description
Retrieves the currently installed lakeFS Enterprise license token.

### Method
GET

### Endpoint
https://your-lakefs-server/api/v1/license

### Response
#### Success Response (200)
- **token** (string) - The JWT license token string.

#### Response Example
{
  "token": "eyJhbGciOiJSUzI1NiIs..."
}
```

--------------------------------

### Example of fetching results from response objects (old vs new)

Source: https://docs.lakefs.io/project/code-migrate-1.0-sdk

Illustrates the change in how to access properties from response objects when migrating from lakefs-client to lakefs-sdk.

```python
response = lakefs_client.branches.diff_branch(repository='repo', branch='main')
diff = response.results[0] # 'results' is a 'DiffList' object
path = diff.get_property('path') # 'diff' is a dictionary
```

```python
response = lakefs_client.branches_api.diff_branch(repository='repo', branch='main')
diff = response.results[0] # 'results' is a 'DiffList' object
path = diff.path # 'diff' is a 'Diff' object
```

--------------------------------

### New Java SDK Style (Fluent)

Source: https://docs.lakefs.io/project/code-migrate-1.0-sdk

Example of the new fluent style for making API calls in the Java SDK, which is more modular and adaptive to changes, allowing optional parameters to be chained.

```java
ObjectStats objectStat = objectsApi
    .statObject(
        objectLoc.getRepository(), objectLoc.getRef(), objectLoc.getPath()
    )
    .userMetadata(true)
    .execute();

```

--------------------------------

### Example Action File Configuration

Source: https://docs.lakefs.io/guides/hooks

A sample YAML configuration defining multiple webhook hooks with conditional execution logic.

```yaml
name: Good files check
description: set of checks to verify that branch is good
on:
  pre-commit:
  pre-merge:
    branches:
      - main
hooks:
  - id: no_temp
    type: webhook
    description: checking no temporary files found
    properties:
      url: "https://example.com/webhook?notmp=true?t=1za2PbkZK1bd4prMuTDr6BeEQwWYcX2R"
  - id: no_freeze
    type: webhook
    description: check production is not in dev freeze
    properties:
      url: "https://example.com/webhook?nofreeze=true?t=1za2PbkZK1bd4prMuTDr6BeEQwWYcX2R"
  - id: alert
    type: webhook
    if: failure()
    description: notify alert system when check failed
    properties:
      url: "https://example.com/alert"
      query_params:
        title: good files webhook failed
  - id: notification
    type: webhook
    if: true
    description: notify that will always run - no matter if one of the previous steps failed
    properties:
      url: "https://example.com/notification"
      query_params:
        title: good files completed
```

--------------------------------

### Old Java SDK Style

Source: https://docs.lakefs.io/project/code-migrate-1.0-sdk

Example of the previous style for making API calls in the Java SDK, which required passing all parameters, including optional ones, in a single function call.

```java
ObjectStats objectStat = objectsApi.statObject(
    objectLoc.getRepository(), objectLoc.getRef(), objectLoc.getPath(),
    false, false);

```

--------------------------------

### Configure Authentication Methods

Source: https://docs.lakefs.io/reference/python/generated-sdk

Examples of configuring the SDK using either HTTP Basic Authentication or JWT Bearer tokens.

```python
import lakefs_sdk
import os

# HTTP Basic Authentication
configuration = lakefs_sdk.Configuration(
    host="https://example.lakefs.io/api/v1",
    username=os.environ["LAKEFS_ACCESS_KEY"],
    password=os.environ["LAKEFS_SECRET_KEY"]
)

# JWT Bearer Token
configuration = lakefs_sdk.Configuration(
    host="https://example.lakefs.io/api/v1",
    access_token=os.environ["LAKEFS_TOKEN"]
)

api_client = lakefs_sdk.ApiClient(configuration)
```

--------------------------------

### Initialize LakeFSFileSystem

Source: https://docs.lakefs.io/reference/python/lakefs-spec

Instantiate the filesystem client using either auto-discovered credentials from ~/.lakectl.yaml or explicit connection parameters.

```python
from lakefs_spec import LakeFSFileSystem

# Auto-discover credentials from ~/.lakectl.yaml
fs = LakeFSFileSystem()

# Or provide explicit credentials
fs = LakeFSFileSystem(
    host="http://localhost:8000",
    username="your-access-key",
    password="your-secret-key"
)
```

--------------------------------

### Initialize AWS Glue Client

Source: https://docs.lakefs.io/guides/hooks/lua

Initialize the Glue client using AWS credentials.

```lua
local aws = require("aws")
-- pass valid AWS credentials
local glue = aws.glue_client("ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "REGION")
```

--------------------------------

### Install lakefs-spec

Source: https://docs.lakefs.io/reference/python/lakefs-spec

Commands to install or upgrade the lakefs-spec package via pip.

```bash
pip install lakefs-spec
```

```bash
pip install --upgrade lakefs-spec
```

--------------------------------

### Clone and Initialize lakeFS Samples

Source: https://docs.lakefs.io/use-cases/isolated-environments

Commands to clone the repository and initialize submodules for the local environment.

```bash
git clone https://github.com/treeverse/lakeFS-samples.git
cd lakeFS-samples
```

```bash
git submodule init && git submodule update
docker compose up
```

--------------------------------

### Create a bare repository

Source: https://docs.lakefs.io/reference/cli/enterprise

Initializes a repository without a default branch or initial commit. Use with caution as this is a plumbing command.

```bash
lakectl repo create-bare <repository URI> <storage namespace> [flags]
```

```bash
lakectl repo create-bare lakefs://my-repo s3://my-bucket
```

--------------------------------

### Delta table descriptor example

Source: https://docs.lakefs.io/guides/hooks/lua

Example YAML configuration for a Delta table descriptor.

```yaml
---
name: my_table_name
type: delta
path: path/to/delta/table/data
catalog: my-catalog
```

--------------------------------

### Create a Repository

Source: https://docs.lakefs.io/reference/r

Create a new repository by sending a POST request with the repository name and storage namespace.

```R
library(httr)
lakefs_api_url <- "lakefs.mycorp.com:8000/api/v1"
lakefsAccessKey <- "AKIAIOSFODNN7EXAMPLE"
lakefsSecretKey <- "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
repo_name <- "my_new_repo"

# Define the payload
body=list(name=repo_name, 
          storage_namespace="s3://example-bucket/foo")

# Call the API
r=POST(url=paste0(lakefs_api_url, "/repositories"), 
        authenticate(lakefsAccessKey, lakefsSecretKey),
        body=body, encode="json")
```

--------------------------------

### Install lakeFS SDK

Source: https://docs.lakefs.io/reference/python/getting-started

Commands to install or upgrade the lakeFS Python SDK using pip.

```bash
pip install lakefs
```

```bash
pip install --upgrade lakefs
```

--------------------------------

### Create a user

Source: https://docs.lakefs.io/reference/cli/oss

Creates a new user.

```bash
lakectl auth users create [flags]
```

--------------------------------

### Install lakefs-spec

Source: https://docs.lakefs.io/integrations/huggingface_datasets

Install the lakefs-spec library using pip.

```bash
pip install lakefs-spec

```

--------------------------------

### Create credentials options

Source: https://docs.lakefs.io/reference/cli/oss

Flags for creating credentials.

```bash
  -h, --help        help for create
      --id string   Username (email for password-based users, default: current user)
```

--------------------------------

### Initialize Azure Blob Client

Source: https://docs.lakefs.io/guides/hooks/lua

Initialize the Azure blob client using storage account credentials.

```lua
local azure = require("azure")
-- pass valid Azure credentials
local client = azure.blob_client("AZURE_STORAGE_ACCOUNT", "AZURE_ACCESS_KEY")
```

--------------------------------

### Troubleshoot Installation ID Mismatch

Source: https://docs.lakefs.io/admin/install/enterprise

Error message when the license file is associated with a different installation ID.

```text
Error: license belongs to installation ID X (current installation ID: Y)
```

--------------------------------

### Audit Log Path Examples

Source: https://docs.lakefs.io/admin/auditing

Example S3 paths for audit logs organized by region and organization.

```text
# ACME in us-east-1 
etl/v1/data/region=us-east-1/organization=org-acme/year=2024/month=02/day=12/hour=13/log_abc-snappy.parquet

# ACME in us-west-2 
etl/v1/data/region=us-west-2/organization=org-acme/year=2024/month=02/day=12/hour=13/log_xyz-snappy.parquet
```

--------------------------------

### Configure lakeFS with MinIO

Source: https://docs.lakefs.io/quickstart/learning-more-lakefs

Set environment variables for S3-compatible storage like MinIO and start the lakeFS server.

```bash
export LAKEFS_BLOCKSTORE_TYPE="s3"
export LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE="true"
export LAKEFS_BLOCKSTORE_S3_ENDPOINT="http://<minio_endpoint>"
export LAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION="false"
export LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID="<minio_access_key>"
export LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY="<minio_secret_key>"
lakefs run --local-settings
```

--------------------------------

### Configure Local Blockstore

Source: https://docs.lakefs.io/admin/install/onprem

Example configuration for using a local POSIX-compatible storage location, including database settings, encryption keys, and import path permissions.

```yaml
database:
  type: "postgres"
  postgres:
    connection_string: "[DATABASE_CONNECTION_STRING]"

auth:
  encrypt:
    # replace this with a randomly-generated string. Make sure to keep it safe!
    secret_key: "[ENCRYPTION_SECRET_KEY]"

blockstore:
  type: local
  local:
    path: /shared/location/lakefs_data    # location where data and metadata kept by lakeFS
    import_enabled: true                  # required to be true to enable import files
                                          # from `allowed_external_prefixes` locations
    allowed_external_prefixes:
      - /shared/location/files_to_import  # location with files we can import into lakeFS, require access from lakeFS
```

--------------------------------

### Example S3 Request URL

Source: https://docs.lakefs.io/admin/virtual-host-addressing

An example of an S3 request URL that can be interpreted as either virtual-host or path-based.

```text
GET http://foo.example.com/some/location
```

--------------------------------

### Create Service Accounts

Source: https://docs.lakefs.io/admin/install/gcp

Provision the metadata and data service accounts with specific descriptions.

```bash
# Metadata service account (accesses _lakefs/** from anywhere)
gcloud iam service-accounts create "${SA_OPEN_ID}" \
  --project="${PROJECT_ID}" \
  --description="lakeFS metadata - RW only under /_lakefs/** (from anywhere)" \
  --display-name="lakeFS Metadata Service Account"

# Data service account (accesses everything except _lakefs/**, network-restricted)
gcloud iam service-accounts create "${SA_RESTRICTED_ID}" \
  --project="${PROJECT_ID}" \
  --description="lakeFS data - RW everywhere except /_lakefs/** (network-restricted)" \
  --display-name="lakeFS Data Service Account"
```

--------------------------------

### Install lakectl via Homebrew

Source: https://docs.lakefs.io/reference/cli/enterprise

Use Homebrew to install the lakeFS enterprise CLI tool on Linux or macOS.

```bash
brew tap treeverse/lakefs
brew install lakefs-enterprise
```

--------------------------------

### Create user options

Source: https://docs.lakefs.io/reference/cli/oss

Flags for creating a user.

```bash
  -h, --help        help for create
      --id string   Username
```

--------------------------------

### Install lakeFS Helm Chart

Source: https://docs.lakefs.io/admin/install/enterprise

Command to install the lakeFS Helm chart using a custom values.yaml file.

```bash
helm install lakefs lakefs/lakefs -f values.yaml
```

--------------------------------

### Initialize AWS S3 Client

Source: https://docs.lakefs.io/guides/hooks/lua

Initialize the S3 client using AWS credentials.

```lua
local aws = require("aws")
-- pass valid AWS credentials
local client = aws.s3_client("ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "REGION")
```

--------------------------------

### Auth help options

Source: https://docs.lakefs.io/reference/cli/oss

Standard help flag for auth commands.

```bash
  -h, --help   help for help
```

--------------------------------

### Remote Authenticator HTTP Request Example

Source: https://docs.lakefs.io/security/remote-authenticator

An example of the full HTTP POST request sent by lakeFS to the remote authenticator.

```http
POST https://remote-authenticator.example.com/auth
Content-Type: application/json

{
  "username": "testy.mctestface@example.com",
  "password": "Password1"
}
```

--------------------------------

### ACL Server Configuration Example

Source: https://docs.lakefs.io/security/access-control-lists

Example of an .aclserver.yaml configuration file for the ACL server.

```yaml
---
listen_address: "[ACL_SERVER_LISTEN_ADDRESS]"
database:
    type: "postgres"
    postgres:
    connection_string: "[DATABASE_CONNECTION_STRING]"

encrypt:
    # This should be the same encryption key as in lakeFS
    secret_key: "[ENCRYPTION_SECRET_KEY]"


```

--------------------------------

### Create a mirror via replication API

Source: https://docs.lakefs.io/admin/mirroring

Use this POST request to initiate a mirror on the destination repository. Ensure the replication service is running on both source and destination before execution.

```bash
curl '<REPLICATION_ENDPOINT>/service/replication/v1/repositories/<SOURCE_REPO>/mirrors' \
--header 'Content-Type: application/json' \
-u <ACCESS_KEY_ID>:<SECRET_ACCESS_KEY> \
-X POST \
--data '{
    "name": "<MIRROR_NAME>",
    "region": "<DESTINATION_REGION>",
    "storage_namespace": "<MIRROR_STORAGE_NAMESPACE>"
}'
```

--------------------------------

### Create Bare Repository via CLI

Source: https://docs.lakefs.io/admin/backup-and-restore

Create a bare repository using the lakectl CLI with S3 or Azure storage backends.

```bash
lakectl repo create-bare lakefs://target-example-repo s3://target-bucket-name/example-repo --default-branch "same-default-branch-as-in-source-repo"
```

```bash
lakectl repo create-bare lakefs://target-example-repo https://target-storage-account-name.blob.core.windows.net/targetContainer/example-repo --default-branch "same-default-branch-as-in-source-repo"
```

--------------------------------

### Install the lakeFS Airflow package

Source: https://docs.lakefs.io/integrations/airflow

Command to install the lakeFS Airflow provider package using pip.

```bash
pip install airflow-provider-lakefs

```

--------------------------------

### Create Glue Database

Source: https://docs.lakefs.io/guides/hooks/lua

Create a new database in the Glue Catalog with specified options.

```lua
local opts = {
    error_on_already_exists = false,
    create_db_input = {DatabaseInput = {Description = "Created via LakeFS Action"}, Tags = {Owner = "Joe"}}
}
glue.create_database(db, opts)
```

--------------------------------

### Handling Mount Initialization Delays

Source: https://docs.lakefs.io/reference/matlab

Use a pause command to ensure the mount is fully initialized before attempting to access files.

```matlab
everest.mount('lakefs://repo/branch/', 'data/');
pause(2);  % Give mount time to initialize
dir('data/')  % Should now show files
```

--------------------------------

### Install lakeFS Python SDK with AWS IAM support

Source: https://docs.lakefs.io/security/external-principals-aws

Install the necessary dependencies to enable AWS IAM authentication in the lakeFS Python SDK.

```bash
pip install "lakefs[aws-iam]"
```

--------------------------------

### Example pipeline with lakeFS steps

Source: https://docs.lakefs.io/integrations/kubeflow

An example of a Kubeflow pipeline that includes steps for branch creation and commits using lakeFS operations.

```python
def lakectl_pipeline():
    create_branch_task = create_branch_op('example-repo', 'example-branch', 'main') # A function-based component
    extract_task = example_extract_op()
    commit_task = commit_op()
    transform_task = example_transform_op()
    commit_task = commit_op()
    load_task = example_load_op()
```

--------------------------------

### Repository create-bare options

Source: https://docs.lakefs.io/reference/cli/oss

Available flags for the create-bare command.

```bash
  -d, --default-branch string   the default branch name of this repository (will not be created) (default "main")
  -h, --help                    help for create-bare
```

--------------------------------

### Google Storage CORS configuration example

Source: https://docs.lakefs.io/security/presigned-url

Example of CORS configuration for Google Storage to allow lakeFS UI to fetch objects using presigned URLs.

```json
  [
    {
        "origin": ["lakefs.endpoint"],
        "responseHeader": ["ETag"],
        "method": ["PUT", "GET", "HEAD"],
        "maxAgeSeconds": 3600
    }
  ]


```

--------------------------------

### Run lakeFS Binary

Source: https://docs.lakefs.io/admin/install/onprem

Command to start the lakeFS server using a specific configuration file.

```bash
lakefs --config config.yaml run
```

--------------------------------

### AWS S3 CORS configuration example

Source: https://docs.lakefs.io/security/presigned-url

Example of CORS configuration for AWS S3 to allow lakeFS UI to fetch objects using presigned URLs.

```json
  [
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "PUT",
            "HEAD"
        ],
        "AllowedOrigins": [
            "lakefs.endpoint"
        ],
        "ExposeHeaders": [
            "ETag"
        ]
    }
  ]


```

--------------------------------

### GET /datasets

Source: https://docs.lakefs.io/security/rbac

Lists all datasets.

```APIDOC
## GET /datasets

### Description
Lists all datasets.

### Method
GET

### Endpoint
/datasets
```

--------------------------------

### Configure logging level via environment variable

Source: https://docs.lakefs.io/admin/garbage-collection/standalone-gc

Example of setting configuration using environment variables with the required prefix.

```bash
export LAKEFS_SGC_LOGGING_LEVEL=info
```

--------------------------------

### Azure Blob Storage CORS configuration example

Source: https://docs.lakefs.io/security/presigned-url

Example of CORS configuration for Azure Blob Storage to allow lakeFS UI to fetch objects using presigned URLs.

```xml
  <Cors>
      <CorsRule>
          <AllowedOrigins>lakefs.endpoint</AllowedOrigins>
          <AllowedMethods>PUT,GET,HEAD</AllowedMethods>
          <AllowedHeaders>*</AllowedHeaders>
          <ExposedHeaders>ETag,x-ms-*</ExposedHeaders>
          <MaxAgeInSeconds>3600</MaxAgeInSeconds>
      </CorsRule>
  </Cors>


```

--------------------------------

### everest.verify

Source: https://docs.lakefs.io/reference/matlab

Verifies lakeFS Mount installation.

```APIDOC
## everest.verify()

### Description
Check lakeFS Mount installation and accessibility.
```

--------------------------------

### Configure lakectl

Source: https://docs.lakefs.io/quickstart/branch

Initializes the lakectl tool with server credentials.

```bash
lakectl config
```

--------------------------------

### GET /auth/groups

Source: https://docs.lakefs.io/security/ACL-server-implementation

Returns a list of groups.

```APIDOC
## GET /auth/groups

### Description
Returns a list of groups, sorted by the group name.

### Method
GET

### Endpoint
/auth/groups

### Parameters
#### Query Parameters
- **prefix** (string) - Optional - Pagination prefix.
- **after** (string) - Optional - Pagination marker.
- **amount** (integer) - Optional - Number of items to return.
```

--------------------------------

### Mount and Use Repository Locally

Source: https://docs.lakefs.io/mount

Mounts a repository for read-only access and demonstrates usage with external tools.

```bash
everest mount lakefs://image-repo/main/datasets/pets/ ./pets

# Run a python script
pytorch_train.py --input ./pets

# Query data with DuckDB
duckdb "SELECT * FROM read_parquet('pets/labels.parquet')"

everest umount ./pets
```

--------------------------------

### GET /auth/users

Source: https://docs.lakefs.io/security/ACL-server-implementation

Returns a list of all users.

```APIDOC
## GET /auth/users

### Description
Returns a list of all users, sorted by username.

### Method
GET

### Endpoint
/auth/users

### Parameters
#### Query Parameters
- **prefix** (string) - Optional - Pagination prefix.
- **after** (string) - Optional - Pagination marker.
- **amount** (integer) - Optional - Number of items to return.

### Response
#### Success Response (200)
- **pagination** (object) - Pagination information.
- **results** (array) - List of User objects.
```

--------------------------------

### Create a lakeFS Repository

Source: https://docs.lakefs.io/reference/python/branches

Initializes a new repository at a specified storage location.

```python
import lakefs

repo = lakefs.repository("my-repo").create(
    storage_namespace="s3://my-bucket/data"
)
```

--------------------------------

### List repositories

Source: https://docs.lakefs.io/reference/cli/oss

Lists all repositories with optional pagination and filtering.

```bash
lakectl repo list [flags]
```

--------------------------------

### GET /auth/policies

Source: https://docs.lakefs.io/security/ACL-server-implementation

Retrieves a list of all policies, sorted by name.

```APIDOC
## GET /auth/policies

### Description
Returns a list of policies. The results are sorted by the policy name.

### Method
GET

### Endpoint
/auth/policies

### Parameters
#### Query Parameters
- **prefix** (string) - Optional - Filter policies by prefix
- **after** (string) - Optional - Pagination marker
- **amount** (integer) - Optional - Number of items to return

### Response
#### Success Response (200)
- **pagination** (object) - Pagination information
- **results** (array) - List of Policy objects
```

--------------------------------

### GET /auth/groups/{groupId}

Source: https://docs.lakefs.io/security/ACL-server-implementation

Returns the details of a specific group.

```APIDOC
## GET /auth/groups/{groupId}

### Description
Returns the details of a specific group.

### Method
GET

### Endpoint
/auth/groups/{groupId}

### Parameters
#### Path Parameters
- **groupId** (string) - Required - The ID of the group.
```

--------------------------------

### GET /auth/users/{userId}

Source: https://docs.lakefs.io/security/ACL-server-implementation

Retrieves the details of a specific user.

```APIDOC
## GET /auth/users/{userId}

### Description
Returns the details of a specific user.

### Method
GET

### Endpoint
/auth/users/{userId}

### Parameters
#### Path Parameters
- **userId** (string) - Required - The unique identifier of the user.

### Response
#### Success Response (200)
- **username** (string) - A unique identifier for the user.
- **creation_date** (integer) - Unix Epoch in seconds.
- **friendly_name** (string) - A name for the user that is not necessarily unique.
- **email** (string) - User email.
- **source** (string) - User source.
```

--------------------------------

### Start Spark Shell for lakeFS Community

Source: https://docs.lakefs.io/reference/spark-client

Commands to initialize the Spark shell with the necessary lakeFS Community packages.

```bash
spark-shell --packages io.lakefs:lakefs-spark-client_2.12:0.22.0
```

```bash
spark-shell --packages io.lakefs:lakefs-spark-client_2.13:0.22.0
```

--------------------------------

### Configure Spark SQL Catalog and S3A FileSystem

Source: https://docs.lakefs.io/iceberg

Set up the Spark SQL catalog and S3A FileSystem with lakeFS connection details.

```python
.config("spark.sql.catalog.lakefs", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.lakefs.catalog-impl", "io.lakefs.iceberg.LakeFSCatalog") \
.config("spark.sql.catalog.lakefs.warehouse", f"lakefs://{repo_name}") \
.config("spark.sql.catalog.lakefs.cache-enabled", "false")
```

```python
.config("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config("spark.hadoop.fs.s3a.endpoint", "https://example-org.us-east-1.lakefscloud.io") \
.config("spark.hadoop.fs.s3a.access.key", "AKIAIO5FODNN7EXAMPLE") \
.config("spark.hadoop.fs.s3a.secret.key", "wJalrXUtnFEMI/K3MDENG/bPxRfiCYEXAMPLEKEY") \
.config("spark.hadoop.fs.s3a.path.style.access", "true")
```
Tokens:
Raw