### Build and Serve Documentation

Source: https://github.com/lance-format/lance/blob/main/docs/CONTRIBUTING.md

Builds the documentation and starts a local development server for previewing changes. Ensure dependencies are installed first.

```bash
uv run mkdocs serve
```

--------------------------------

### Setup Python Environment and Clone Repository

Source: https://github.com/lance-format/lance/blob/main/benchmarks/bigann/README.md

Sets up a virtual environment, activates it, clones the big-ann-benchmarks repository, and installs necessary Python dependencies.

```bash
python -m venv venv
. ./venv/bin/activate
git clone https://github.com/harsha-simhadri/big-ann-benchmarks.git
cd big-ann-benchmarks
pip install -r requirements_py3.10.txt
```

--------------------------------

### Install Dependencies and Setup Hooks

Source: https://github.com/lance-format/lance/blob/main/python/AGENTS.md

Run this command once in a new worktree or after a fresh checkout to install all Python dependencies and set up pre-commit hooks. Use optional groups like `--group benchmarks` or `--extra torch` when needed.

```bash
make install
```

--------------------------------

### Install Python Bindings

Source: https://github.com/lance-format/lance/blob/main/python/CONTRIBUTING.md

Navigate to the python directory and run 'make install' to install the necessary requirements for the Python bindings.

```bash
cd python
make install
```

--------------------------------

### Setup and Run Integration Tests

Source: https://github.com/lance-format/lance/blob/main/AGENTS.md

This snippet demonstrates setting up the environment for integration testing by starting Docker Compose and then running pytest for S3 and DynamoDB integration tests.

```bash
cd test_data && docker compose up -d
AWS_DEFAULT_REGION=us-east-1 pytest --run-integration python/tests/test_s3_ddb.py
```

--------------------------------

### Example Git Commit with Pre-commit Hooks

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Demonstrates the output when a git commit is attempted after pre-commit hooks are installed. Linters and formatters will run automatically.

```shell
$ git commit -m"Changed some python files"
black....................................................................Passed
isort (python)...........................................................Passed
ruff.....................................................................Passed
[main daf91ed] Changed some python files
 1 file changed, 1 insertion(+), 1 deletion(-)
```

--------------------------------

### Install Pre-commit Tool

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Install the pre-commit tool, which is used to run formatters and linters automatically on commit.

```shell
uv tool install pre-commit
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/lance-format/lance/blob/main/docs/CONTRIBUTING.md

Installs the necessary development requirements for building the main documentation website. Run this command from the 'docs' directory.

```bash
cd docs
uv sync --dev
```

--------------------------------

### Install uv

Source: https://github.com/lance-format/lance/blob/main/docs/README.md

Installs uv using a script. Ensure you have curl installed.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Prepare Python Virtual Environment and Install Dependencies

Source: https://github.com/lance-format/lance/blob/main/benchmarks/hd-vila/README.md

Use these commands to create a Python virtual environment, activate it, and install project dependencies from requirements.txt.

```bash
python3 -m venv venv
. ./venv/bin/activate
pip install -r requirements.txt
```

--------------------------------

### Install Build Dependencies on macOS

Source: https://github.com/lance-format/lance/wiki/How-to-Build

Installs the protobuf compiler using Homebrew on macOS.

```bash
brew install protobuf
```

--------------------------------

### Start S3 Integration Tests

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Run this command to start the local minio and dynamodb services required for integration tests.

```shell
docker compose up
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/lance-format/lance/blob/main/benchmarks/tpch/README.md

Installs the necessary Python libraries for running the benchmark, including duckdb, pyarrow, and pylance.

```bash
python3 -m pip install duckdb pyarrow pylance
```

--------------------------------

### Install Lance Python SDK

Source: https://github.com/lance-format/lance/blob/main/docs/src/quickstart/full-text-search.md

Install the necessary Python libraries for using Lance, including pylance and pyarrow.

```bash
pip install pylance pyarrow
```

--------------------------------

### Install Dependencies

Source: https://github.com/lance-format/lance/blob/main/notebooks/youtube_transcript_search.ipynb

Installs necessary Python packages including openai, tqdm, ratelimiter, retry, datasets, and pylance. Use this command in your terminal or notebook environment.

```python
pip install --quiet openai tqdm ratelimiter retry datasets pylance
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Install the git pre-commit hooks. This will enable automatic linting and formatting checks before each commit.

```shell
pre-commit install
```

--------------------------------

### Install Lance

Source: https://github.com/lance-format/lance/blob/main/README.md

Install the Lance library using pip. Use the --pre flag for preview releases.

```shell
pip install pylance
```

```shell
pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance
```

--------------------------------

### Install Linux Performance Tools (perf)

Source: https://github.com/lance-format/lance/wiki/Debug

Install the necessary Linux performance event tools on Ubuntu. This includes setting the perf_event_paranoid level to allow profiling.

```sh
sudo apt install linux-tools-common linux-tools-generic linux-tools-`uname -r`
sudo sh -c "echo -1 >  /proc/sys/kernel/perf_event_paranoid"
```

--------------------------------

### SBBF Serialization Example (64 bytes)

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/index/scalar/bloom_filter.md

Provides a concrete example of byte offsets for a 64-byte SBBF with 2 blocks.

```plaintext
Offset  0-3:   Block 0, Word 0 (32-bit LE)
Offset  4-7:   Block 0, Word 1 (32-bit LE)
...
Offset 28-31:  Block 0, Word 7 (32-bit LE)
Offset 32-35:  Block 1, Word 0 (32-bit LE)
...
Offset 60-63:  Block 1, Word 7 (32-bit LE)
```

--------------------------------

### Example PQ Quantized Vector Storage

Source: https://github.com/lance-format/lance/blob/main/docs/src/guide/performance.md

Example calculation for PQ storage with 100M rows and 96 sub-vectors. Shows the significant storage requirement for PQ.

```text
100M * (96 + 8) = ~9.7 GiB
```

--------------------------------

### Build Documentation for Production

Source: https://github.com/lance-format/lance/blob/main/docs/README.md

Builds the documentation for production, creating a 'site/' directory with the static files.

```bash
make build
```

--------------------------------

### Jieba Tokenizer Configuration Example

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/index/scalar/fts.md

This JSON structure defines the configuration for the Jieba tokenizer, specifying paths to main and user dictionaries. Ensure these paths are correct for your model setup.

```json
{
  "main": "path/to/main/dictionary",
  "users": ["path/to/user/dict1", "path/to/user/dict2"]
}
```

--------------------------------

### Example Field ID Assignment

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/table/schema.md

Illustrates depth-first field ID assignment for a nested structure. Field IDs are assigned sequentially starting from 0, with parent IDs linking nested fields.

```text
Field order: a, b, c.x, c.y, c.z, d

Assigned IDs with parent relationships:
- a: 0 (parent_id: -1)
- b: 1 (parent_id: -1)
- c: 2 (parent_id: -1, struct type)
- c.x: 3 (parent_id: 2)
- c.y: 4 (parent_id: 2)
- c.z: 5 (parent_id: 2)
- d: 6 (parent_id: -1)
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/lance-format/lance/blob/main/docs/README.md

Builds and serves the documentation locally using the make command. The documentation will be accessible at http://localhost:8000.

```bash
make serve
```

--------------------------------

### Install Rust Toolchains for Cross-Compilation

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Install the required Rust toolchains for building manylinux wheels on Linux. Ensure zig is installed prior to running these commands.

```shell
rustup target add x86_64-unknown-linux-gnu
rustup target add aarch64-unknown-linux-gnu
```

--------------------------------

### Hot/Cold Tiering Example

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/table/layout.md

Demonstrates how to use multiple base paths to store recent data on a hot tier and historical data on a cold tier, enabling seamless querying without data movement.

```text
Manifest base_paths:
[
  { id: 0, is_dataset_root: true, path: "s3://hot-bucket/dataset" },
  { id: 1, is_dataset_root: true, path: "s3://cold-bucket/dataset-archive" }
]

Fragment 0 (recent data):
  DataFile { path: "fragment-0.lance", base_id: 0 }
  → resolves to: s3://hot-bucket/dataset/data/fragment-0.lance

Fragment 100 (historical data):
  DataFile { path: "fragment-100.lance", base_id: 1 }
  → resolves to: s3://cold-bucket/dataset-archive/data/fragment-100.lance
```

--------------------------------

### CLIP Model and Optimizer Setup

Source: https://github.com/lance-format/lance/blob/main/docs/src/examples/python/clip_training.md

Initializes the CLIP model components, including image and text encoders and projection heads, along with a tokenizer. It then defines the optimizer with specific learning rates for each component.

```python
# Define image encoder, image head, text encoder, text head and a tokenizer for tokenizing the caption
img_encoder = ImageEncoder(model_name=Config.img_encoder_model).to('cuda')
img_head = Head(Config.img_embed_dim, Config.projection_dim).to('cuda')

tokenizer = AutoTokenizer.from_pretrained(Config.text_encoder_model)
text_encoder = TextEncoder(model_name=Config.text_encoder_model).to('cuda')
text_head = Head(Config.text_embed_dim, Config.projection_dim).to('cuda')

# Since we are optimizing two different models together, we will define parameters manually
parameters = [
    {"params": img_encoder.parameters(), "lr": Config.img_enc_lr},
    {"params": text_encoder.parameters(), "lr": Config.text_enc_lr},
    {
        "params": itertools.chain(
            img_head.parameters(),
            text_head.parameters(),
        ),
        "lr": Config.head_lr,
    },
]

optimizer = torch.optim.Adam(parameters)
```

--------------------------------

### Verify Pylance Installation

Source: https://github.com/lance-format/lance/blob/main/skills/lance-user-guide/SKILL.md

Verify the Pylance installation by checking the version.

```python
python -c "import lance; print(lance.__version__)"
```

--------------------------------

### Install Lance using Cargo

Source: https://github.com/lance-format/lance/blob/main/rust/lance/README.md

Install the Lance CLI tool using cargo.

```shell
cargo install lance
```

--------------------------------

### Install Lance Python SDK

Source: https://github.com/lance-format/lance/blob/main/docs/src/quickstart/versioning.md

Install the Python SDK for Lance using pip.

```bash
pip install pylance
```

--------------------------------

### Configurable Segment Partitioning with num_segments

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/index/scalar/fmindex.md

Demonstrates how to configure the FM-Index partitioning using the `num_segments` parameter at index creation time to balance build cost and search performance.

```text
- **`num_segments` parameter**: Configured at index-creation time. If `num_segments` is specified (e.g. `num_segments = 4`), Lance splits the target dataset fragments into disjoint subsets and builds independent FM-Index segments over each chunk.
```

--------------------------------

### Setup LLM Training Environment with Transformers and Lance

Source: https://github.com/lance-format/lance/blob/main/docs/src/examples/python/llm_training.md

Initializes the environment for LLM training by importing necessary libraries, loading a pre-trained tokenizer and model (GPT-2), and defining key hyperparameters for training. Ensure the 'wikitext_500K.lance' dataset is available.

```python
import numpy as np
import lance

import torch
from torch.utils.data import Dataset, DataLoader, Sampler

from transformers import AutoTokenizer, AutoModelForCausalLM
from tqdm.auto import tqdm

# We'll be training the pre-trained GPT2 model in this example
model_name = 'gpt2'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Also define some hyperparameters
lr = 3e-4
nb_epochs = 10
block_size = 1024
batch_size = 8
device = 'cuda:0'
dataset_path = 'wikitext_500K.lance'
```

--------------------------------

### Install Build Dependencies on Ubuntu

Source: https://github.com/lance-format/lance/wiki/How-to-Build

Installs necessary packages for building Lance on Ubuntu 22.04.

```bash
sudo apt install protobuf-compiler libssl-dev build-essential pkg-config gfortran
```

--------------------------------

### Build Release

Source: https://github.com/lance-format/lance/blob/main/memtest/README.md

Install the package using make. This command builds the release version of the package.

```shell
make build-release
```

--------------------------------

### Shallow Clone Example

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/table/layout.md

Demonstrates how a shallow clone creates a new dataset referencing data files from a source dataset without copying, using a named base path for the source.

```text
Source dataset: s3://production/main-dataset
Clone dataset:  s3://experiments/test-variant

Clone manifest base_paths:
[
  { id: 0, is_dataset_root: true, path: "s3://experiments/test-variant" },
  { id: 1, is_dataset_root: true, path: "s3://production/main-dataset",
    name: "v1.0" }
]

Original fragments (inherited):
  DataFile { path: "fragment-0.lance", base_id: 1 }
  → resolves to: s3://production/main-dataset/data/fragment-0.lance

New fragments (clone-specific):
  DataFile { path: "fragment-new.lance", base_id: 0 }
  → resolves to: s3://experiments/test-variant/data/fragment-new.lance
```