### Build and Serve Documentation

Source: https://github.com/lance-format/lance/blob/main/docs/CONTRIBUTING.md

Builds the documentation and starts a local development server for previewing changes. Ensure dependencies are installed first.

```bash
uv run mkdocs serve
```

--------------------------------

### Setup Python Environment and Clone Repository

Source: https://github.com/lance-format/lance/blob/main/benchmarks/bigann/README.md

Sets up a virtual environment, activates it, clones the big-ann-benchmarks repository, and installs necessary Python dependencies.

```bash
python -m venv venv
. ./venv/bin/activate
git clone https://github.com/harsha-simhadri/big-ann-benchmarks.git
cd big-ann-benchmarks
pip install -r requirements_py3.10.txt
```

--------------------------------

### Install Python Bindings

Source: https://github.com/lance-format/lance/blob/main/python/CONTRIBUTING.md

Navigate to the python directory and run 'make install' to install the necessary requirements for the Python bindings.

```bash
cd python
make install
```

--------------------------------

### Example Git Commit with Pre-commit Hooks

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Demonstrates the output when a git commit is attempted after pre-commit hooks are installed. Linters and formatters will run automatically.

```shell
$ git commit -m"Changed some python files"
black....................................................................Passed
isort (python)...........................................................Passed
ruff.....................................................................Passed
[main daf91ed] Changed some python files
 1 file changed, 1 insertion(+), 1 deletion(-)
```

--------------------------------

### Install preview version of pylance

Source: https://github.com/lance-format/lance/blob/main/docs/src/quickstart/index.md

Commands for installing the preview version of the SDK using pip or uv.

```bash
pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance
```

```bash
uv venv
uv pip install --prerelease allow --index https://pypi.fury.io/lance-format pylance

# To add to pyproject.toml, just do:
uv add --prerelease allow --index https://pypi.fury.io/lance-format pylance
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/lance-format/lance/blob/main/docs/CONTRIBUTING.md

Installs the necessary development requirements for building the main documentation website. Run this command from the 'docs' directory.

```bash
cd docs
uv sync --dev
```

--------------------------------

### Install Pre-commit Tool

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Install the pre-commit tool, which is used to run formatters and linters automatically on commit.

```shell
uv tool install pre-commit
```

--------------------------------

### Install uv

Source: https://github.com/lance-format/lance/blob/main/docs/README.md

Use this command to install the uv package manager on your system.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Prepare Python Virtual Environment and Install Dependencies

Source: https://github.com/lance-format/lance/blob/main/benchmarks/hd-vila/README.md

Use these commands to create a Python virtual environment, activate it, and install project dependencies from requirements.txt.

```bash
python3 -m venv venv
. ./venv/bin/activate
pip install -r requirements.txt
```

--------------------------------

### Install OpenTelemetry Support for Python

Source: https://github.com/lance-format/lance/blob/main/docs/src/guide/observability.md

Install the pylance otel extra to enable OpenTelemetry metric support.

```bash
pip install "pylance[otel]"
```

--------------------------------

### Install pylance via pip

Source: https://github.com/lance-format/lance/blob/main/docs/src/quickstart/index.md

Standard installation command for the Lance Python SDK.

```bash
pip install pylance
```

--------------------------------

### Install Build Dependencies on macOS

Source: https://github.com/lance-format/lance/wiki/How-to-Build

Installs the protobuf compiler using Homebrew on macOS.

```bash
brew install protobuf
```

--------------------------------

### Install Prometheus Recorder in Rust

Source: https://github.com/lance-format/lance/blob/main/docs/src/guide/observability.md

Install a metrics-compatible recorder at application startup before opening datasets.

```rust
metrics_exporter_prometheus::PrometheusBuilder::new()
    .install()
    .expect("install Prometheus recorder");
```

--------------------------------

### Start S3 Integration Tests

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Run this command to start the local minio and dynamodb services required for integration tests.

```shell
docker compose up
```

--------------------------------

### Install Beta Python Wheels

Source: https://github.com/lance-format/lance/blob/main/python/AGENTS.md

Use this command sequence to install specific beta or release candidate versions of pylance from the project's fury.io index.

```shell
uv venv /path/to/venv
uv pip install --python /path/to/venv/bin/python --pre \
    --extra-index-url https://pypi.fury.io/lance-format/ \
    "pylance==7.2.0b4"
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/lance-format/lance/blob/main/benchmarks/tpch/README.md

Installs the necessary Python libraries for running the benchmark, including duckdb, pyarrow, and pylance.

```bash
python3 -m pip install duckdb pyarrow pylance
```

--------------------------------

### Install Dependencies

Source: https://github.com/lance-format/lance/blob/main/notebooks/youtube_transcript_search.ipynb

Installs necessary Python packages including openai, tqdm, ratelimiter, retry, datasets, and pylance. Use this command in your terminal or notebook environment.

```python
pip install --quiet openai tqdm ratelimiter retry datasets pylance
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Install the git pre-commit hooks. This will enable automatic linting and formatting checks before each commit.

```shell
pre-commit install
```

--------------------------------

### Install Linux Performance Tools (perf)

Source: https://github.com/lance-format/lance/wiki/Debug

Install the necessary Linux performance event tools on Ubuntu. This includes setting the perf_event_paranoid level to allow profiling.

```sh
sudo apt install linux-tools-common linux-tools-generic linux-tools-`uname -r`
sudo sh -c "echo -1 >  /proc/sys/kernel/perf_event_paranoid"
```

--------------------------------

### Example Field ID Assignment

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/table/schema.md

Illustrates depth-first field ID assignment for a nested structure. Field IDs are assigned sequentially starting from 0, with parent IDs linking nested fields.

```text
Field order: a, b, c.x, c.y, c.z, d

Assigned IDs with parent relationships:
- a: 0 (parent_id: -1)
- b: 1 (parent_id: -1)
- c: 2 (parent_id: -1, struct type)
- c.x: 3 (parent_id: 2)
- c.y: 4 (parent_id: 2)
- c.z: 5 (parent_id: 2)
- d: 6 (parent_id: -1)
```

--------------------------------

### Install Rust Toolchains for Cross-Compilation

Source: https://github.com/lance-format/lance/blob/main/python/DEVELOPMENT.md

Install the required Rust toolchains for building manylinux wheels on Linux. Ensure zig is installed prior to running these commands.

```shell
rustup target add x86_64-unknown-linux-gnu
rustup target add aarch64-unknown-linux-gnu
```

--------------------------------

### Sample index creation output

Source: https://github.com/lance-format/lance/blob/main/docs/src/quickstart/vector-search.md

Example console output generated during the index building process.

```text
Building vector index: IVF256,PQ16
CPU times: user 2min 23s, sys: 2.77 s, total: 2min 26s
Wall time: 22.7 s
Sample 65536 out of 1000000 to train kmeans of 128 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
```

--------------------------------

### Verify Pylance Installation

Source: https://github.com/lance-format/lance/blob/main/skills/lance-user-guide/SKILL.md

Verify the Pylance installation by checking the version.

```python
python -c "import lance; print(lance.__version__)"
```

--------------------------------

### Install lance-tensorflow

Source: https://github.com/lance-format/lance/blob/main/docs/src/integrations/tensorflow.md

Install the required package to enable TensorFlow integration.

```bash
pip install lance-tensorflow
```

--------------------------------

### Hot/Cold Tiering Example

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/table/layout.md

Demonstrates how to use multiple base paths to store recent data on a hot tier and historical data on a cold tier, enabling seamless querying without data movement.

```text
Manifest base_paths:
[
  { id: 0, is_dataset_root: true, path: "s3://hot-bucket/dataset" },
  { id: 1, is_dataset_root: true, path: "s3://cold-bucket/dataset-archive" }
]

Fragment 0 (recent data):
  DataFile { path: "fragment-0.lance", base_id: 0 }
  → resolves to: s3://hot-bucket/dataset/data/fragment-0.lance

Fragment 100 (historical data):
  DataFile { path: "fragment-100.lance", base_id: 1 }
  → resolves to: s3://cold-bucket/dataset-archive/data/fragment-100.lance
```

--------------------------------

### Install Lance Python SDK

Source: https://github.com/lance-format/lance/blob/main/docs/src/quickstart/full-text-search.md

Install the necessary dependencies for Lance and PyArrow.

```bash
pip install pylance pyarrow
```

--------------------------------

### CLIP Model and Optimizer Setup

Source: https://github.com/lance-format/lance/blob/main/docs/src/examples/python/clip_training.md

Initializes the CLIP model components, including image and text encoders and projection heads, along with a tokenizer. It then defines the optimizer with specific learning rates for each component.

```python
# Define image encoder, image head, text encoder, text head and a tokenizer for tokenizing the caption
img_encoder = ImageEncoder(model_name=Config.img_encoder_model).to('cuda')
img_head = Head(Config.img_embed_dim, Config.projection_dim).to('cuda')

tokenizer = AutoTokenizer.from_pretrained(Config.text_encoder_model)
text_encoder = TextEncoder(model_name=Config.text_encoder_model).to('cuda')
text_head = Head(Config.text_embed_dim, Config.projection_dim).to('cuda')

# Since we are optimizing two different models together, we will define parameters manually
parameters = [
    {"params": img_encoder.parameters(), "lr": Config.img_enc_lr},
    {"params": text_encoder.parameters(), "lr": Config.text_enc_lr},
    {
        "params": itertools.chain(
            img_head.parameters(),
            text_head.parameters(),
        ),
        "lr": Config.head_lr,
    },
]

optimizer = torch.optim.Adam(parameters)
```

--------------------------------

### Bit-Reversed Position Example

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/table/mem_wal.md

Example of a bit-reversed position 5 encoded as a filename.

```text
1010000000000000000000000000000000000000000000000000000000000000.arrow
```

--------------------------------

### Configurable Segment Partitioning with num_segments

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/index/scalar/fmindex.md

Demonstrates how to configure the FM-Index partitioning using the `num_segments` parameter at index creation time to balance build cost and search performance.

```text
- **`num_segments` parameter**: Configured at index-creation time. If `num_segments` is specified (e.g. `num_segments = 4`), Lance splits the target dataset fragments into disjoint subsets and builds independent FM-Index segments over each chunk.
```

--------------------------------

### JSON Document Input Example

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/index/scalar/fts.md

Example of a nested JSON document structure.

```json
{
  "name": "Lance",
  "legal.age": 30,
  "address": {
    "city": "San Francisco",
    "zip:us": 94102
  }
}
```

--------------------------------

### Text Document Input Example

Source: https://github.com/lance-format/lance/blob/main/docs/src/format/index/scalar/fts.md

Example of a raw text document input.

```text
Tom lives in San Francisco.
```

--------------------------------

### Install Lance using Cargo

Source: https://github.com/lance-format/lance/blob/main/rust/lance/README.md

Install the Lance CLI tool using cargo.

```shell
cargo install lance
```

--------------------------------

### Setup LLM Training Environment with Transformers and Lance

Source: https://github.com/lance-format/lance/blob/main/docs/src/examples/python/llm_training.md

Initializes the environment for LLM training by importing necessary libraries, loading a pre-trained tokenizer and model (GPT-2), and defining key hyperparameters for training. Ensure the 'wikitext_500K.lance' dataset is available.

```python
import numpy as np
import lance

import torch
from torch.utils.data import Dataset, DataLoader, Sampler

from transformers import AutoTokenizer, AutoModelForCausalLM
from tqdm.auto import tqdm

# We'll be training the pre-trained GPT2 model in this example
model_name = 'gpt2'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Also define some hyperparameters
lr = 3e-4
nb_epochs = 10
block_size = 1024
batch_size = 8
device = 'cuda:0'
dataset_path = 'wikitext_500K.lance'
```

--------------------------------

### Build Release

Source: https://github.com/lance-format/lance/blob/main/memtest/README.md

Install the package using make. This command builds the release version of the package.

```shell
make build-release
```

--------------------------------

### Install Lance SDK

Source: https://github.com/lance-format/lance/blob/main/README.md

Commands to install the stable or preview versions of the Lance Python package.

```shell
pip install pylance
```

```shell
pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance
```

--------------------------------

### Install Build Dependencies on Ubuntu

Source: https://github.com/lance-format/lance/wiki/How-to-Build

Installs necessary packages for building Lance on Ubuntu 22.04.

```bash
sudo apt install protobuf-compiler libssl-dev build-essential pkg-config gfortran
```