### Build Documentation with Docker Source: https://github.com/activeloopai/deeplake/blob/main/docs/README.md Use this command to build the documentation and start a local server. It requires Docker to be installed and mounts local directories into the container. ```bash docker run --rm -it -p 8000:8000 -v ${PWD}:/docs -v "${PWD}"/../:/indra -v "${PWD}"/../node_modules:/source/typescript/node_modules -v ${PWD}/../python:/source/python $(docker build --build-arg=GITHUB_TOKEN=$GITHUB_TOKEN -q .) ``` -------------------------------- ### Install Google Benchmark Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/benchmark/README.md Commands to install Google Benchmark. Ensure you have CMake and Make installed. ```bash cmake -DCMAKE_BUILD_TYPE=Release make make install ``` -------------------------------- ### Install ColPali and Accelerate Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Install the necessary libraries for using ColPali and accelerate. ```python !pip install colpali-engine accelerate ``` -------------------------------- ### Install Project Dependencies Source: https://github.com/activeloopai/deeplake/blob/main/CONTRIBUTING.md Installs all necessary dependencies for the project, including common, plugin, and testing requirements. ```bash pip3 install -r deeplake/requirements/common.txt pip3 install -r deeplake/requirements/plugins.txt pip3 install -r deeplake/requirements/tests.txt ``` -------------------------------- ### Install Requests Library Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Installs the 'requests' library, which is necessary for fetching images from URLs. ```bash !pip install requests ``` -------------------------------- ### Synchronization Example Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/advanced/sync.md An example demonstrating the synchronization process between a source and a replica dataset. ```APIDOC ## Synchronization Example ```python # Initial dataset creation source_ds = deep lake.create("s3://bucket/source") source_ds.add_column("images", deep lake.types.Image()) source_ds.commit() # Create replica deep lake.copy( src="s3://bucket/source", dst="gcs://bucket/replica" ) replica_ds = deep lake.open("gcs://bucket/replica") # Add data to source source_ds.append({"images": batch1}) source_ds.commit() # Sync replica with source replica_ds.pull("s3://bucket/source") # Add data to replica replica_ds.append({"images": batch2}) replica_ds.commit() # Push replica changes back to source replica_ds.push("s3://bucket/source") ``` ``` -------------------------------- ### Install ColBERT and PyTorch Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Install the colbert-ai library and PyTorch for using ColBERT models. This is a prerequisite for loading and utilizing ColBERT checkpoints. ```bash #!pip install -U colbert-ai torch ``` -------------------------------- ### Install CLIP and Torchvision Libraries Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Installs the necessary libraries for image embedding generation and multi-modal search. Run these commands in your environment before proceeding. ```python !pip install -U torchvision !pip install git+https://github.com/openai/CLIP.git ``` -------------------------------- ### Install MMDetection and Dependencies Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/deep-learning/mmdet.md Installs PyTorch with CUDA support, mmcv-full, and clones and installs the MMDetection library. Ensure you use compatible versions as specified. ```bash python -m pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 -f https://download.pytorch.org/whl/torch_stable.html python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/index.html git clone -b dev-2.x https://github.com/open-mmlab/mmdetection.git cd mmdetection python3 -m pip install -e . ``` -------------------------------- ### Setup Imports and Authentication Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/deep-learning/mmdet.md Imports necessary libraries for MMDetection and Deep Lake, and sets the Deep Lake authentication token from environment variables. ```python import deeplake from mmcv import Config from mmdet.models import build_detector import os import mmcv # Set your Deep Lake token token = os.environ["ACTIVELOOP_TOKEN"] ``` -------------------------------- ### Install MMSegmentation Prerequisites Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/deep-learning/mmseg.md Installs PyTorch with CUDA support, mmcv-full, and MMSegmentation from source. Also installs a specific numpy version to avoid compatibility issues. ```bash python -m pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 -f https://download.pytorch.org/whl/torch_stable.html python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/index.html git clone https://github.com/open-mmlab/mmsegmentation.git cd mmsegmentation git checkout v0.30.0 python -m pip install -e . # Old pytorch version does not work with the new numpy versions python -m pip install numpy==1.24.4 --force-reinstall ``` -------------------------------- ### Install OpenAI Library Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Installs the OpenAI Python client library, which is required for interacting with OpenAI's API. ```bash !pip install openai ``` -------------------------------- ### Makefile for C++ Example Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/docs/basic-usage.md This Makefile compiles the C++ example code, linking against the SQL parser library. It sets necessary CFLAGS for C++11 and includes paths. ```makefile CFLAGS = -std=c++11 -lstdc++ -Wall -I../src/ -L../ all: $(CXX) $(CFLAGS) example.cpp -o example -lsqlparser ``` -------------------------------- ### Install LangChain and Deep Lake Libraries Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/vectorstore.md Install the necessary libraries for LangChain, OpenAI embeddings, and Deep Lake vector store integration. ```bash pip install --upgrade --quiet langchain-openai langchain-deeplake tiktoken ``` -------------------------------- ### Install Deep Lake Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/deep-learning/deep-learning.ipynb Install the Deep Lake library using pip. ```python !pip3 install deeplake ``` -------------------------------- ### Install Matplotlib Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Installs the matplotlib library, which is required for visualizing images. ```bash !pip install matplotlib ``` -------------------------------- ### Synchronization Example: Source to Replica and Back Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/advanced/sync.md Demonstrates a full synchronization workflow: creating a replica, pulling changes from the source to the replica, adding data to the replica, and then pushing those replica changes back to the source. ```python # Initial dataset creation source_ds = deeplake.create("s3://bucket/source") source_ds.add_column("images", deeplake.types.Image()) source_ds.commit() # Create replica deep lake.copy( src="s3://bucket/source", dst="gcs://bucket/replica" ) replica_ds = deeplake.open("gcs://bucket/replica") # Add data to source source_ds.append({"images": batch1}) source_ds.commit() # Sync replica with source replica_ds.pull("s3://bucket/source") # Add data to replica replica_ds.append({"images": batch2}) replica_ds.commit() # Push replica changes back to source replica_ds.push("s3://bucket/source") ``` -------------------------------- ### Install Deep Lake Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/getting-started/quickstart.md Install the Deep Lake library using pip. This is the first step to using Deep Lake in your project. ```bash pip install deeplake ``` -------------------------------- ### Build SQL Parser Library Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/docs/basic-usage.md Use make commands to build and install the SQL parser library. Ensure you have a modern C++ compiler like gcc 4.8 or clang 3.4. ```bash make # creates libsqlparser.so make install # copies the library to /usr/local/lib/ ``` -------------------------------- ### TQL - Aggregations Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/llms.txt Examples of performing statistical and array aggregations within TQL. ```APIDOC ## TQL - Aggregations ### Description Examples of performing statistical and array aggregations within TQL. ### Query Examples ```sql -- Statistical functions SELECT COUNT(*), AVG(confidence), MAX(score) FROM dataset GROUP BY label -- Array statistics SELECT AVG(embeddings, axis=0), STD(embeddings, axis=1) FROM dataset ``` ``` -------------------------------- ### Install AV Package for Video Processing Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/annotations/labelbox.md Install the 'av' Python package, which is required for extracting frames from videos when using the video ontology. ```bash python -m pip install av ``` -------------------------------- ### Index Management for Columns Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/column.md Provides examples for creating, listing, and dropping indexes on columns. Supports text search and embedding similarity indexes. ```python # Create text search index (Column only) column.create_index(deeplake.types.TextIndex(deeplake.types.BM25)) # Create embedding similarity index (Column only) column.create_index(deeplake.types.EmbeddingIndex()) # List existing indexes print(f"Current indexes: {column.indexes}") # Drop an index column.drop_index(deeplake.types.TextIndex(deeplake.types.BM25)) ``` -------------------------------- ### Install Labelbox Python Package Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/annotations/labelbox.md Install the Labelbox Python package using pip. This is a prerequisite for using Labelbox integrations. ```bash python -m pip install labelbox ``` -------------------------------- ### TQL Text Search Queries Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/advanced/tql.md Provides examples of TQL queries utilizing text indexes for keyword, semantic, and pattern matching searches. ```sql -- Keyword search (requires Inverted index) SELECT * WHERE CONTAINS(description, 'machine learning') ``` ```sql -- Semantic search (requires BM25 index) SELECT * ORDER BY BM25_SIMILARITY(content, 'deep learning tutorial') DESC LIMIT 10 ``` ```sql -- Pattern matching (requires Inverted index) SELECT * WHERE description LIKE '*neural network*' ``` -------------------------------- ### Start MMDetection Training with Deep Lake Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/deep-learning/mmdet.md Loads the MMDetection configuration, builds the detector model, creates the working directory, and initiates the training process using Deep Lake's integration. ```python # Load config cfg = Config.fromfile(config_path) # Build the detector model = build_detector(cfg.model) # Create work directory mmcv.mkdir_or_exist(os.path.abspath(cfg.work_dir)) # Start training from deeplake.integrations import mmdet as mmdet_deeplake mmdet_deeplake.train_detector( model, cfg, distributed=False, # Set to True for multi-GPU training validate=False # Set to True if you have validation data ) ``` -------------------------------- ### List and Get Read-Only Branch Views Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/version_control.md Demonstrates how to access a read-only view of branches, list their names, and retrieve specific branch views. ```python # Access read-only branches brances_view = ds.branches # List branch names for name in branches_view.names(): print(f"Found branch: {name}") # Get specific branch branch_view = branches_view["B1"] ``` -------------------------------- ### Test Storage Provider Fixture Usage Source: https://github.com/activeloopai/deeplake/blob/main/CONTRIBUTING.md Example of using the `@enabled_storages` fixture to run tests for each enabled storage provider. Defaults to memory if no providers are explicitly enabled. ```python @enabled_storages def test_storage(storage: StorageProvider): # this test will run once per enabled storage provider. if no providers are explicitly enabled, # only memory will be used. pass ``` -------------------------------- ### Deep Lake TQL Cross-Cloud Joins Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/llms.txt Illustrates how to perform joins between datasets stored in different cloud providers (S3, GCS, Azure) using TQL. The example joins images, embeddings, and metadata based on common IDs. ```sql -- Join datasets across cloud providers SELECT i.image, e.embedding, m.metadata FROM "s3://bucket1/images" AS i JOIN "gcs://bucket2/embeddings" AS e ON i.id = e.image_id JOIN "azure://container/meta" AS m ON i.id = m.image_id WHERE m.verified = true ORDER BY COSINE_SIMILARITY(e.embedding, ARRAY[0.1,0.2,0.3]) DESC ``` -------------------------------- ### Joining Datasets Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/query.md Join data across different datasets and cloud storage locations. The example demonstrates selecting and joining data from multiple S3 buckets based on a common ID. ```APIDOC ## Joining Datasets Join data across different datasets and across different clouds: ```python # Join datasets from different storage results = deeplake.query(""" SELECT i.image, i.embedding, m.labels, m.metadata FROM \"s3://bucket1/images\" AS i JOIN \"s3://bucket2/metadata\" AS m ON i.id = m.image_id WHERE m.verified = true ") ``` ``` -------------------------------- ### TQL - Vector Similarity Search Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/llms.txt Examples of performing vector similarity searches using various distance metrics like Cosine, L2, and L1 norms, as well as Inner Product. ```APIDOC ## TQL - Vector Similarity Search ### Description Examples of performing vector similarity searches using various distance metrics like Cosine, L2, and L1 norms, as well as Inner Product. ### Query Examples ```sql -- Cosine similarity (higher = more similar) SELECT * ORDER BY COSINE_SIMILARITY(embeddings, ARRAY[0.1,0.2,0.3]) DESC LIMIT 100 -- L2/Euclidean distance (lower = more similar) SELECT * ORDER BY L2_NORM(embeddings - ARRAY[0.1,0.2,0.3]) ASC LIMIT 100 -- L1/Manhattan distance SELECT * ORDER BY L1_NORM(embeddings - ARRAY[0.1,0.2,0.3]) ASC LIMIT 100 -- Inner product similarity SELECT * ORDER BY INNER_PRODUCT(embeddings, ARRAY[0.1,0.2,0.3]) DESC LIMIT 100 ``` ``` -------------------------------- ### Create and Open a Dataset Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/dataset.md Demonstrates how to create a new dataset or open an existing one using local or cloud storage. Includes adding columns, appending data, and committing changes. ```python ds = deeplake.create("s3://bucket/dataset") # or ds = deeplake.open("s3://bucket/dataset") # Can modify ds.add_column("images", deeplake.types.Image()) ds.add_column("labels", deeplake.types.ClassLabel("int32")) ds.add_column("confidence", "float32") ds["labels"].metadata["class_names"] = ["cat", "dog"] ds.append([{"images": image_array, "labels": 0, "confidence": 0.9}]) ds.commit() ``` -------------------------------- ### Configure hsql Project and Library Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/CMakeLists.txt Sets up the project name and defines the library to be built. It also links against the fmt library. ```cmake project(hsql) file(GLOB_RECURSE SOURCES "src/*.cpp" "src/*.c") ADD_LIBRARY(hsql ${SOURCES}) target_link_libraries(hsql PRIVATE fmt::fmt-header-only) ``` -------------------------------- ### Configure vcpkg for Building Source: https://github.com/activeloopai/deeplake/blob/main/postgres/README.md Clone the vcpkg repository, set the VCPKG_ROOT environment variable, checkout a specific commit, bootstrap vcpkg, and add it to your bashrc for persistent access. ```bash git clone https://github.com/microsoft/vcpkg.git cd vcpkg/ export VCPKG_ROOT=`pwd` git checkout 6f29f12e82a8293156836ad81cc9bf5af41fe836 ./bootstrap-vcpkg.sh echo "export VCPKG_ROOT=$VCPKG_ROOT" >> ~/.bashrc # or path to your shell config file echo "export PATH=$PATH:$VCPKG_ROOT" >> ~/.bashrc # or path to your shell config file source ~/.bashrc cd ../ ``` -------------------------------- ### Connect to PostgreSQL with psql Source: https://github.com/activeloopai/deeplake/blob/main/postgres/README.md Connect to the PostgreSQL instance using the psql command-line client. Specify the host, port, and user. ```bash psql -h localhost -p 5432 -U postgres ``` -------------------------------- ### Prepared Statements Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/docs/syntax-support.md Illustrates the definition and execution of prepared statements with parameter placeholders. ```sql PREPARE select_test FROM 'SELECT * FROM customer WHERE c_name = ?;'; EXECUTE select_test('Max Mustermann'); ``` -------------------------------- ### Define Document Class and Softmax Function Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Sets up a Document class using Pydantic for structured data representation and implements a softmax function for normalizing retrieval scores. Install required libraries using 'pip install numpy pydantic'. ```bash pip install numpy pydantic ``` ```python import math import numpy as np from typing import Any, Dict, List, Optional from pydantic import BaseModel class Document(BaseModel): id: str data: Dict[str, Any] score: Optional[float] = None def softmax(retrieved_score: list[float], max_weight: int = 700) -> Dict[str, Document]: # Compute the exponentials exp_scores = [math.exp(min(score, max_weight)) for score in retrieved_score] # Compute the sum of the exponentials sum_exp_scores = sum(exp_scores) # Update the scores of the documents using softmax new_weights = [] for score in exp_scores: new_weights.append(score / sum_exp_scores) return new_weights ``` -------------------------------- ### TQL - Basic Syntax Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/llms.txt Demonstrates the fundamental syntax for querying single and cross-dataset operations using Tensor Query Language (TQL). ```APIDOC ## TQL - Basic Syntax ### Description Demonstrates the fundamental syntax for querying single and cross-dataset operations using Tensor Query Language (TQL). ### Syntax Examples ```sql -- Single dataset query (no FROM needed) SELECT * WHERE id > 10 -- Cross-dataset query (FROM required) SELECT * FROM "s3://bucket/dataset" WHERE condition ``` ``` -------------------------------- ### Create and Open Datasets in Deep Lake Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/llms.txt Demonstrates how to create new datasets or open existing ones with different access modes (read-write, read-only). Also shows how to copy dataset schemas and import data from Parquet files. ```python ds = deeplake.create("s3://bucket/path") # Create new dataset ``` ```python ds = deeplake.open("s3://bucket/path") # Read-write access ``` ```python ds = deeplake.open_read_only("path") # Read-only access ``` ```python ds = deeplake.like(source_ds, "new/path") # Copy schema ``` ```python ds = deeplake.from_parquet("file.parquet", "path") # Import from Parquet ``` -------------------------------- ### UPDATE Statement Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/docs/syntax-support.md Provides an example of updating existing records in a table based on a WHERE clause. ```sql UPDATE students SET name='Max Mustermann' WHERE name = 'Ralf Mustermann'; ``` -------------------------------- ### Tag Operations Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/version_control.md Provides examples of common operations performed on tags, such as opening datasets at a specific tag. ```APIDOC ## Tag Operations ### Description Perform common operations using tags, such as opening a dataset at a specific tagged version in synchronous or asynchronous mode. ### Usage ```python # Open dataset at a specific tag (e.g., "v2.0") latest_ds = ds.tags["v2.0"].open() # Asynchronously open dataset at a specific tag (e.g., "v1.0") stable_ds = ds.tags["v1.0"].open_async() ``` ``` -------------------------------- ### Instantiate SelfQueryRetriever Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/vectorstore.md Set up the SelfQueryRetriever with an LLM, the Deep Lake vector store, a description of document content, and detailed metadata field information. ```python from langchain.llms import OpenAI from langchain.retrievers.self_query.base import SelfQueryRetriever from langchain.chains.query_constructor.base import AttributeInfo metadata_field_info = [ AttributeInfo( name="genre", description="The genre of the movie", type="string or list[string]", ), AttributeInfo( name="year", description="The year the movie was released", type="integer", ), AttributeInfo( name="director", description="The name of the movie director", type="string", ), AttributeInfo( name="rating", description="A 1-10 rating for the movie", type="float" ), ] document_content_description = "Brief summary of a movie" llm = OpenAI(temperature=0) retriever = SelfQueryRetriever.from_llm( llm, vectorstore, document_content_description, metadata_field_info, verbose=True ) ``` -------------------------------- ### TQL - Array Operations Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/llms.txt Examples of performing operations on array data within TQL, such as slicing, filtering, and aggregation. ```APIDOC ## TQL - Array Operations ### Description Examples of performing operations on array data within TQL, such as slicing, filtering, and aggregation. ### Query Examples ```sql -- Array slicing SELECT features[:, 0:10] FROM dataset -- Array filtering SELECT * WHERE features[0] > 0.5 -- Array aggregation SELECT AVG(features, axis=0) FROM dataset ``` ``` -------------------------------- ### TQL - Cross-Cloud Joins Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/llms.txt Demonstrates how to join datasets residing in different cloud storage providers using TQL. ```APIDOC ## TQL - Cross-Cloud Joins ### Description Demonstrates how to join datasets residing in different cloud storage providers using TQL. ### Query Examples ```sql -- Join datasets across cloud providers SELECT i.image, e.embedding, m.metadata FROM "s3://bucket1/images" AS i JOIN "gcs://bucket2/embeddings" AS e ON i.id = e.image_id JOIN "azure://container/meta" AS m ON i.id = m.image_id WHERE m.verified = true ORDER BY COSINE_SIMILARITY(e.embedding, ARRAY[0.1,0.2,0.3]) DESC ``` ``` -------------------------------- ### Set Up OpenAI and Activeloop Environment Variables Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/vectorstore.md Configure the OpenAI API key and Activeloop token as environment variables, prompting the user if they are not already set. ```python if "OPENAI_API_KEY" not in os.environ: os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:") if "ACTIVELOOP_TOKEN" not in os.environ: os.environ["ACTIVELOOP_TOKEN"] = getpass.getpass("activeloop token:") ``` -------------------------------- ### Document Structure Example Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Illustrates the structure of a Document object within Deep Lake, including its ID, data payload, and associated score. ```python '2496': Document(id='2496', data={'restaurant_name': 'Seasons Noodles & Dumplings Garden', 'restaurant_review': 'Comfort food, excellent service! Feel like back to home.'}, score=0.02172369191268401), '10788': Document(id='10788', data={'restaurant_name': 'Casa Lupe', 'restaurant_review': 'Run by a family that makes you feel like part of the family. Awesome food. I love their wet Chili Verde burritos'}, score=0.02088547418398944) ``` -------------------------------- ### Run PostgreSQL Extension Tests Source: https://github.com/activeloopai/deeplake/blob/main/postgres/README.md Navigate to the postgres/tests directory and run the tests using the make test command. ```bash cd postgres/tests make test ``` -------------------------------- ### List and Access Branches Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/version_control.md Illustrates how to list all available branches by name, access specific branches, and open datasets associated with them. Includes error handling for non-existent branches. ```python # Create branch ds.branch("B1") # List all branches for name in ds.branches.names(): br = ds.branches[name] print(f"Branch: {br.name} based on {br.base}") # Check number of branches num_branches = len(ds.branches) # Access specific branch branch = ds.branches["main"] # Common operations with branches branch_ds = ds.branches["B1"].open() # Open branch branch_future = ds.branches["B1"].open_async() # Async open # Error handling try: branch = ds.branches["non_existent"] except deeplake.BranchNotFoundError: print("Branch not found") ``` -------------------------------- ### Create and Manage Dataset Tags Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/version_control.md Demonstrates how to create tags for the current or specific versions of a dataset, list all tags, and access individual tags. Includes error handling for non-existent tags. ```python ds.tag("v1.0") # Tag current version specific_version = ds.version ds.tag("v2.0", version=specific_version) # Tag specific version # List all tags for name in ds.tags.names(): tag = ds.tags[name] print(f"Tag {tag.name} points to version {tag.version}") # Check number of tags num_tags = len(ds.tags) # Access specific tag tag = ds.tags["v1.0"] # Common operations with tags latest_ds = ds.tags["v2.0"].open() # Open dataset at tag stable_ds = ds.tags["v1.0"].open_async() # Async open # Error handling try: tag = ds.tags["non_existent"] except deeplake.TagNotFoundError: print("Tag not found") ``` -------------------------------- ### Format Information and Generate Answer Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Formats retrieved restaurant information into a list of strings and then calls the `generate_question` function to get an LLM-generated answer. ```python information = [f'Review: {el["restaurant_review"]}, Restaurant name: {el["restaurant_name"]}' for el in view_vs] result = generate_question(query, information) print(result) ``` -------------------------------- ### Build and Test SQL Parser Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/docs/dev-docs.md Commands to build the parser, library, and run tests. Rerun these steps whenever you change part of the parser. ```bash make parser make library make test ``` ```bash make cleanall make test ``` -------------------------------- ### Query Single and Multiple Datasets with TQL Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/advanced/tql.md Demonstrates how to query a single dataset without a FROM clause and how to query across multiple datasets using the FROM clause. ```python # Query on a single dataset (no FROM needed) ds = deeplake.open("al://org_name/dataset_name") result = ds.query("SELECT * WHERE id > 10") # Query across datasets (requires FROM) result = deeplake.query('SELECT * FROM "al://my_org/dataset_name" WHERE id > 10') ``` -------------------------------- ### Run Q&A Chain Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/vectorstore.md Execute the RetrievalQA chain with a specific question to get an answer based on the documents in the Deep Lake vector store. ```python qa.run('What programming language is most of the SimClusters written in?') ``` -------------------------------- ### Manage Deep Lake Indexes in Python Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/advanced/tql.md Provides Python code examples for checking, dropping, and committing index changes in a Deep Lake dataset. ```python # Check existing indexes print(ds["column_name"].indexes) ``` ```python # Drop an index - use columns we created earlier ds["article"].drop_index(deeplake.types.TextIndex(deeplake.types.BM25)) ds["age"].drop_index( deeplake.types.NumericIndex(deeplake.types.Inverted) ) ``` ```python # Commit changes ds.commit() ``` -------------------------------- ### Initialize Deep Lake Dataset Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/deep-learning/async-data-loader.md Initializes a temporary Deep Lake dataset for testing purposes. This setup is common before defining custom dataset classes. ```python import numpy as np import torch import deeplake from deeplake import types from typing import Callable ds = deeplake.create("tmp://") ``` -------------------------------- ### Array Dimensions with SHAPE Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/advanced/tql.md The SHAPE function returns the dimensions of an array. Use it to filter based on array size, for example, checking the number of dimensions or elements. ```sql SELECT * WHERE SHAPE(embedding)[0] = 768 ``` ```sql SELECT * WHERE SHAPE(boxes)[0] > 10 -- More than 10 bounding boxes ``` -------------------------------- ### Basic SELECT Statement Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/docs/syntax-support.md Demonstrates a basic SELECT statement with joins, WHERE clauses, and ORDER BY. ```sql SELECT name, city, * FROM students AS t1 JOIN students AS t2 ON t1.city = t2.city WHERE t1.grade < 2.0 AND t2.grade > 2.0 AND t1.city = 'Frohnau' ORDER BY t1.grade DESC; ``` -------------------------------- ### Initialize Chat Model and RetrievalQA Chain Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/vectorstore.md Instantiate a ChatOpenAI model and create a RetrievalQA chain, linking the LLM with the configured retriever for question answering. ```python model = ChatOpenAI(model='gpt-3.5-turbo') qa = RetrievalQA.from_llm(model, retriever=retriever) ``` -------------------------------- ### Deep Learning Training with PyTorch DataLoader Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/index.md Integrate Deep Lake with PyTorch for efficient deep learning training. This example shows how to create a DataLoader for batch processing. ```python # PyTorch integration from torch.utils.data import DataLoader loader = DataLoader(ds.pytorch(), batch_size=32, shuffle=True) for batch in loader: images = batch["images"] labels = batch["labels"] # training code... ``` -------------------------------- ### Prepared Queries Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/query.md Explains how to use prepared queries for reusability with different parameters, including single and batch execution, and asynchronous calls. ```APIDOC ## Prepared Queries Prepare queries for reuse with different parameters: ```python # Prepare a parameterized query s_executor = deeplake.prepare_query(""" SELECT * FROM "s3://bucket/dataset" WHERE label = 'cat' AND confidence > 0.5 ") m_executor = deeplake.prepare_query("SELECT * FROM "s3://bucket/dataset"\n WHERE label = ? AND confidence > ?") # Execute with different parameters cats_high = s_executor.run_single() # Batch execution results = m_executor.run_batch([ ["cat", 0.9], ["dog", 0.8], ["bird", 0.7] ]) # Async execution future = m_executor.run_single_async(["cat", 0.95]) result = future.result() # Get the query string print(f"Query: {m_executor.get_query_string()}") ``` ``` -------------------------------- ### Vector Similarity Search Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/query.md Perform vector similarity search using cosine similarity. The example demonstrates querying embeddings from a specified bucket and ordering results by similarity. ```APIDOC ## Vector Similarity Search Search by vector similarity: ```python # Cosine similarity search text_vector = ','.join(str(x) for x in search_vector) results = deeplake.query(f""" SELECT * FROM \"s3://bucket/embeddings\" ORDER BY COSINE_SIMILARITY(embeddings, ARRAY[{text_vector}]) DESC LIMIT 100 ") ``` ``` -------------------------------- ### Create LangChain Documents Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/vectorstore.md Prepare a list of Document objects, each containing page content and associated metadata for the vector store. ```python from langchain_core.documents import Document docs = [ Document( page_content="A bunch of scientists bring back dinosaurs and mayhem breaks loose", metadata={"year": 1993, "rating": 7.7, "genre": "science fiction"}, ), Document( page_content="Leo DiCaprio gets lost in a dream within a dream within a dream within a ...", metadata={"year": 2010, "director": "Christopher Nolan", "rating": 8.2}, ), Document( page_content="A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea", metadata={"year": 2006, "director": "Satoshi Kon", "rating": 8.6}, ), Document( page_content="A bunch of normal-sized women are supremely wholesome and some men pine after them", metadata={"year": 2019, "director": "Greta Gerwig", "rating": 8.3}, ), Document( page_content="Toys come alive and have a blast doing so", metadata={"year": 1995, "genre": "animated"}, ), Document( page_content="Three men walk into the Zone, three men walk out of the Zone", metadata={ "year": 1979, "rating": 9.9, "director": "Andrei Tarkovsky", "genre": "science fiction", "rating": 9.9, }, ), ] ``` -------------------------------- ### Async Query Operations Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/dataset.md Shows how to perform asynchronous queries and data access operations. Futures are returned and can be resolved to get results, improving performance for I/O-bound tasks. ```python # Async query works on all types future = ds.query_async("SELECT * WHERE labels = 'cat'") results = future.result() # Async data access future = ds["images"].get_async(slice(0, 1000)) images = future.result() ``` -------------------------------- ### Computer Vision Data Storage with Annotations Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/index.md Store computer vision data such as images, bounding boxes, and segmentation masks in Deep Lake. Includes examples of compression options. ```python # Store images and annotations ds = deeplake.create("s3://my-bucket/dataset") # or local path ds.add_column("images", deeplake.types.Image(sample_compression="jpeg")) ds.add_column("boxes", deeplake.types.BoundingBox()) ds.add_column("masks", deeplake.types.SegmentMask(sample_compression='lz4')) # Add data ds.append({ "images": imgs, "boxes": bboxes, "masks": smasks }) ``` -------------------------------- ### Async Queries Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/query.md Illustrates how to execute queries asynchronously using `query_async` and retrieve results once they are ready. ```APIDOC ## Async Queries Execute queries asynchronously: ```python # Run query asynchronously future = deeplake.query_async(""" SELECT * FROM "s3://bucket/dataset" ORDER BY COSINE_SIMILARITY(embeddings, ARRAY[0.1,0.2, 0.3]) DESC ") # Get results when ready results = future.result() # Check completion if future.is_completed(): results = future.result() else: print("Query still running") ``` ``` -------------------------------- ### Text Search Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/query.md Execute text searches using BM25 for semantic relevance or CONTAINS for keyword matching. Examples show how to query documents and metadata based on text content. ```APIDOC ## Text Search Text search using BM25 or keyword matching: ```python # Semantic search using BM25 results = deeplake.query(""" SELECT * FROM \"s3://bucket/documents\" ORDER BY BM25_SIMILARITY(text, 'search query') DESC LIMIT 10 ") # Keyword search using CONTAINS results = deeplake.query(""" SELECT * FROM \"s3://bucket/metadata\" WHERE CONTAINS(keywords, 'specific term') ") ``` ``` -------------------------------- ### Basic pg_deeplake Usage in SQL Source: https://github.com/activeloopai/deeplake/blob/main/postgres/README.md Enable the pg_deeplake extension, create a table with DeepLake storage, create a DeepLake index, insert data, and perform a cosine similarity query. ```sql -- 1. Enable extension CREATE EXTENSION pg_deeplake; -- 2. Create a table with DeepLake storage CREATE TABLE vectors ( id SERIAL PRIMARY KEY, v1 float4[], v2 float4[] ) USING deeplake; -- 3. Create an index CREATE INDEX index_for_v1 ON vectors USING deeplake_index (v1 DESC); -- 4. Insert data INSERT INTO vectors (v1, v2) VALUES (ARRAY[1.0, 2.0, 3.0], ARRAY[1.0, 2.0, 3.0]), (ARRAY[4.0, 5.0, 6.0], ARRAY[4.0, 5.0, 6.0]), (ARRAY[7.0, 8.0, 9.0], ARRAY[7.0, 8.0, 9.0]); -- 5. Query with cosine similarity SELECT id, v1 <#> ARRAY[1.0, 2.0, 3.0] AS score FROM vectors ORDER BY score DESC LIMIT 10; ``` -------------------------------- ### Test Memory Storage Fixture Usage Source: https://github.com/activeloopai/deeplake/blob/main/CONTRIBUTING.md Example of using the `memory_storage` fixture for tests that run only with a memory storage provider. This test is skipped if the `--memory-skip` option is provided. ```python def test_memory_storage(memory_storage: StorageProvider): # this test will run only once with a memory storage provider. if the `--memory-skip` option is provided, # this test will be skipped. ``` -------------------------------- ### Test Local Dataset Fixture Usage Source: https://github.com/activeloopai/deeplake/blob/main/CONTRIBUTING.md Example of using the `local_ds` fixture for tests that specifically require a local dataset. This test is skipped if the `--local` option is not provided. ```python def test_local_dataset(local_ds: Dataset): # this test will run only once with a local dataset. if the `--local` option is not provided, # this test will be skipped. ``` -------------------------------- ### Prepare and Execute Parameterized Queries Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/query.md Prepare queries with placeholders for reuse, allowing execution with different parameters. Supports single and batch execution, as well as asynchronous runs. ```python # Prepare a parameterized query s_executor = deeplake.prepare_query(""" SELECT * FROM "s3://bucket/dataset" WHERE label = 'cat' AND confidence > 0.5 ") m_executor = deeplake.prepare_query("SELECT * FROM \"s3://bucket/dataset\"\n WHERE label = ? AND confidence > ?") # Execute with different parameters cats_high = s_executor.run_single() # Batch execution results = m_executor.run_batch([ ["cat", 0.9], ["dog", 0.8], ["bird", 0.7] ]) # Async execution future = m_executor.run_single_async(["cat", 0.95]) result = future.result() # Get the query string print(f"Query: {m_executor.get_query_string()}") ``` -------------------------------- ### Test Dataset Fixture Usage Source: https://github.com/activeloopai/deeplake/blob/main/CONTRIBUTING.md Example of using the `@enabled_datasets` fixture to run tests across different enabled storage providers for datasets. If no providers are enabled, only memory will be used. ```python @enabled_datasets def test_dataset(ds: Dataset): # this test will run once per enabled storage provider. if no providers are explicitly enabled, # only memory will be used. pass ``` -------------------------------- ### Construct and Execute TQL Query Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Use Python f-strings to dynamically build a TQL query with filtering and ordering. Then, execute the query against the dataset and process the results. ```python tql_colbert = f" SELECT *, maxsim(embedding, {q_str}) as score ORDER BY maxsim(embedding, {q_str}) DESC LIMIT {n_res} " # Execute the query and append the results results = medical_dataset.query(tql_colbert) ``` ```python for res in results: print(f"Text: {res['text']}") ``` -------------------------------- ### Access Read-Only Columns and Data Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/column.md Demonstrates how to open a dataset in read-only mode and access individual columns. Includes examples for reading single items, batches, and performing asynchronous reads. ```python # Get read-only column ro_ds = deeplake.open_read_only("s3://bucket/dataset") ro_column = ro_ds["images"] # Read data image = ro_column[0] batch = ro_column[0:100] # Async read future = ro_column.get_async(slice(0, 100)) batch = future.result() ``` -------------------------------- ### TQL Vector Similarity Search (Cosine, L2, L1, L-infinity) Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/advanced/tql.md Demonstrates TQL syntax for performing vector similarity searches using Cosine, L2 norm, L1 norm, and L-infinity norm. ```sql -- Cosine similarity (higher is more similar) SELECT * ORDER BY COSINE_SIMILARITY(embeddings, ARRAY[0.1, 0.2, ...]) DESC LIMIT 100 -- L2 norm/Euclidean distance (lower is more similar) SELECT * ORDER BY L2_NORM(embeddings - ARRAY[0.1, 0.2, ...]) ASC LIMIT 100 -- L1 norm/Manhattan distance SELECT * ORDER BY L1_NORM(embeddings - ARRAY[0.1, 0.2, ...]) ASC LIMIT 100 -- L∞ norm/Chebyshev distance SELECT * ORDER BY LINF_NORM(embeddings - ARRAY[0.1, 0.2, ...]) ASC LIMIT 100 ``` -------------------------------- ### Working with Schema Objects Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/schemas.md Illustrates how to access and manipulate dataset schemas after a dataset has been opened. This includes getting column definitions, checking the number of columns, and accessing schemas in read-only mode. ```APIDOC ## Working with Schema Objects Access and manipulate dataset schemas: ```python # Access dataset schema ds = deeplake.open("s3://bucket/dataset") schema = ds.schema # Get column definition image_col = schema["images"] print(f"Image column type: {image_col.dtype}") # Get number of columns num_columns = len(schema) print(f"Dataset has {num_columns} columns") # Read-only schema access ro_ds = deeplake.open_read_only("s3://bucket/dataset") ro_schema = ro_ds.schema # Access column definition (read-only) label_col = ro_schema["labels"] print(f"Label column type: {label_col.dtype}") ``` ``` -------------------------------- ### Creating and Using a Numeric Index Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/types.md Illustrates how to create a numeric index for a column and then use it in queries with comparison operators. ```APIDOC ## Creating and Using a Numeric Index This example demonstrates creating a numeric index on the 'age' column for efficient range queries and then using it in `ds.query`. ### Code Example ```python # Create numeric index for efficient range queries ds.add_column("age", deeplake.types.Int32()) ds["age"].create_index( deeplake.types.NumericIndex(deeplake.types.Inverted) ) # Use in queries with comparison operators results = ds.query("SELECT * WHERE age > 25") results = ds.query("SELECT * WHERE age BETWEEN 18 AND 65") results = ds.query("SELECT * WHERE age IN (25, 30, 35)") ``` ``` -------------------------------- ### Load CLIP Model and Set Device Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Loads the CLIP model ('ViT-B/32') and its preprocessing steps, setting the device to CUDA if available, otherwise CPU. This prepares the model for multi-modal tasks. ```python import torch import clip device = "cuda" if torch.cuda.is_available() else "cpu" model, preprocess = clip.load("ViT-B/32", device=device) ``` -------------------------------- ### Dataset Management in Deep Lake Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/dataset.md Provides examples for managing Deep Lake datasets, including checking existence, creating new datasets, enabling auto-commit, refreshing data, and deleting datasets. ```python # Check if dataset exists if deeplake.exists("s3://bucket/dataset"): ds = deeplake.open("s3://bucket/dataset") else: ds = deeplake.create("s3://bucket/dataset") # Auto-commit functionality ds.auto_commit_enabled = True # Enable automatic commits # Refresh dataset to get latest changes ds.refresh() # Delete dataset (irreversible!) deelake.delete("s3://old-bucket/dataset") ``` -------------------------------- ### Test Cache Chain Fixture Usage Source: https://github.com/activeloopai/deeplake/blob/main/CONTRIBUTING.md Example of using the `@enabled_cache_chains` fixture to run tests for cache chains. Caches are provided as `StorageProvider`s. This test is skipped if only memory is enabled and no other providers are explicitly enabled. ```python @enabled_cache_chains def test_cache(cache_chain: StorageProvider): # note: caches are provided as `StorageProvider`s # this test runs for every cache chain that contains all enabled storage providers. # if only memory is enabled (no providers are explicitly enabled), this test will be skipped. pass ``` -------------------------------- ### Basic SQL Parsing in C++ Source: https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/README.md Demonstrates how to parse a SQL query string into C++ objects using the SQLParser library. It includes checking for validity and accessing the first statement, specifically a SELECT statement. ```cpp #include "hsql/SQLParser.h" /* ... */ { // Basic Usage Example const std::string query = "..."; hsql::SQLParserResult result; hsql::SQLParser::parse(query, &result); if (result.isValid() && result.size() > 0) { const hsql::SQLStatement* statement = result.getStatement(0); if (statement->isType(hsql::kStmtSelect)) { const auto* select = static_cast(statement); /* ... */ } } } ``` -------------------------------- ### Query Explanation Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/query.md Details how to analyze query execution plans using `explain_query` to understand and optimize query performance. ```APIDOC ## Query Explanation Analyze query execution plans: ```python # Explain query performance explanation = deeplake.explain_query(""" SELECT * FROM "s3://bucket/large_dataset" WHERE category = 'cat' ORDER BY COSINE_SIMILARITY(embeddings, ARRAY[0.1, 0.2, 0.3]) DESC LIMIT 1000 ") # Print explanation print(explanation) # Get explanation as dictionary explain_dict = explanation.to_dict() print(f"Execution plan: {explain_dict}") # Use explanation to optimize queries if "index_used" in explain_dict: print("Query will use indexes for optimization") ``` ``` -------------------------------- ### Read, Write, and Async Operations on Mutable Columns Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/api/column.md Demonstrates how to open a mutable dataset, access a column, and perform read, write, and asynchronous set operations. Ensure the dataset is opened in a mutable mode to use these operations. ```python # Get mutable column ds = deeplake.open("s3://bucket/dataset") column = ds["images"] # Read data image = column[0] batch = column[0:100] # Write data column[0] = new_image column[0:100] = new_batch # Async operations future = column.set_async(0, new_image) future.wait() ``` -------------------------------- ### Configure GCP Bucket CORS Policy Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/getting-started/storage-and-creds/managed-credentials/gcp/cors.md Add this JSON snippet to the CORS section of your GCP bucket's Permissions tab. It allows the Deep Lake app to make GET and HEAD requests. ```json [ { "origin": ["https://app.activeloop.ai"], "method": ["GET", "HEAD"], "responseHeader": ["*"], "maxAgeSeconds": 3600 } ] ``` -------------------------------- ### Query Vector Search with Cosine Similarity and Filter Source: https://github.com/activeloopai/deeplake/blob/main/docs/docs/guide/rag.md Performs a vector search similar to the previous example but adds a filter to include only results where the 'owner_answer' column contains a specific word. This utilizes an inverted index for efficient filtering. ```python word = "Thank you" query_vs = f" SELECT *, cosine_similarity(embedding, ARRAY[{str_query}]) as score FROM ( SELECT *, ROW_NUMBER() AS row_id ) WHERE CONTAINS(owner_answer, '{word}') ORDER BY cosine_similarity(embedding, ARRAY[{str_query}]) DESC LIMIT 3 " view_vs = vector_search.query(query_vs) for row in view_vs: print(f"Restaurant name: {row['restaurant_name']} Review: {row['restaurant_review']} Owner Answer: {row['owner_answer']}") ```