### Install pyvespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/query.ipynb

Installs the pyvespa library using pip3, a prerequisite for interacting with Vespa instances and running the examples in this guide.

```python
!pip3 install pyvespa
```

--------------------------------

### Install Pyvespa Library

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/pyvespa-examples.ipynb

Installs the pyvespa Python library using pip3. This is a necessary first step to set up the environment for running the subsequent examples.

```python
!pip3 install pyvespa
```

--------------------------------

### Install pyvespa and Vespa CLI

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

This command installs the necessary Python library `pyvespa` and the `vespacli` command-line interface. `pyvespa` is used for interacting with Vespa, and `vespacli` is essential for managing data and control plane keys in Vespa Cloud.

```python
!pip3 install pyvespa vespacli
```

--------------------------------

### Clone pyvespa Repository and Start Jupyter Notebook

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/index.rst

Commands to clone the pyvespa repository from GitHub and then launch Jupyter Notebook, pointing to the documentation source directory. This setup allows users to run and experiment with the provided notebooks and guides locally.

```bash
$ git clone --depth 1 https://github.com/vespa-engine/pyvespa.git
$ jupyter notebook --notebook-dir pyvespa/docs/sphinx/source
```

--------------------------------

### Initialize Vespa App for Advanced Querybuilder Examples

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/query.ipynb

This snippet initializes a `Vespa` application instance, pointing it to the public Vespa documentation search API (`https://api.search.vespa.ai`). This setup is crucial for running subsequent advanced Querybuilder examples that require specific schemas not available in a local sample app.

```python
app = Vespa(url="https://api.search.vespa.ai")
```

--------------------------------

### Install pyvespa and Check Docker Memory

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/advanced-configuration.ipynb

This snippet demonstrates the necessary initial setup steps: installing the `pyvespa` library using `pip` and verifying the available memory in the Docker environment. These steps are crucial prerequisites for deploying and running Vespa applications locally.

```python
!pip3 install pyvespa
!docker info | grep "Total Memory"
```

--------------------------------

### Prepare for Vespa Cloud Deployment

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

This Python snippet imports necessary modules (`VespaCloud` from `vespa.deployment` and `os`) to begin the process of deploying a previously defined Vespa application package to Vespa Cloud.

```python
from vespa.deployment import VespaCloud
import os
```

--------------------------------

### Perform Plain Keyword Search with BM25 Ranking in Vespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa.ipynb

This example demonstrates how to execute a plain keyword search against the Vespa instance using `app.syncio` for efficient connection pooling. It specifies a YQL query, the search term, and utilizes the 'bm25' ranking profile configured in the application package. The results are then displayed using the helper function.

```python
with app.syncio(connections=1) as session:
    query = "How Fruits and Vegetables Can Treat Asthma?"
    response: VespaQueryResponse = session.query(
        yql="select * from sources * where userQuery() limit 5",
        query=query,
        ranking="bm25",
    )
    assert response.is_successful()
    print(display_hits_as_df(response, ["id", "title"]))
```

--------------------------------

### Install Poppler Utilities for PDF Processing

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/visual_pdf_rag_with_vespa_colpali_cloud.ipynb

Installs 'poppler-utils' using 'apt-get', a prerequisite for the 'pdf2image' Python package, which is used for converting PDF pages into images.

```bash
!sudo apt-get update && sudo apt-get install poppler-utils -y
```

--------------------------------

### Example of Multiple Relevant Documents per Query

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/evaluating-vespa-application-cloud.ipynb

This snippet provides an example dictionary structure for `qrels` where a query can have multiple relevant documents. While the NanoMSMARCO dataset used in this guide has only one relevant document per query, this demonstrates how to represent scenarios with multiple relevant documents for evaluation.

```python
# multiple relevant docs per query
qrels = {
    "q1": {"doc1", "doc2"},
    "q2": {"doc3", "doc4"},
    # etc. 
}
```

--------------------------------

### Install Python Dependencies for AI Assistant

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/scaling-personal-ai-assistants-with-streaming-mode-cloud.ipynb

This command installs the necessary Python libraries required to build retrieval-augmented personal AI assistants. It includes `pyvespa` for Vespa integration, `llama-index` for LLM application development, and `vespacli` for Vespa command-line utilities.

```python
!pip3 install -U pyvespa llama-index vespacli
```

--------------------------------

### Install pyvespa and Cohere dependencies

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/billion-scale-vector-search-with-cohere-embeddings-cloud.ipynb

Installs the necessary Python packages, including pyvespa, cohere (version 4.57), and vespacli, required to run the examples and interact with Vespa and Cohere APIs.

```python
!pip3 install -U pyvespa cohere==4.57 vespacli
```

--------------------------------

### Check Installed pyvespa Version

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/troubleshooting.md

Use this command to verify the currently installed version of pyvespa, which is crucial for compatibility and troubleshooting.

```bash
python3 -m pip show pyvespa
```

--------------------------------

### Deploy Vespa Application from Disk using Docker

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/lightgbm-with-categorical-mapping.ipynb

Deploys a Vespa application package, specifically 'lightgbm', from a local directory using the pyvespa Docker integration. This initializes and starts the Vespa instance, preparing it for data feeding and querying.

```python
from vespa.deployment import VespaDocker

vespa_docker = VespaDocker()
app = vespa_docker.deploy_from_disk(
    application_name="lightgbm", application_root="lightgbm"
)
```

--------------------------------

### Install pyvespa and evaluation dependencies

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/evaluating-vespa-application-cloud.ipynb

Instructions to install pyvespa, vespacli, datasets, and pandas using pip3. These are essential Python packages and CLI tools required for setting up and running the Vespa application evaluation environment.

```python
!pip3 install pyvespa vespacli datasets pandas
```

--------------------------------

### Install Python Dependencies for Vespa, LangChain, and PDF Processing

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/chat_with_your_pdfs_using_colbert_langchain_and_Vespa-cloud.ipynb

Installs necessary Python packages including pyvespa, langchain, langchain-community, langchain-openai, pypdf, openai, and vespacli to set up the environment for building RAG applications with Vespa and LangChain.

```python
!pip3 install -U pyvespa langchain langchain-community langchain-openai pypdf==5.0.1 openai vespacli
```

--------------------------------

### Deploy Vespa Application with pyvespa and Docker

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa.ipynb

This snippet demonstrates how to deploy a Vespa application package on a local machine using `VespaDocker`. It initializes `VespaDocker` to connect to the local Docker daemon and deploys the specified application package, returning a `Vespa` instance for further interaction.

```python
from vespa.deployment import VespaDocker

vespa_docker = VespaDocker()
app = vespa_docker.deploy(application_package=package)
```

--------------------------------

### Install Required Python Libraries

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/video_search_twelvelabs_cloud.ipynb

Installs necessary Python packages including `pyvespa`, `vespacli`, `twelvelabs`, and `pandas`. These libraries are essential for interacting with Vespa, TwelveLabs API, and for data manipulation.

```python
!pip3 install pyvespa vespacli twelvelabs pandas
```

--------------------------------

### Install Required Python Packages

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/lightgbm-with-categorical-mapping.ipynb

Installs the necessary Python libraries including numpy, pandas, pyvespa, and lightgbm for data manipulation, Vespa integration, and LightGBM model training.

```python
!pip3 install numpy pandas pyvespa lightgbm
```

--------------------------------

### Initialize PersonalAssistantVespaRetriever with default rank profile

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/scaling-personal-ai-assistants-with-streaming-mode-cloud.ipynb

Demonstrates initializing the `PersonalAssistantVespaRetriever` with the default Vespa rank profile and performing a simple retrieval query. This setup uses the standard ranking configured in the Vespa application.

```python
retriever = PersonalAssistantVespaRetriever(
    app=app, user="bergum@vespa.ai", vespa_rank_profile="default"
)
retriever.retrieve("When is my dentist appointment?")
```

--------------------------------

### Install pyvespa Python Package

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/index.rst

Instructions to install the pyvespa library using pip, the Python package installer. This command ensures the necessary dependencies for interacting with Vespa are available.

```bash
$ python3 -m pip install pyvespa
```

--------------------------------

### Example of Printing Another PDF Page Text (Commented)

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/visual_pdf_rag_with_vespa_colpali_cloud.ipynb

This is a commented-out line demonstrating how to access and print the text from a different page (at index 95) within the `pdf_pages` list. It serves as an example for further inspection but is not actively executed in the provided context.

```python
# print(pdf_pages[95]["text"])
```

--------------------------------

### Install Python Dependencies for Vespa and Cohere

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/multilingual-multi-vector-reps-with-cohere-cloud.ipynb

This command installs the necessary Python libraries for interacting with Vespa, Cohere, and handling datasets. It includes `pyvespa` for Vespa integration, `cohere` for embedding models, `datasets` for data loading, and `vespacli` for Vespa command-line tools.

```python
!pip3 install -U pyvespa cohere==4.57 datasets vespacli
```

--------------------------------

### Install Python Packages for LightGBM and PyVespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/lightgbm-with-categorical.ipynb

Installs essential Python libraries including numpy, pandas, pyvespa, and lightgbm using pip, required for data manipulation, model training, and Vespa interaction.

```bash
!pip3 install numpy pandas pyvespa lightgbm
```

--------------------------------

### Instantiate VespaStreamingColBERTRetriever

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/chat_with_your_pdfs_using_colbert_langchain_and_Vespa-cloud.ipynb

Demonstrates how to initialize an instance of the `VespaStreamingColBERTRetriever` class. This example sets up the retriever with a Vespa application object, a specific user ID, and parameters for the number of pages and chunks to retrieve.

```python
vespa_hybrid_retriever = VespaStreamingColBERTRetriever(
    app=app, user="jo-bergum", pages=1, chunks_per_page=3
)
```

--------------------------------

### Install Python Project Dependencies

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/visual_pdf_rag_with_vespa_colpali_cloud.ipynb

Installs all necessary Python packages for the Visual PDF RAG demo, including 'colpali-engine' for embeddings, 'pdf2image' and 'pypdf' for PDF handling, 'pyvespa' for Vespa interaction, and 'google-generativeai' for AI model integration.

```bash
!pip3 install colpali-engine==0.3.10 pdf2image pypdf==5.0.1 pyvespa>=0.50.0 vespacli numpy==1.26.4 pillow==10.4.0 google-generativeai==0.8.3 transformers python-dotenv
```

--------------------------------

### Install Python Dependencies for ColBERT and PyVespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/colbert_standalone_Vespa-cloud.ipynb

Installs necessary Python packages including `pyvespa`, `colbert-ai`, `numpy`, `torch`, and `transformers` to set up the environment for running ColBERT with Vespa.

```python
!pip3 install -U pyvespa colbert-ai numpy torch transformers<=4.49.0
```

--------------------------------

### Install Python Dependencies for ColPali and Vespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/colpali-benchmark-vqa-vlm_Vespa-cloud.ipynb

This command installs all necessary Python packages, including `transformers`, `accelerate`, `pyvespa`, `vespacli`, `requests`, `numpy`, `scipy`, `ir_measures`, `pillow`, and `datasets`, required for running ColPali ranking experiments with Vespa.

```python
!pip3 install transformers==4.51.3 accelerate pyvespa vespacli requests numpy scipy ir_measures pillow datasets
```

--------------------------------

### Install Poppler Utilities for PDF Processing

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/colpali-document-retrieval-vision-language-models-cloud.ipynb

This command installs `poppler-utils` on a Debian/Ubuntu-based system. `poppler-utils` is a crucial dependency for the `pdf2image` Python package, enabling the conversion of PDF pages into images for subsequent visual processing and embedding generation.

```shell
!sudo apt-get update && sudo apt-get install poppler-utils -y
```

--------------------------------

### Establish Vespa Dataplane Connection with mTLS (Python)

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

Shows how to set up a `requests` session for secure communication with the Vespa dataplane. This involves configuring the session with a mTLS certificate and key pair, typically generated during Vespa Cloud deployment.

```python
import requests

cert_path = app.cert
key_path = app.key
session = requests.Session()
session.cert = (cert_path, key_path)
```

--------------------------------

### Install Python Dependencies for RAG with Vespa and LangChain

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/turbocharge-rag-with-langchain-and-vespa-streaming-mode-cloud.ipynb

This command installs all necessary Python libraries required to run the RAG application. It includes pyvespa for Vespa integration, langchain and langchain-community for building the RAG pipeline, pypdf for PDF document loading, and openai for potential OpenAI model interactions.

```python
!pip3 install -U pyvespa langchain langchain-community pypdf==5.0.1 openai
```

--------------------------------

### Deploy Application to Vespa Cloud Dev Zone

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

This code snippet shows how to deploy the application package to the Vespa Cloud development zone. The first deployment typically takes a few minutes as Vespa Cloud sets up the necessary resources. The `app` variable will hold a reference to the deployed Vespa instance.

```python
app = vespa_cloud.deploy()
```

--------------------------------

### Install pyvespa library

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/multi-vector-indexing.ipynb

Installs the pyvespa Python library using pip3. This is a prerequisite for running the Vespa application steps described in the guide, ensuring all necessary dependencies are met.

```python
!pip3 install pyvespa
```

--------------------------------

### Example Vespa services.xml Configuration

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/advanced-configuration.ipynb

Illustrates the equivalent XML representation of the `ServicesConfiguration` object defined in Python, showing the structure of `services`, `container`, and `content` tags with their attributes and nested elements, including document expiry and garbage collection.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<services>
  <container id="music_container" version="1.0">
    <search></search>
    <document-api></document-api>
    <document-processing></document-processing>
  </container>
  <content id="music_content" version="1.0">
    <redundancy>1</redundancy>
    <documents garbage-collection="true">
      <document type="music" mode="index" selection="music.timestamp &gt; now() - 86400"></document>
    </documents>
    <nodes>
      <node distribution-key="0" hostalias="node1"></node>
    </nodes>
  </content>
</services>
```

--------------------------------

### Deploy Vespa Application with pyvespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/query.ipynb

Initializes and deploys a sample Vespa application using `VespaDocker` and `sample_package` from pyvespa. This establishes a local Vespa instance and a connection to it, preparing for data feeding and querying.

```python
from vespa.application import Vespa
from vespa.deployment import VespaDocker
from vespa.io import VespaQueryResponse
from vespa.exceptions import VespaError
from vespa.package import sample_package

vespa_docker = VespaDocker()
app: Vespa = vespa_docker.deploy(sample_package)
```

--------------------------------

### Install Python Dependencies for ColBERT and Vespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/colbert_standalone_long_context_Vespa-cloud.ipynb

This command installs all necessary Python packages, including `pyvespa`, `colbert-ai`, `numpy`, `torch`, `vespacli`, and `transformers`, required for running the ColBERT and Vespa integration examples.

```python
!pip3 install -U pyvespa colbert-ai numpy torch vespacli transformers<=4.49.0
```

--------------------------------

### Initialize VespaCloud Client for Deployment

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/video_search_twelvelabs_cloud.ipynb

This snippet initializes the `VespaCloud` object, which is essential for interacting with Vespa Cloud. It requires tenant, application, and package details, along with an API key for authentication, preparing the environment for application deployment.

```python
vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=application,
    application_package=package,
    key_content=os.getenv("VESPA_TEAM_API_KEY", None),
)
```

--------------------------------

### Install Python Dependencies for Vespa and Sentence-Transformers

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mixedbread-binary-embeddings-with-sentence-transformers-cloud.ipynb

This command installs the necessary Python libraries: `pyvespa` for Vespa integration, `sentence-transformers` for embedding models, and `vespacli` for Vespa command-line tools. It ensures all packages are up-to-date, preparing the environment for the subsequent code examples.

```python
!pip3 install -U pyvespa sentence-transformers vespacli
```

--------------------------------

### Initialize PersonalAssistantVespaRetriever with fusion rank profile and multiple sources

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/scaling-personal-ai-assistants-with-streaming-mode-cloud.ipynb

Illustrates setting up the retriever with the 'fusion' rank profile, which combines results from multiple sources. This example specifies 'calendar' and 'mail' as data sources and applies a `vespa_score_cutoff` for combined relevance filtering.

```python
retriever = PersonalAssistantVespaRetriever(
    app=app,
    user="bergum@vespa.ai",
    vespa_rank_profile="fusion",
    sources=["calendar", "mail"],
    vespa_score_cutoff=0.80,
)
retriever.retrieve("When is my dentist appointment?")
```

--------------------------------

### Feed Sample Documents and Perform Queries in Vespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/advanced-configuration.ipynb

This section shows how to prepare sample documents, feed them into the deployed Vespa application, define various YQL queries, and execute them to compare query performance with different thread configurations. It includes defining query bodies, warming up the system, and analyzing query times.

```python
sample_docs = [
    {"id": i, "fields": {"text": text}}
    for i, text in enumerate(
        [
            "'To Kill a Mockingbird' is a novel by Harper Lee published in 1960. It was immediately successful, winning the Pulitzer Prize, and has become a classic of modern American literature. The novel 'Moby-Dick' was written by Herman Melville and first published in 1851. Harper Lee, an American novelist widely known for her novel 'To Kill a Mockingbird'. It is considered a masterpiece of American literature and deals with complex themes of obsession, revenge, and the conflict between good and evil.",
            "was born in 1926 in Monroeville, Alabama. She received the Pulitzer Prize for Fiction in 1961. Jane Austen was an English novelist known primarily for her six major novels, ",
            "which interpret, critique and comment upon the British landed gentry at the end of the 18th century. The 'Harry Potter' series, which consists of seven fantasy novels written by British author J.K. Rowling, ",
            "is among the most popular and critically acclaimed books of the modern era. 'The Great Gatsby', a novel written by American author F. Scott Fitzgerald, was published in 1925. The story is set in the Jazz Age and follows the life of millionaire Jay Gatsby and his pursuit of Daisy Buchanan."
        ]
    )
]
app.feed_iterable(sample_docs, schema="doc")

# Define the query body
query_body = {
    "yql": "select * from sources * where userQuery();",
    "query": "who wrote to kill a mockingbird?",
    "timeout": "10s",
    "input.query(q)": "embed(tokenizer, @query)",
    "presentation.timing": "true"
}

# Warm-up query
app.query(body=query_body)
query_body_reranking = {
    **query_body,
    "ranking.profile": "reranking"
}
# Query with default persearch threads (set to 4)
with app.syncio() as sess:
    response_default = app.query(body=query_body_reranking)

# Query with num-threads-per-search overridden to 1
query_body_one_thread = {
    **query_body,
    "ranking.profile": "one-thread-profile",
    # "ranking.matching.numThreadsPerSearch": 1, Could potentiall also set numThreadsPerSearch in query parameters.
}
with app.syncio() as sess:
    response_one_thread = sess.query(body=query_body_one_thread)

# Extract query times
timing_default = response_default.json["timing"]["querytime"]
timing_one_thread = response_one_thread.json["timing"]["querytime"]
# Beautifully formatted statement of - num threads and ratio of query times
print(f"Query time with 4 threads: {timing_default:.2f}s")
print(f"Query time with 1 thread: {timing_one_thread:.2f}s")
ratio = timing_one_thread / timing_default
print(f"4 threads is approximately {ratio:.2f}x faster than 1 thread")
```

--------------------------------

### Install pyvespa and Check Docker Memory

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa.ipynb

This code block installs the `pyvespa` library using pip and then checks the total memory available to Docker. It's a prerequisite step to ensure the environment is set up correctly for running Vespa applications, specifically validating that Docker has at least 6GB of memory.

```python
!pip3 install pyvespa
!docker info | grep "Total Memory"
```

--------------------------------

### Perform Filtered Hybrid Search with Vespa (Python)

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

Illustrates how to integrate a filter into a hybrid search query in pyvespa. This example restricts the nearest neighbor search to only consider documents that contain a specific keyword (e.g., 'vegetable') in their title.

```python
with app.syncio(connections=1) as session:
    query = "How Fruits and Vegetables Can Treat Asthma?"
    response: VespaQueryResponse = session.query(
        yql='select * from sources * where title contains "vegetable" and rank({targetHits:1000}nearestNeighbor(embedding,q), userQuery()) limit 5',
        query=query,
        ranking="fusion",
        body={"input.query(q)": f"embed({query})"},
    )
    assert response.is_successful()
    print(display_hits_as_df(response, ["id", "title"]))
```

--------------------------------

### Example Vespa Disk Space Warning Log Message

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/troubleshooting.md

This log snippet illustrates a common warning message from Vespa indicating that write operations are blocked due to disk space reaching its limit. It provides details on capacity, used space, and the disk limit.

```log
WARNING searchnode
proton.proton.server.disk_mem_usage_filter   Write operations are now blocked:
'diskLimitReached: { action: "add more content nodes",
reason: "disk used (0.939172) > disk limit (0.9)",
stats: { capacity: 50406772736, used: 47340617728, diskUsed: 0.939172, diskLimit: 0.9}}'
```

--------------------------------

### Deploy Application to Vespa Cloud

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/video_search_twelvelabs_cloud.ipynb

This code snippet executes the deployment of the configured application package to Vespa Cloud. It uses the `deploy()` method on the initialized `VespaCloud` instance, making the application live and accessible.

```python
app = vespa_cloud.deploy()
```

--------------------------------

### Perform Filtered Hybrid Search in pyvespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa.ipynb

This example extends hybrid search by adding a filter to the YQL query, restricting nearest neighbor results to only consider documents that contain a specific term (e.g., 'vegetable') in their title. It combines filtering with the RANK operator for hybrid retrieval and fusion ranking.

```python
with app.syncio(connections=1) as session:
    query = "How Fruits and Vegetables Can Treat Asthma?"
    response: VespaQueryResponse = session.query(
        yql='select * from sources * where title contains "vegetable" and rank({targetHits:1000}nearestNeighbor(embedding,q), userQuery()) limit 5',
        query=query,
        ranking="fusion",
        body={"input.query(q)": f"embed({query})"},
    )
    assert response.is_successful()
    print(display_hits_as_df(response, ["id", "title"]))
```

--------------------------------

### Prepare Hugging Face Dataset for Vespa Feeding

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

This example demonstrates how to load a streaming dataset from Hugging Face (BeIR/nfcorpus) and transform it into the format expected by `pyvespa` for feeding. The `map` function converts dataset fields into a dictionary with 'id' and 'fields' keys, suitable for Vespa documents.

```python
from datasets import load_dataset

dataset = load_dataset("BeIR/nfcorpus", "corpus", split="corpus", streaming=True)
vespa_feed = dataset.map(
    lambda x: {
        "id": x["_id"],
        "fields": {"title": x["title"], "body": x["text"], "id": x["_id"]},
    }
)
```

--------------------------------

### Deploy Vespa Application to Cloud in Python

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/chat_with_your_pdfs_using_colbert_langchain_and_Vespa-cloud.ipynb

This Python code demonstrates how to deploy a configured Vespa application to the Vespa Cloud using the initialized `VespaCloud` client. It shows the final step in pushing the application to the cloud environment, typically taking a few minutes for the first deployment.

```python
from vespa.application import Vespa

app: Vespa = vespa_cloud.deploy()
```

--------------------------------

### Execute Hybrid Search using OR Operator in pyvespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa.ipynb

This example shows how to combine semantic and sparse keyword search using the logical OR operator in Vespa's YQL. It leverages `userQuery()` for keyword search and `nearestNeighbor` for semantic search, with results re-ranked by 'fusion' which includes cross-hits feature normalization and reciprocal rank fusion.

```python
with app.syncio(connections=1) as session:
    query = "How Fruits and Vegetables Can Treat Asthma?"
    response: VespaQueryResponse = session.query(
        yql="select * from sources * where userQuery() or ({targetHits:1000}nearestNeighbor(embedding,q)) limit 5",
        query=query,
        ranking="fusion",
        body={"input.query(q)": f"embed({query})"},
    )
    assert response.is_successful()
    print(display_hits_as_df(response, ["id", "title"]))
```

--------------------------------

### Initialize Vespa Cloud Deployment Client in Python

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/chat_with_your_pdfs_using_colbert_langchain_and_Vespa-cloud.ipynb

This Python snippet initializes the `VespaCloud` client, preparing for application deployment. It requires the tenant name and optionally an API key for CI/CD environments, demonstrating how to set up the connection to Vespa Cloud. The API key is parsed to handle potential newline characters.

```python
from vespa.deployment import VespaCloud
import os

# Replace with your tenant name from the Vespa Cloud Console
tenant_name = "vespa-team"

# Key is only used for CI/CD. Can be removed if logging in interactively
key = os.getenv("VESPA_TEAM_API_KEY", None)
if key is not None:
    key = key.replace(r"\n", "\n")  # To parse key correctly

vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=vespa_app_name,
    key_content=key,  # Key is only used for CI/CD. Can be removed if logging in interactively
    application_package=vespa_application_package,
)
```

--------------------------------

### Initialize VespaCloud Client for Deployment

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/scaling-personal-ai-assistants-with-streaming-mode-cloud.ipynb

This snippet demonstrates how to initialize the `VespaCloud` client in pyvespa, setting up the connection to a specific Vespa Cloud tenant. It shows how to configure the tenant name, application name, and optionally an API key for CI/CD environments, ensuring proper parsing of the key.

```python
from vespa.deployment import VespaCloud

# Replace with your tenant name from the Vespa Cloud Console
tenant_name = "vespa-team"

# Key is only used for CI/CD. Can be removed if logging in interactively
key = os.getenv("VESPA_TEAM_API_KEY", None)
if key is not None:
    key = key.replace(r"\n", "\n")  # To parse key correctly

vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=vespa_app_name,
    key_content=key,  # Key is only used for CI/CD. Can be removed if logging in interactively
    application_package=vespa_application_package,
)
```

--------------------------------

### Initialize VespaCloud Client for Deployment

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/multilingual-multi-vector-reps-with-cohere-cloud.ipynb

This snippet initializes the `VespaCloud` client object, which is essential for interacting with Vespa Cloud. It requires the tenant name, application name, and an application package. An optional API key can be provided for CI/CD environments, which is parsed to handle newline characters correctly.

```python
from vespa.deployment import VespaCloud
import os

# Replace with your tenant name from the Vespa Cloud Console
tenant_name = "vespa-team"

# Key is only used for CI/CD. Can be removed if logging in interactively
key = os.getenv("VESPA_TEAM_API_KEY", None)
if key is not None:
    key = key.replace(r"\n", "\n")  # To parse key correctly

vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=vespa_app_name,
    key_content=key,  # Key is only used for CI/CD. Can be removed if logging in interactively
    application_package=vespa_application_package,
)
```

--------------------------------

### Install pyvespa Python Package

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/index.md

Installs the `pyvespa` library using `pip`, the Python package installer. This command ensures that the necessary Python API for interacting with Vespa is available in your environment.

```bash
python3 -m pip install pyvespa
```

--------------------------------

### Initialize VespaCloud Client for Deployment

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/turbocharge-rag-with-langchain-and-vespa-streaming-mode-cloud.ipynb

This Python snippet demonstrates how to initialize the `VespaCloud` client from the `pyvespa` library. It sets up the connection parameters, including the tenant name, application name (from `vespa_app_name`), and an optional API key (`key_content`) for CI/CD environments. The `application_package` variable is expected to hold the application definition.

```python
from vespa.deployment import VespaCloud
import os

# Replace with your tenant name from the Vespa Cloud Console
tenant_name = "vespa-team"

# Key is only used for CI/CD. Can be removed if logging in interactively
key = os.getenv("VESPA_TEAM_API_KEY", None)
if key is not None:
    key = key.replace(r"\n", "\n")  # To parse key correctly

vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=vespa_app_name,
    key_content=key,  # Key is only used for CI/CD. Can be removed if logging in interactively
    application_package=vespa_application_package,
)
```

--------------------------------

### Deploy Vespa Application using Docker

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/evaluating-with-snowflake-arctic-embed.ipynb

Initializes a Vespa Docker instance and deploys the defined application package. This step sets up the Vespa engine environment, making it ready for indexing documents and serving queries.

```python
from vespa.deployment import VespaDocker

vespa_docker = VespaDocker()
app = vespa_docker.deploy(app_package)
```

--------------------------------

### Combine Keyword and Semantic Search using Hybrid Fusion Ranking in Vespa

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

This example demonstrates a hybrid search strategy in Vespa, combining both keyword search (`userQuery()`) and semantic search (`nearestNeighbor`) using a logical OR operator within the YQL. It applies a `fusion` ranking profile and passes the embedded query vector in the request body, allowing for a comprehensive retrieval that leverages the strengths of both sparse and dense retrieval methods.

```python
with app.syncio(connections=1) as session:
    query = "How Fruits and Vegetables Can Treat Asthma?"
    response: VespaQueryResponse = session.query(
        yql="select * from sources * where userQuery() or ({targetHits:1000}nearestNeighbor(embedding,q)) limit 5",
        query=query,
        ranking="fusion",
        body={"input.query(q)": f"embed({query})"},
    )
    assert response.is_successful()
    print(display_hits_as_df(response, ["id", "title"]))
```

--------------------------------

### Install pyvespa Library

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/application-packages.ipynb

Installs the pyvespa library using pip, which is a prerequisite for programmatically interacting with Vespa application packages and deployments.

```python
!pip3 install pyvespa
```

--------------------------------

### Configure Vespa Cloud Application Details

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/getting-started-pyvespa-cloud.ipynb

This snippet defines variables for the Vespa Cloud tenant name and the application name. These values are crucial for connecting to and deploying applications on Vespa Cloud. Users must replace the placeholder values with their actual tenant and desired application names.

```python
# Replace with your tenant name from the Vespa Cloud Console
tenant_name = "vespa-team"
# Replace with your application name (does not need to exist yet)
application = "hybridsearch"
```

--------------------------------

### Install pyvespa and Vespa CLI

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/authenticating-to-vespa-cloud.ipynb

This command installs the necessary Python library `pyvespa` (version >= 0.45) and the `vespacli` command-line interface, which are prerequisites for interacting with Vespa Cloud.

```python
!pip3 install pyvespa vespacli
```

--------------------------------

### Export Vespa Application Package to Disk

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/lightgbm-with-categorical.ipynb

Creates a directory named 'lightgbm' and exports all necessary files for the Vespa application package into it, preparing the application for local deployment.

```python
from pathlib import Path

Path("lightgbm").mkdir(parents=True, exist_ok=True)
app_package.to_files("lightgbm")
```

--------------------------------

### Initialize Vespa Cloud Deployment Client

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/cohere-binary-vectors-in-vespa-cloud.ipynb

This code demonstrates how to initialize the `VespaCloud` client for deploying applications. It requires the tenant name, application name, and optionally an API key for non-interactive CI/CD environments. The `application_package` created earlier is passed to this client.

```python
from vespa.deployment import VespaCloud
import os

# Replace with your tenant name from the Vespa Cloud Console
tenant_name = "vespa-team"

# Key is only used for CI/CD. Can be removed if logging in interactively
key = os.getenv("VESPA_TEAM_API_KEY", None)
if key is not None:
    key = key.replace(r"\n", "\n")  # To parse key correctly

vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=vespa_app_name,
    key_content=key,  # Key is only used for CI/CD. Can be removed if logging in interactively
    application_package=vespa_application_package,
)
```

--------------------------------

### Clone Vespa Sample Web Application

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/visual_pdf_rag_with_vespa_colpali_cloud.ipynb

Uses `git clone` with sparse checkout to download only the `visual-retrieval-colpali` directory from the `vespa-engine/sample-apps` repository, preparing for web app deployment.

```bash
!git clone --depth 1 --filter=blob:none --sparse https://github.com/vespa-engine/sample-apps.git src && cd src && git sparse-checkout set visual-retrieval-colpali
```

--------------------------------

### Deploy Vespa Application to Cloud

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mixedbread-binary-embeddings-with-sentence-transformers-cloud.ipynb

This final snippet demonstrates the simple yet crucial step of deploying the configured Vespa application package to the Vespa Cloud development zone. The `deploy()` method of the `VespaCloud` object initiates the deployment process, typically taking a few minutes for the endpoint to become active.

```python
from vespa.application import Vespa

app: Vespa = vespa_cloud.deploy()
```

--------------------------------

### Install pyvespa with development dependencies

Source: https://github.com/vespa-engine/pyvespa/blob/master/README.md

This command installs an editable version of the pyvespa library along with its development dependencies. It sets up the environment for linting and formatting using Ruff, which is also integrated via a pre-commit hook.

```Python
uv sync --extra dev
```

--------------------------------

### Import Python Packages

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/video_search_twelvelabs_cloud.ipynb

Imports all required Python modules for defining Vespa application schemas, deploying to Vespa Cloud, handling Vespa I/O, interacting with the TwelveLabs API, processing data with pandas, and managing timestamps.

```python
import os
import hashlib
import json

from vespa.package import (
    ApplicationPackage,
    Field,
    Schema,
    Document,
    HNSW,
    RankProfile,
    FieldSet,
    SecondPhaseRanking,
    Function,
)

from vespa.deployment import VespaCloud
from vespa.io import VespaResponse, VespaQueryResponse

from twelvelabs import TwelveLabs
from twelvelabs.models.embed import EmbeddingsTask

import pandas as pd

from datetime import datetime
```

--------------------------------

### Initialize Vespa Cloud Deployment Client

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mixedbread-binary-embeddings-with-sentence-transformers-cloud.ipynb

This Python code initializes the `VespaCloud` object, which is essential for deploying applications to Vespa Cloud. It requires the tenant name, application name, and optionally an API key for non-interactive deployments (e.g., CI/CD environments). The `application_package` created earlier is linked here.

```python
from vespa.deployment import VespaCloud
import os

# Replace with your tenant name from the Vespa Cloud Console
tenant_name = "vespa-team"

# Key is only used for CI/CD. Can be removed if logging in interactively
key = os.getenv("VESPA_TEAM_API_KEY", None)
if key is not None:
    key = key.replace(r"\n", "\n")  # To parse key correctly

vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=vespa_app_name,
    key_content=key,  # Key is only used for CI/CD. Can be removed if logging in interactively
    application_package=vespa_application_package,
)
```

--------------------------------

### Install Python Dependencies for Vespa and Cohere

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/cohere-binary-vectors-in-vespa-cloud.ipynb

Installs the necessary Python packages including `pyvespa`, `cohere` (specific version 4.57), and `vespacli`. These libraries are essential for interacting with Vespa and Cohere APIs in a Python environment.

```python
!pip3 install -U pyvespa cohere==4.57 vespacli
```

--------------------------------

### Install Vespa CLI using pip

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/colbert_standalone_Vespa-cloud.ipynb

This command installs the `vespacli` Python package using `pip3`. The Vespa CLI is essential for interacting with Vespa instances, including deployment to Vespa Cloud and managing applications.

```python
!pip3 install vespacli
```

--------------------------------

### Clone pyvespa Repository and Launch Jupyter Notebook

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/index.md

Clones the `pyvespa` GitHub repository with a shallow clone (`--depth 1`) to save space, then launches a Jupyter Notebook server. The notebook server is configured to serve files from the `pyvespa/docs/sphinx/source` directory, allowing users to run documentation notebooks directly.

```bash
git clone --depth 1 https://github.com/vespa-engine/pyvespa.git
jupyter notebook --notebook-dir pyvespa/docs/sphinx/source
```

--------------------------------

### Retrieve Document Data with Short Tensor Format

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/pyvespa-examples.ipynb

This example demonstrates how to fetch a specific document from Vespa and format its tensor fields concisely using the 'format.tensors: short-value' parameter, which is a native Vespa document API parameter.

```python
from vespa.io import VespaResponse

response: VespaResponse = app.get_data(
    schema="neighbors", data_id=0, **{"format.tensors": "short-value"}
)
print(json.dumps(response.get_json(), indent=4))
```

--------------------------------

### Initialize Vespa with Data Plane Key/Certificate

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/feed_performance_cloud.ipynb

Illustrates direct initialization of a `Vespa` instance using an endpoint, tenant, and key/certificate files for data plane authentication. This is the recommended way to connect to an already running Vespa Cloud instance.

```python
from vespa.application import Vespa

app: Vespa = Vespa(
    url="https://my-endpoint.z.vespa-app.cloud",
    tenant="my-tenant",
    key_file="path/to/private-key.pem",
    cert_file="path/to/certificate.pem",
)
```

--------------------------------

### Install Python Dependencies for BGE-M3 Integration

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/mother-of-all-embedding-models-cloud.ipynb

This command installs the necessary Python packages: `pyvespa` for Vespa integration, `FlagEmbedding` for the BGE-M3 model, and `vespacli` for Vespa command-line tools. These dependencies are crucial for setting up the environment to work with BGE-M3 embeddings and Vespa.

```python
!pip3 install -U pyvespa FlagEmbedding vespacli
```

--------------------------------

### Execute Basic Vespa Query with Connection Pooling and Timing

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/query.ipynb

Demonstrates how to perform a basic Vespa query using `app.syncio` for efficient connection re-use. It also shows how to enable `presentation.timing` in the query body to include server-side processing times in the response, aiding in latency debugging.

```python
with app.syncio(connections=12) as session:
    response: VespaQueryResponse = session.query(
        hits=1,
        body={
            "yql": "select title, body from doc where userQuery()",
            "query": "Is statin use connected to breast cancer?",
            "ranking": "bm25",
            "presentation.timing": True,
        },
    )
    print(response.is_successful())
```

--------------------------------

### Install Python Dependencies for Vespa and OpenAI

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/examples/Matryoshka_embeddings_in_Vespa-cloud.ipynb

Installs necessary Python packages like `pyvespa`, `ir_datasets`, `openai`, `pytrec_eval`, and `vespacli` using pip. These libraries are essential for interacting with Vespa, handling datasets, and utilizing OpenAI's API.

```python
!pip3 install -U pyvespa ir_datasets openai pytrec_eval vespacli
```

--------------------------------

### Initialize VespaCloud Client for Deployment

Source: https://github.com/vespa-engine/pyvespa/blob/master/docs/sphinx/source/evaluating-vespa-application-cloud.ipynb

This code demonstrates how to create a VespaCloud object, which is essential for interacting with the Vespa Cloud deployment service. It requires specifying the tenant, application name, and an optional API key for authentication, especially useful in CI/CD environments.

```python
from vespa.deployment import VespaCloud
import os

# Key is only used for CI/CD. Can be removed if logging in interactively

vespa_cloud = VespaCloud(
    tenant=tenant_name,
    application=application,
    key_content=os.getenv(
        "VESPA_TEAM_API_KEY", None
    ),  # Key is only used for CI/CD. Can be removed if logging in interactively
    application_package=package,
)
```