Seldon LLM Module

https://docs.seldon.ai/llm-module
Admin
The Seldon LLM Module provides tools and components for integrating and managing Large Language...
Tokens:93,800
Snippets:602
Trust Score:4.8
Update:11 months ago
Show doc for...
Tokens:
Raw
### Install Required Python Packages

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

This command installs the necessary Python packages for the Seldon LLM module example, including 'requests', 'pg8000' for PostgreSQL interaction, and 'qdrant-client' for the Qdrant vector database. The '-q' flag ensures a quiet installation process.

```shell
!pip install requests pg8000==1.31.2 qdrant-client==1.12.0 -q
```

--------------------------------

### Start File System Runtime

Source: https://docs.seldon.ai/llm-module/components/memory/example

Starts the MLServer LLM Memory runtime using Docker, mounting a local directory for file system storage and exposing the necessary port. This command initializes the runtime for local development.

```sh
docker run -it --rm -p 8080:8080 \
  -v ${PWD}/filesys-memory:/filesys-memory \
  europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-llm-memory:0.7.0 \
  mlserver start /filesys-memory
```

--------------------------------

### Install Google Cloud Storage Client

Source: https://docs.seldon.ai/llm-module/use-cases/local

This command installs the necessary Python client library for interacting with Google Cloud Storage. The '-q' flag ensures quiet installation, suppressing verbose output.

```shell
!pip install google-cloud-storage -q
```

--------------------------------

### PostgreSQL Deployment and Service (Kubernetes YAML)

Source: https://docs.seldon.ai/llm-module/components/memory/example

This YAML defines a Kubernetes Deployment for a PostgreSQL instance and a corresponding Service to expose it within the cluster. It specifies the PostgreSQL image, environment variables for credentials and database name, and readiness probe configurations. This setup is intended for development and testing; production environments should use more secure credential management.

```yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:latest
        env:
        - name: POSTGRES_USER
          value: admin
        - name: POSTGRES_PASSWORD
          value: admin
        - name: POSTGRES_DB
          value: db
        ports:
        - containerPort: 5432
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "admin", "-d", "db"]
          initialDelaySeconds: 5
          periodSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  type: ClusterIP
  selector:
    app: postgres
  ports:
  - protocol: TCP
    port: 5432
    targetPort: 5432

```

--------------------------------

### Install aiohttp Package

Source: https://docs.seldon.ai/llm-module/use-cases/local

This command installs the `aiohttp` Python package, which is a library for building asynchronous HTTP clients and servers. The `-q` flag indicates a quiet installation.

```shell
!pip install aiohttp -q
```

--------------------------------

### Loan Model Configuration (JSON)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Defines the settings for the 'loan-model', including its implementation as an SKLearnModel and the path to the pre-trained joblib file.

```json
{
    "name": "loan-model",
    "implementation": "mlserver_sklearn.SKLearnModel",
    "parameters": {
        "uri": "./loan-predictor-rf.joblib",
        "version": "v0.1.0"
    }
}
```

--------------------------------

### Kubernetes Deployment Commands for LLM

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Shell commands to deploy the Mistral LLM to Kubernetes using its manifest file and then verify its readiness. This follows the deployment of the preprocessor.

```bash
!kubectl apply -f manifests/mistral.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon
```

--------------------------------

### Deploy Loan Model (Shell)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Applies the Kubernetes manifest for the loan model and waits for its deployment to become ready. This command assumes you have kubectl configured to interact with your cluster.

```shell
!kubectl apply -f manifests/loan-model.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon
```

--------------------------------

### Preprocessor Model Settings (JSON)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

JSON configuration file for the preprocessor model, specifying its name, implementation, and version. This file is used by MLServer to load and manage the model.

```json
{
    "name": "preprocessor",
    "implementation": "model.PreProcessorModel",
    "parameters": {
        "version": "v0.1.0"
    }
}
```

--------------------------------

### Display Qdrant Client Model Settings (Bash)

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

Displays the content of the model-settings.json file used for the Qdrant retrieval client. This file defines how the client connects to Qdrant and extracts information.

```bash
!cat models/qdrant-client/model-settings.json
```

--------------------------------

### Deploy Qdrant Client and Wait for Readiness (Bash)

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

Applies the Kubernetes manifest for the Qdrant client and then waits until the model is in a ready state. This ensures the retrieval client is operational before sending requests.

```bash
!kubectl apply -f manifests/qdrant-client.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon
```

--------------------------------

### Start Local MLServer LLM Runtime

Source: https://docs.seldon.ai/llm-module/components/models/local/chat

Starts the MLServer LLM local runtime Docker container, exposing port 8080 and mounting local models. It requires a Hugging Face token and specifies the model configuration path.

```sh
docker run -it --rm -p 8080:8080 \
  -v ${PWD}/models:/models \
  -e HF_TOKEN=<your_hf_token> \
  --shm-size=1g \
  europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-llm-local:0.7.0 \
  mlserver start /models/local-chat-completions
```

--------------------------------

### Define and Deploy Seldon Pipeline

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Defines a Seldon pipeline using a YAML manifest and deploys it using kubectl. Includes commands for applying the manifest and waiting for the pipeline to become ready.

```shell
!kubectl apply -f manifests/pipeline.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s pipeline --all -n seldon
```

--------------------------------

### Kubernetes Deployment Commands

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Shell commands to apply the preprocessor model manifest to a Kubernetes cluster and wait for the model to become ready. Assumes kubectl and Seldon CRDs are configured.

```bash
!kubectl apply -f manifests/preprocessor.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon
```

--------------------------------

### Deploy Pgvector Client and Wait for Readiness

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

This snippet demonstrates deploying the pgvector-client using kubectl and then waiting for the model to become ready. It ensures the necessary resources are available before proceeding with inference.

```bash
!kubectl apply -f manifests/pgvector-client.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon
```

--------------------------------

### Define User Prompt for LLM

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Defines the user prompt in Python, which is the actual text input for the LLM. This prompt contains the message from which information needs to be extracted.

```python
prompt = """Hi,
I am looking for a loan to buy a new car.
I am a software engineer and I'm employed by Google. I am married and have a wife and 2 kids. I have a house in Central London and My income is 50,000.
I would like a loan amount of 100,000 and I am looking for a term of 10 years. I'm a graduate.
Mr John Doe"""
```

--------------------------------

### Start Gemini LLM API Runtime Docker Container

Source: https://docs.seldon.ai/llm-module/components/models/api/gemini

Starts the LLM API runtime Docker container, mapping port 8080 and mounting the local models directory. It configures the container to use the Gemini provider by setting the MLSERVER_MODEL_GEMINI_API_KEY environment variable.

```shell
docker run -it --rm -p 8080:8080 \
  -v ${PWD}/models:/models \
  -e MLSERVER_MODEL_GEMINI_API_KEY=<your_openai_key> \
  europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-llm-api:0.7.0 \
  mlserver start /models
```

--------------------------------

### Deploy Qdrant Server and Check Status (Bash)

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

Deploys the Qdrant server to the Kubernetes cluster using a manifest file and then checks the rollout status of the deployment. This ensures the Qdrant instance is running before proceeding.

```bash
!kubectl apply -f manifests/qdrant-deployment.yaml -n seldon
!kubectl rollout status deployment/qdrant -n seldon --timeout=600s
```

--------------------------------

### Display Qdrant Client Kubernetes Manifest (Bash)

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

Displays the YAML content of the Kubernetes manifest file for the Qdrant client. This defines the model resource, including its storage URI and requirements.

```bash
!cat manifests/qdrant-client.yaml
```

--------------------------------

### Start Local Embeddings MLServer Runtime

Source: https://docs.seldon.ai/llm-module/components/embeddings/example

Starts the MLServer local embeddings runtime instance using Docker. This command mounts the local models directory and sets the Hugging Face token for model loading. It exposes port 8080 for inference requests.

```bash
docker run -it --rm -p 8080:8080 \
  -v ${PWD}/models:/models \
  -e HF_TOKEN=<your_hf_token> \
  --shm-size=1g \
  europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-local-embeddings:0.7.0 \
  mlserver start /models/embed
```

--------------------------------

### View Loan Model Settings (Python)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Displays the JSON configuration file for the loan Random Forest model. This file specifies the model's name, implementation type, and the URI to its trained artifact.

```python
!cat models/loan-model/model-settings.json
```

--------------------------------

### Preprocessor Model Manifest (YAML)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Kubernetes manifest file in YAML format for deploying the preprocessor model as a Seldon Model CRD. It defines the storage URI and requirements for the model.

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: preprocessor
spec:
  storageUri: "gs://seldon-models/llm-runtimes-settings/getting-started/quickstart/models/preprocessor"
  requirements:
  - mlserver
```

--------------------------------

### Construct and Send Inference Request

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Constructs a JSON payload for an inference request to a Seldon pipeline and sends it using the requests library. It includes the system prompt, user prompt, and specifies content types.

```python
import json
import requests

inference_request = {
    "inputs": [
        {
            "name": "role",
            "shape": [2],
            "datatype": "BYTES",
            "data": ["system", "user"],
            "parameters": {"content_type": "str"},
        },
        {
            "name": "content",
            "shape": [2],
            "datatype": "BYTES",
            "data": [
                json.dumps([system_prompt]),
                json.dumps([prompt]),
            ],
            "parameters": {"content_type": "str"},
        },
        {
            "name": "type",
            "shape": [2],
            "datatype": "BYTES",
            "data": [
                json.dumps(["text"]),
                json.dumps(["text"]),
            ],
            "parameters": {"content_type": "str"},
        }
    ],
}

endpoint = f"http://{get_mesh_ip()}/v2/pipelines/entity-extraction/infer"
response = requests.post(endpoint, json=inference_request)
```

--------------------------------

### Model Settings for Conversational Memory

Source: https://docs.seldon.ai/llm-module/components/memory/example

Defines the configuration for the Conversational Memory runtime, specifying the implementation and parameters for the file system database backend. This JSON file dictates how the runtime should operate.

```json
{
  "name": "filesys-memory",
  "implementation": "mlserver_memory.ConversationalMemory",
  "parameters": {
    "extra": {
      "database": "filesys",
      "config": {
        "window_size": 50,
        "tensor_names": ["role", "content", "type"]
      }
    }
  }
}
```

--------------------------------

### Mistral LLM Manifest (YAML)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Kubernetes manifest for deploying the Mistral LLM using Seldon. It specifies the storage URI pointing to the model artifacts and lists 'llm-local' as a requirement.

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: mistral
spec:
  storageUri: "gs://seldon-models/llm-runtimes-settings/getting-started/quickstart/models/mistral"
  requirements:
  - llm-local
```

--------------------------------

### Display Inference Results

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Prints the prediction and extracted data from the Seldon pipeline's inference response. The output is accessed from the JSON response object.

```python
print("Prediction:", response.json()['outputs'][0]['data'][0])
print("Data:", response.json()['outputs'][1]['data'][0])
```

--------------------------------

### Apply Local Runtime Manifest using Kubectl

Source: https://docs.seldon.ai/llm-module/introduction/installation

This shell command applies the server-local.yaml manifest file to the 'seldon' namespace, deploying the Local Runtime server with GPU support.

```shell
kubectl apply -f server-local.yaml -n seldon
```

--------------------------------

### Define System Prompt for LLM

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Defines a system prompt in Python for an LLM, specifying its role as a data entry assistant. It outlines the expected JSON output format and data validation rules for extracted fields.

```python
system_prompt = """\
You are a helpful data entry assistant whose responsibility is extracting data from a message sent by a user. The following is
such a message. Please extract the users details and return in a json dict with keys: \"ApplicantIncome\", \"LoanAmount\", 
\"Loan_Amount_Term\", \"Gender\", \"Married\", \"Dependents\", \"Education\", \"Self_Employed\", \"Property_Area\"

Please ensure that \"ApplicantIncome\" is an integer greater than 0
Please ensure that \"LoanAmount\" is an integer greater than 0
Please ensure that \"Loan_Amount_Term\" is an integer value corresponding to number of days, do not provide an expression.
Please ensure that \"Gender\" is either \"Male\" or \"Female\"
Please ensure that \"Married\" is either \"No\" or \"Yes\"
Please ensure that \"Dependents\" is one of \"0\", \"1\", \"2\" or \"3+\" 
Please ensure that \"Education\" is one of \"Graduate\" or \"Not_Graduate\"
Please ensure that \"Self_Employed\" is one of \"Yes\" or \"No\"
Please ensure that \"Property_Area\" is one of \"Rural\", \"Semiurban\" or \"Urban\"

Please only return JSON do not add any other text! If values are missing set them to a string: \"none\"."""
```

--------------------------------

### Apply Prompt Runtime Configuration (Shell)

Source: https://docs.seldon.ai/llm-module/introduction/installation

This shell command applies the Prompt Runtime configuration defined in `server-prompt.yaml` to the 'seldon' namespace. This action deploys the Prompt Runtime onto your Kubernetes cluster.

```shell
kubectl apply -f server-prompt.yaml -n seldon
```

--------------------------------

### Clean Up Qdrant Resources (Bash)

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

Deletes the Qdrant client model and the Qdrant deployment from the Kubernetes cluster using kubectl. This command cleans up all resources created during the Qdrant integration process.

```bash
!kubectl delete -f manifests/qdrant-client.yaml -n seldon
!kubectl delete -f manifests/qdrant-deployment.yaml -n seldon
```

--------------------------------

### Python Preprocessor Model Logic

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Contains Python code for a preprocessor model, including JSON decoding, response building, and feature value extraction. It utilizes libraries like numpy and json for data manipulation.

```python
except JSONDecodeError as err:
            logger.info("error decoding json!")
            logger.info(request_data["content"][0])
            logger.info(err)
            data = {}
        logger.info(data)
        return data

    def _build_response(self, state):
        none_to_string = lambda x: x if x is not None else "none"
        data = {key: none_to_string(value) for key, value in state.items()}
        logger.info(data)
        data = self._preprocess_input(data)
        inference_response = InferenceResponse(
            model_name="preprocessor",
            outputs=[data, StringCodec.encode_output("data", [json.dumps(state)])],
        )
        return inference_response

    def _preprocess_input(self, vals):
        tensor = np.array([0] * len(feature_list))
        for ind, feature in enumerate(feature_list):
            data = self._get_feature_value(feature, vals)
            tensor[ind] = data
        return NumpyCodec.encode_output("predict", np.array([tensor]))

    def _get_feature_value(self, feature, vals):
        feature, *category = feature.split("-")
        if not category:
            value = vals[feature]
        else:
            value = 1 if (category.pop() == vals[feature]) else 0
        return value


feature_list = [
    "ApplicantIncome",
    "LoanAmount",
    "Loan_Amount_Term",
    "Gender-Female",
    "Gender-Male",
    "Married-No",
    "Married-Yes",
    "Dependents-0",
    "Dependents-1",
    "Dependents-2",
    "Dependents-3+",
    "Education-Graduate",
    "Education-Not Graduate",
    "Self_Employed-No",
    "Self_Employed-Yes",
    "Property_Area-Rural",
    "Property_Area-Semiurban",
    "Property_Area-Urban",
]
```

--------------------------------

### Remove Pgvector Client and PostgreSQL Deployment

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

This snippet shows the kubectl commands to remove the pgvector client model and the associated PostgreSQL deployment. This is a cleanup step to revert the changes made during the setup process.

```bash
!kubectl delete -f manifests/pgvector-client.yaml -n seldon
!kubectl delete -f manifests/pgvector-deployment.yaml -n seldon
```

--------------------------------

### Deploy Prompt Runtime Configuration (YAML)

Source: https://docs.seldon.ai/llm-module/introduction/installation

This YAML configuration defines the deployment of the Prompt Runtime using Seldon. It specifies server details, capabilities, and image pull secrets for the MLServer container. This configuration is essential for utilizing prompt-related functionalities.

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-prompt-utils
spec:
  serverConfig: mlserver
  capabilities:
  - prompt
  podSpec:
    imagePullSecrets:
    - name: seldon-registry
    containers:
    - name: mlserver
      image: europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-prompt-utils:0.7.0
      imagePullPolicy: Always
```

--------------------------------

### Retrieve Inference History using Python

Source: https://docs.seldon.ai/llm-module/components/memory/example

Fetches the entire conversation history by sending a GET request to the inference endpoint. It requires the memory ID and the endpoint URL. The response, a JSON object, is then pretty-printed.

```python
response = get_inference_request(memory_id=memory_id, endpoint=endpoint)
pprint.pprint(response.json())
```

--------------------------------

### Retrieve Conversation History from Seldon AI LLM Module (Python)

Source: https://docs.seldon.ai/llm-module/components/memory/example

This function retrieves the conversation history associated with a specific `memory_id` from the Seldon AI LLM module. It sends a GET request containing only the `memory_id` to the specified endpoint.

```python
import pprint
import requests

def get_inference_request(
    memory_id: str,
    endpoint: str = "http://localhost:8080/v2/models/filesys-memory/infer",
):
    inference_request = {
        "inputs": [
            {
                "name": "memory_id",
                "shape": [1],
                "datatype": "BYTES",
                "data": [memory_id],
                "parameters": {"content_type": "str"},
            }
        ]
    }
    return requests.post(endpoint, json=inference_request)
```

--------------------------------

### Python Function to Get Seldon Mesh IP (Python)

Source: https://docs.seldon.ai/llm-module/components/memory/example

This Python function retrieves the external IP address of the Seldon mesh service within a Kubernetes cluster. It uses `kubectl` to query the service's load balancer status. This IP is crucial for constructing the correct endpoint URL to send inference requests to the deployed Seldon models.

```python
import subprocess
from uuid import uuid4

def get_mesh_ip():
    cmd = f"kubectl get svc seldon-mesh -n seldon -o jsonpath='{{.status.loadBalancer.ingress[0].ip}}'"
    return subprocess.check_output(cmd, shell=True).decode('utf-8')

```

--------------------------------

### Python: Install ipywidgets

Source: https://docs.seldon.ai/llm-module/use-cases/chat-bot

Installs the ipywidgets library, which is a dependency for creating interactive user interfaces within Jupyter notebooks or similar environments.

```python
!pip install ipywidgets -q
```

--------------------------------

### Install httpx-sse for SSE Connections

Source: https://docs.seldon.ai/llm-module/components/models/streaming

Installs the 'httpx-sse' Python package using pip. This package is required for establishing Server-Sent Events (SSE) connections, which are used by MLServer for streaming responses.

```bash
!pip install httpx-sse -q
```

--------------------------------

### Install Pillow for Image Handling

Source: https://docs.seldon.ai/llm-module/components/models/api/gemini

This command installs the Pillow library, which is a dependency for image manipulation and serialization in Python. It is essential for the subsequent image processing functions.

```python
!pip install pillow -q
```

--------------------------------

### PGVector Metadata Filtering Examples (JSON)

Source: https://docs.seldon.ai/llm-module/components/retrieval

Examples demonstrating how to construct filtering formulas for PGVector using comparison and logical operators. These JSON structures define query conditions for metadata.

```json
{
    "id": {"lt": 3, "gt": 0}
}
```

```json
{
    "id": {"in": [1, 2, 3]}
}
```

```json
{
    "id": {"le": 3},
    "category": {"eq": "geography"}
}
```

```json
{
    "id": 1
}
```

```json
{
    "id": [1, 2, 3]
}
```

```json
{
    "name": null
}
```

```json
{
    "id": 1,
    "category": ["geography", "history"],
    "name": null
}
```

```json
{
    "OR": [
        {"name": "London"},
        {"category": "geography"}
    ],
    "id": [1, 2, 3]
}
```

--------------------------------

### Define Synthetic Data for Vector Database (Python)

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

This snippet defines lists of sentences, their associated sentiment, and dummy one-hot encoded embeddings. This data is intended to be used for populating a vector database within the Seldon LLM module example.

```Python
sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium.",
    "The weather is horrible today.",
    "It's so cloudy outside!",
    "She walked to the park.",
]

sentiment = [
    "positive",
    "positive",
    "neutral",
    "negative",
    "negative",
    "neutral",
]

embeddings = [
    [1.0, 0.0, 0.0],
    [0.0, 1.0, 0.0],
    [0.0, 0.0, 1.0],
    [1.0, 0.0, 0.0],
    [0.0, 1.0, 0.0],
    [0.0, 0.0, 1.0],
]
```

--------------------------------

### Mistral LLM Settings (JSON)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

JSON configuration for the Mistral LLM, detailing its name, implementation, and specific parameters for the vLLM backend. Includes model configuration like type, name, tensor parallelism, data type, and token limits.

```json
{
    "name": "mistral",
    "implementation": "mlserver_llm_local.runtime.Local",
    "parameters": {
        "extra": {
            "backend": "vllm",
            "config": {
                "model_type": "chat.completions",
                "model_settings": {
                    "model": "mistralai/Mistral-7B-Instruct-v0.2",
                    "tensor_parallel_size": 4,
                    "dtype": "float16",
                    "max_model_len": 4096,
                    "default_generate_kwargs": {
                        "max_tokens": 1024
                    }
                }
            }
        }
    }
}
```

--------------------------------

### Apply PGVector Deployment and Monitor Rollout using kubectl

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

These `kubectl` commands apply the PGVector deployment manifest to the cluster and then monitor the rollout status of the postgres deployment. This ensures the database is successfully deployed and ready for use. The `-n seldon` flag specifies the namespace for the deployment.

```bash
!kubectl apply -f manifests/pgvector-deployment.yaml -n seldon
!kubectl rollout status deployment/postgres -n seldon --timeout=600s

```

--------------------------------

### Kubernetes Deployment and Rollout Commands (Shell)

Source: https://docs.seldon.ai/llm-module/components/memory/example

These shell commands are used to deploy the PostgreSQL database and monitor its rollout status within a Kubernetes cluster. The `kubectl apply` command applies the PostgreSQL deployment configuration, and `kubectl rollout status` waits for the deployment to complete successfully, ensuring the database is ready before proceeding.

```shell
!kubectl apply -f manifests/psql-deployment.yaml -n seldon
!kubectl rollout status deployment/postgres -n seldon --timeout=600s

```

--------------------------------

### Example LLM Inference Response with Tool Call (JSON)

Source: https://docs.seldon.ai/llm-module/use-cases/function-calling/openai-func-call

This is an example JSON response from the Seldon AI LLM after an inference request. It indicates that the model has decided to call the 'get_weather' function with specific arguments ('location': 'Boston', 'unit': 'c'). The response also includes metadata about the model and usage statistics.

```json
{
    "id": "08ec4f69-9702-4e93-9126-3df5ada10542",
    "model_name": "openai-chat-completions_1",
    "model_version": "1",
    "outputs": [
        {
            "data": ["assistant"],
            "datatype": "BYTES",
            "name": "role",
            "parameters": {"content_type": "str"},
            "shape": [1, 1]
        },
        {
            "data": [""],
            "datatype": "BYTES",
            "name": "content",
            "parameters": {"content_type": "str"},
            "shape": [1, 1]
        },
        {
            "data": ["text"],
            "datatype": "BYTES",
            "name": "type",
            "parameters": {"content_type": "str"},
            "shape": [1, 1]
        },
        {
            "data": ["{\"id\":\"call_gUpJlV6hD7VMJBDr4QhMp042\",\"function\":{\"arguments\":\"{\\\"location\\\":\\\"Boston\\\",\\\"unit\\\":\\\"c\\\"}\",\"name\":\"get_weather\"},\"type\":\"function\"}"],
            "datatype": "BYTES",
            "name": "tool_calls",
            "parameters": {"content_type": "str"},
            "shape": [1, 1]
        },
        {
            "data": [
                "{\n  \"id\": \"chatcmpl-ANHzZJGR402hVSHXaOS7z3WrcPDlf\",\n  \"choices\": [\n    {\n      \"finish_reason\": \"tool_calls\",\n      \"index\": 0,\n      \"logprobs\": null,\n      \"message\": {\n        \"content\": null,\n        \"refusal\": null,\n        \"role\": \"assistant\",\n        \"tool_calls\": [\n          {\n            \"id\": \"call_gUpJlV6hD7VMJBDr4QhMp042\",\n            \"function\": {\n              \"arguments\": \"{\\\"location\\\":\\\"Boston\\\",\\\"unit\\\":\\\"c\\\"}\",\n              \"name\": \"get_weather\"\n            },\n            \"type\": \"function\"\n          }\n        ]\n      }\n    }\n  ],\n  \"created\": 1730114069,\n  \"model\": \"gpt-4o-2024-08-06\",\n  \"object\": \"chat.completion\",\n  \"system_fingerprint\": \"fp_90354628f2\",\n  \"usage\": {\n    \"completion_tokens\": 18,\n    \"prompt_tokens\": 71,\n    \"total_tokens\": 89,\n    "
            "datatype": "BYTES",
            "name": "response",
            "parameters": {"content_type": "str"},
            "shape": [1, 1]
        }
    ]
}
```

--------------------------------

### Get Seldon Mesh Service IP

Source: https://docs.seldon.ai/llm-module/use-cases/agentic-workflows/planning

Python function to retrieve the IP address of the 'seldon-mesh' service in the 'seldon' namespace using kubectl. This is used to determine the endpoint for sending requests.

```python
import requests
import subprocess

def get_mesh_ip():
    cmd = f"kubectl get svc seldon-mesh -n seldon -o jsonpath='{{.status.loadBalancer.ingress[0].ip}}'"
    return subprocess.check_output(cmd, shell=True).decode('utf-8')
```

--------------------------------

### Get Memory ID from MLServer Request

Source: https://docs.seldon.ai/llm-module/use-cases/agentic-workflows/planning

A Python function `get_memory_id` that retrieves and decodes the 'memory_id' from an MLServer `InferenceRequest`. It utilizes `extract_tensor_by_name` and `StringCodec` to locate and return the memory identifier as a string.

```python
import json
from typing import Any, Dict, Union, List

from mlserver.codecs import StringCodec
from mlserver.types import InferenceResponse, InferenceRequest, RequestInput, ResponseOutput

# Assuming extract_tensor_by_name is defined as above
def extract_tensor_by_name(
    tensors: Union[List[RequestInput], List[ResponseOutput]], name: str
) -> Optional[Union[RequestInput, ResponseOutput]]:
    for tensor in tensors:
        if tensor.name == name:
            return tensor

    return None

def get_memory_id(payload: InferenceRequest) -> str:
    memory_id = extract_tensor_by_name(payload.inputs, "memory_id")
    return StringCodec.decode_input(memory_id)[0]
```

--------------------------------

### Populate Qdrant Collection with Data (Python)

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

Initializes a Qdrant client, creates or deletes a collection, and then upserts data points into the collection. Each point includes a vector, an ID, and payload with text and sentiment. Assumes 'embeddings', 'sentences', and 'sentiment' are pre-defined lists.

```python
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
from qdrant_client.models import PointStruct

qdrant = QdrantClient(location=f"http://localhost:6333")
collection_name = "collection"

if collection_name in [c.name for c in qdrant.get_collections().collections]:
    qdrant.delete_collection(collection_name)

qdrant.create_collection(
    collection_name=collection_name,
    vectors_config=VectorParams(size=3, distance=Distance.COSINE),
)

qdrant.upsert(
    collection_name=collection_name,
    points=[
        PointStruct(
            id=i, 
            vector=embeddings[i], 
            payload={"text": sentences[i], "sentiment": sentiment[i]}
        )
        for i in range(len(sentences))
    ],
)

qdrant.close()
```

--------------------------------

### TailModel: Model Configuration (JSON)

Source: https://docs.seldon.ai/llm-module/use-cases/agentic-workflows/planning

This JSON file configures the TailModel, specifying its name as 'get-weather', its implementation class as 'model.TailModel', and its version. It's part of the MLServer deployment setup for the model.

```json
{
    "name": "get-weather",
    "implementation": "model.TailModel",
    "parameters": {
        "version": "v0.1.0"
    }
}
```

--------------------------------

### Kubernetes Model Application and Readiness Check (Shell)

Source: https://docs.seldon.ai/llm-module/components/memory/example

These shell commands are used to apply the Seldon model deployment for PostgreSQL memory and wait for it to become ready. `kubectl apply` deploys the model defined in `psql-memory.yaml`, and `kubectl wait` ensures that the model is in a ready state before any inference requests are sent, preventing potential errors.

```shell
!kubectl apply -f manifests/psql-memory.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon

```

--------------------------------

### Seldon Model Deployment for PostgreSQL Memory (YAML)

Source: https://docs.seldon.ai/llm-module/components/memory/example

This YAML defines a Seldon `Model` resource for deploying the conversational memory component. It specifies the `storageUri` where the model's configuration and potentially other artifacts are stored (e.g., in a Google bucket) and lists the required package (`memory`). This allows Kubernetes to manage and deploy the conversational memory service.

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: psql-memory
spec:
  storageUri: "gs://seldon-models/llm-runtimes-settings/memory/example/models/psql-memory"
  requirements:
  - memory

```

--------------------------------

### Apply VectorDB Runtime Manifest using Kubectl

Source: https://docs.seldon.ai/llm-module/introduction/installation

This shell command applies the server-vector-db.yaml manifest file to the 'seldon' namespace, deploying the VectorDB Runtime server.

```shell
kubectl apply -f server-vector-db.yaml -n seldon
```

--------------------------------

### Model Settings for Gemini Chat Completions

Source: https://docs.seldon.ai/llm-module/components/models/api/gemini

JSON configuration file for setting up the Gemini chat completions model, including specifying the prompt template URI using prompt_utils.

```json
{
    "name": "gemini-prompt-chat-completions",
    "implementation": "mlserver_llm_api.LLMRuntime",
    "parameters": {
      "extra": {
        "provider_id": "gemini",
        "config": {
          "model_id": "gemini-1.5-flash",
          "model_type": "chat.completions"
        },
        "prompt_utils": {
            "prompt_options": {
                "uri": "code.jinja"
            }
        }
      }
    }
  }
```

--------------------------------

### Delete Seldon Pipeline Resources

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Commands to delete the deployed Seldon pipeline and its associated models using kubectl. This cleans up the resources created during the deployment process.

```shell
!kubectl delete -f manifests/pipeline.yaml -n seldon
!kubectl delete -f manifests/mistral.yaml -n seldon
!kubectl delete -f manifests/preprocessor.yaml -n seldon
!kubectl delete -f manifests/loan-model.yaml -n seldon
```

--------------------------------

### Configure PGVector Client Settings for MLServer

Source: https://docs.seldon.ai/llm-module/components/retrieval/example

This JSON file configures the `mlserver_vector_db.VectorDBRuntime` for interacting with a PGVector database. It specifies connection details, authentication, table, and column names. The `search_parameters` section enables filtered searches based on metadata, limiting results to specific criteria.

```json
{
    "name": "pgvector-client",
    "implementation": "mlserver_vector_db.VectorDBRuntime",
    "parameters": {
        "extra": {
            "provider_id": "pgvector",
            "config": {
                "host": "pgvector-db",
                "port": 5432,
                "database": "db",
                "user": "admin",
                "password": "admin",
                "table": "embedding_table",
                "embedding_column": "embedding",
                "search_parameters": {
                    "columns": ["id", "text", "sentiment"],
                    "where": {
                        "AND": [
                            {"id": {"lt": 10}},
                            {"sentiment": "negative"}
                        ]
                    },
                    "limit": 1
                }
            }
        }
    }
}

```

--------------------------------

### Custom Preprocessor Model Implementation (Python)

Source: https://docs.seldon.ai/llm-module/introduction/quickstart

Provides the Python class for a custom MLServer preprocessor model. It handles unpacking JSON input from an LLM, parsing it, and preparing it for downstream classification models. It includes methods for loading the model and predicting.

```python
import json
from json import JSONDecodeError

import numpy as np
from mlserver import MLModel
from mlserver.logging import logger
from mlserver.codecs import StringCodec, NumpyCodec
from mlserver.types import InferenceRequest, InferenceResponse


enc_np = lambda k, v: NumpyCodec.encode_output(k, np.array([v]))
enc_str = lambda k, v: StringCodec.encode_output(k, [v])


class PreProcessorModel(MLModel):
    features = [
        "ApplicantIncome",
        "LoanAmount",
        "Loan_Amount_Term",
        "Gender",
        "Married",
        "Dependents",
        "Education",
        "Self_Employed",
        "Property_Area",
    ]

    async def load(self) -> bool:
        self.ready = True
        return self.ready

    async def predict(self, payload: InferenceRequest) -> InferenceResponse:
        logger.info("in preprocessor")
        data = self._unpack_input(payload)
        return self._build_response(data)

    def _unpack_input(self, payload: InferenceRequest):
        request_data = {}
        for inp in payload.inputs:
            request_data[inp.name] = StringCodec.decode_input(inp)
        try:
            logger.info(request_data)
            data = request_data["content"][0]
            data = data[data.index("{") : data.rindex("}") + 1]
            logger.info(data)
            data = json.loads(data)

```

--------------------------------

### Cleanup Seldon LLM Servers (Shell)

Source: https://docs.seldon.ai/llm-module/introduction/installation

These shell commands are used to remove Seldon LLM servers from your cluster. Each command targets a specific server configuration file for deletion.

```shell
kubectl delete -f server-api.yaml
kubectl delete -f server-memory.yaml
kubectl delete -f server-local.yaml
kubectl delete -f server-vector-db.yaml
```

--------------------------------

### Apply and Wait for AWQ Model Deployment

Source: https://docs.seldon.ai/llm-module/components/models/local/quantization

These commands apply the Seldon manifest file to deploy the AWQ model and then wait for the model to become ready. This ensures the model is fully deployed before proceeding with inference.

```bash
!kubectl apply -f manifests/vllm/llama-awq.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon
```

--------------------------------

### Configure Model Deployment (local-chat-completions.yaml)

Source: https://docs.seldon.ai/llm-module/use-cases/local

This YAML defines a Seldon ML model resource. It specifies the storage location for the model artifacts ('gs://seldon-models/llm-runtimes-settings/monitoring/local/models/local-chat-completions') and lists the necessary requirements ('llm-local').

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: local-chat-completions
spec:
  storageUri: "gs://seldon-models/llm-runtimes-settings/monitoring/local/models/local-chat-completions"
  requirements:
  - llm-local
```

--------------------------------

### Apply Conversational Memory Runtime Manifest using Kubectl

Source: https://docs.seldon.ai/llm-module/introduction/installation

This shell command applies the server-memory.yaml manifest file to the 'seldon' namespace, deploying the Conversational Memory Runtime server.

```shell
kubectl apply -f server-memory.yaml -n seldon
```

--------------------------------

### Apply API Runtime Manifest using Kubectl

Source: https://docs.seldon.ai/llm-module/introduction/installation

This shell command applies the server-api.yaml manifest file to the specified Kubernetes namespace ('seldon'), deploying the API Runtime server.

```shell
kubectl apply -f server-api.yaml -n seldon
```

--------------------------------

### Kubernetes Model Creation Confirmation

Source: https://docs.seldon.ai/llm-module/use-cases/local

This output confirms that the Kubernetes Model resource 'local-chat-completions' has been successfully created in the 'seldon' namespace.

```text
model.mlops.seldon.io/local-chat-completions created
```

--------------------------------

### Apply Model Deployment to Kubernetes

Source: https://docs.seldon.ai/llm-module/use-cases/local

This command applies the Kubernetes manifest to create or update the 'local-chat-completions' model resource within the 'seldon' namespace. It ensures the model is deployed and ready for use.

```shell
!kubectl apply -f manifests/local-chat-completions.yaml -n seldon
```

--------------------------------

### Conversational Memory Settings with PostgreSQL (JSON)

Source: https://docs.seldon.ai/llm-module/components/memory/example

This JSON configuration file defines settings for the `mlserver_memory.ConversationalMemory` implementation, specifically connecting to a PostgreSQL database. It includes the database connection URL, driver information (pg8000), and parameters for managing conversation history like window size and tensor names. The URL format is `postgresql+pg8000://user:password@host:port/database`.

```json
{
    "name": "psql-memory",
    "implementation": "mlserver_memory.ConversationalMemory",
    "parameters": {
        "extra": {
            "database": "sql",
            "config": {
                "url": "postgresql+pg8000://admin:admin@postgres:5432/db",
                "connection_options": {},
                "window_size": 50,
                "tensor_names": ["role", "content", "type"]
            }
        }
    }
}

```

--------------------------------

### Chat Completions Prompt Deployment (Kubernetes YAML)

Source: https://docs.seldon.ai/llm-module/components/prompting/example

Kubernetes manifest for deploying the 'chat-completions' prompt model. This YAML defines a Seldon 'Model' resource and crucially links it to the 'tiny-llama' LLM using the `modelRef` field. It specifies requirements including the 'prompt' library.

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: chat-completions
spec:
  storageUri: "gs://seldon-models/llm-runtimes-settings/prompting/models/chat-completions"
  llm:
    modelRef: tiny-llama
  requirements:
  - prompt
```

--------------------------------

### Deploy VectorDB Runtime

Source: https://docs.seldon.ai/llm-module/introduction/installation

This YAML configuration defines the deployment of the VectorDB Runtime server. It uses mlserver as the server configuration and enables the 'vector-db' capability. The image specified is for the VectorDB runtime.

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-vector-db
spec:
  serverConfig: mlserver
  capabilities:
  - vector-db
  podSpec:
    imagePullSecrets:
    - name: seldon-registry
    containers:
    - name: mlserver
      image: europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-vector-db:0.7.0
      imagePullPolicy: Always
```

--------------------------------

### Base64 Encoding of API Key/Token

Source: https://docs.seldon.ai/llm-module/introduction/installation

This command demonstrates how to base64 encode an API key or token, which is required when creating Kubernetes secrets for sensitive information like API keys and tokens.

```shell
echo -n '[key/token-to-encode]' | base64
```

--------------------------------

### Pull MLServer LLM Memory Docker Image

Source: https://docs.seldon.ai/llm-module/components/memory/example

Pulls the specified Docker image for the MLServer LLM Memory runtime. Ensure you are authenticated with Docker CLI before running this command.

```sh
docker pull \
    europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-llm-memory:0.7.0
```

--------------------------------

### Download Audio Sample using cURL

Source: https://docs.seldon.ai/llm-module/components/models/api/gemini

This snippet downloads a sample MP3 audio file ('apollo.mp3') from a specified URL using the cURL command-line tool. It saves the downloaded file to the 'assets/' directory. No specific Python dependencies are required for this command itself.

```shell
!curl -o assets/apollo.mp3 https://storage.googleapis.com/generativeai-downloads/data/Apollo-11_Day-01-Highlights-10s.mp3
```

--------------------------------

### Chat Completions Prompt Configuration (JSON)

Source: https://docs.seldon.ai/llm-module/components/prompting/example

Configuration file for a 'chat-completions' prompt model. This JSON specifies the implementation as 'mlserver_prompt_utils.runtime.PromptRuntime' and defines the prompt utility model type as 'chat.completions'. This model is designed to compile and forward chat completion prompts.

```json
{
    "name": "chat-completions",
    "implementation": "mlserver_prompt_utils.runtime.PromptRuntime",
    "parameters": {
        "extra": {
            "prompt_utils": {
                "model_type": "chat.completions"
            }
        }
    }
}
```

--------------------------------

### Deploy Local Runtime with GPU Support

Source: https://docs.seldon.ai/llm-module/introduction/installation

This YAML configuration defines the deployment of the Local Runtime server with GPU support. It specifies mlserver as the server configuration and enables the 'llm-local' capability. It configures GPU resource requests and limits, and mounts an emptyDir volume for shared memory.

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-llm-local
spec:
  serverConfig: mlserver
  capabilities:
  - llm-local
  podSpec:
    imagePullSecrets:
    - name: seldon-registry
    containers:
    - name: mlserver
      image: europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-llm-local:0.7.0
      imagePullPolicy: Always
      env:
      - name: HF_TOKEN
        valueFrom:
          secretKeyRef:
            name: huggingface-token
            key: HF_TOKEN
      resources:
        requests:
          nvidia.com/gpu: 4
          memory: 4Gi
        limits:
          memory: 32Gi
          nvidia.com/gpu: 4
      volumeMounts:
      - mountPath: /dev/shm
        name: dshm
        readOnly: false
    volumes:
    - emptyDir:
        medium: Memory
        sizeLimit: 1Gi
      name: dshm
```

--------------------------------

### Deploy and Test OpenVINO Model with Kubectl

Source: https://docs.seldon.ai/llm-module/components/embeddings/advanced

These commands demonstrate the deployment of an OpenVINO model to a Kubernetes cluster using Seldon Core. It includes applying the manifest, waiting for the model to become ready, and sending an inference request to test its functionality.

```bash
!kubectl apply -f manifests/embed-openvino.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldon
```

```bash
send_request(endpoint=f"http://{get_mesh_ip()}/v2/models/embed-openvino/infer")
```

--------------------------------

### Deploy Seldon Pipeline using Kubectl

Source: https://docs.seldon.ai/llm-module/use-cases/agentic-workflows/planning

Commands to apply the pipeline manifest and wait for the pipeline to become ready in the Seldon namespace.

```bash
!kubectl apply -f manifest/pipeline.yaml -n seldon
!kubectl wait --for condition=ready --timeout=300s pipelines --all -n seldon
```
Tokens:
Raw