GPTCache

https://github.com/zilliztech/gptcache
Admin
GPTCache is a semantic cache library for LLM queries that reduces API costs by 10x and improves...
Tokens:66,081
Snippets:514
Trust Score:8.4
License:MIT
Update:6 months ago
Show doc for...
Tokens:
Raw
### Quickstart: Initialize LLM and Load QA Chain with GPTCache

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/langchain/question_answering.ipynb

This snippet shows the recommended way to initialize an LLM with GPTCache and load a question-answering chain. It uses LangChainLLMs with OpenAI and the 'stuff' chain type for quick setup.

```python
from gptcache.adapter.langchain import LangChainLLMs
from langchain.llms import OpenAI
from langchain.chains.qa.utils import load_qa_chain

# Assuming 'docs' is a list of Document objects
docs = [...] 

llm = LangChainLLMs(llm=OpenAI(temperature=0))
chain = load_qa_chain(llm, chain_type="stuff")
query = "What did the president say about Justice Breyer"
chain.run(input_documents=docs, question=query)
```

--------------------------------

### Install GPTCache from source

Source: https://github.com/zilliztech/gptcache/blob/main/docs/index.rst

Instructions for cloning the repository and installing the development version of GPTCache.

```bash
git clone -b dev https://github.com/zilliztech/GPTCache.git
cd GPTCache
pip install -r requirements.txt
python setup.py install
```

--------------------------------

### Start GPTCache Server

Source: https://github.com/zilliztech/gptcache/blob/main/examples/README.md

Instructions on how to start the GPTCache server from the command line, including optional arguments for host, port, cache directory, and configuration file.

```APIDOC
## Start GPTCache Server

Once you have GPTCache installed, you can start the server with the following command:

```shell
$ gptcache_server -s [HOST] -p [PORT] -d [CACHE_DIRECTORY] -f [CACHE_CONFIG_FILE]
```

The arguments are optional:
- `-s/--host`: Specify the host to start GPTCache service, defaults to "0.0.0.0".
- `-p/--port`: Specify the port to access the service, defaults to 8000.
- `-d/--cache-dir`: Specify the directory of the cache, defaults to `gptcache_data` folder.
- `-f/--cache-config-file`: Specify the YAML file to config GPTCache service, defaults to None.
```

--------------------------------

### Start GPTCache Server with Docker

Source: https://github.com/zilliztech/gptcache/blob/main/examples/README.md

Instructions for using Docker to start the GPTCache server, including pulling the image and running containers with default or custom configurations.

```APIDOC
## Use the docker to start the GPTCache server

Also, you can start the service in a docker container:

- Get image from the dockerhub
    ```shell
    $ docker pull zilliz/gptcache:latest
    ```
- Run the service in a container with default port
    ```shell
    $ docker run -p 8000:8000 -it zilliz/gptcache:latest
    ```
- Run the service in a container with certain port (e.g. 8000) and config file (e.g. gptcache.yml)
    ```shell
    $ docker run -p 8000:8000 -it gptcache:v0 gptcache_server -s 0.0.0.0 -p 8000 -f gptcache.yml
    ```
```

--------------------------------

### Start GPTCache Server (Command Line)

Source: https://github.com/zilliztech/gptcache/blob/main/examples/README.md

Command to start the GPTCache server with optional host, port, cache directory, and configuration file. Defaults are provided for each argument.

```shell
gptcache_server -s [HOST] -p [PORT] -d [CACHE_DIRECTORY] -f [CACHE_CONFIG_FILE]
```

--------------------------------

### Start GPTCache Server with Docker

Source: https://github.com/zilliztech/gptcache/blob/main/docs/usage.md

Launches the GPTCache server using a Docker container. This method simplifies setup by abstracting dependencies. It maps port 8000 from the container to the host machine.

```shell
docker pull zilliz/gptcache:latest
docker run -p 8000:8000 -it zilliz/gptcache:latest
```

--------------------------------

### Start GPTCache Server

Source: https://github.com/zilliztech/gptcache/blob/main/docs/usage.md

Starts the GPTCache server on a specified IP address and port. This server provides caching and conversation functionalities. Ensure GPTCache is installed before running.

```shell
gptcache_server -s 127.0.0.1 -p 8000
```

--------------------------------

### Integrate Langchain LLMs with GPTCache

Source: https://github.com/zilliztech/gptcache/blob/main/docs/usage.md

Provides an example of using the Langchain adapter with GPTCache. It sets up a Langchain LLM, initializes a GPTCache instance with custom functions, and then uses the cached LLM to get an answer.

```python
from gptcache.adapter.langchain import LangChainLLMs
from gptcache.core import Cache
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate

# Dummy functions for demonstration
def get_prompt(messages, llm, **kwargs):
    return messages[0]['content']

def postnop(messages, llm, **kwargs):
    return messages

template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate(template=template, input_variables=["question"])

llm = OpenAI()

question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"

llm_cache = Cache()
llm_cache.init(
    pre_embedding_func=get_prompt,
    post_process_messages_func=postnop,
)

cached_llm = LangChainLLMs(llm)
answer = cached_llm(question, cache_obj=llm_cache)
print(answer)
```

--------------------------------

### Initialize Similar Cache

Source: https://context7.com/zilliztech/gptcache/llms.txt

Shows how to initialize a similar-match cache using `init_similar_cache`. It covers both the default setup with ONNX embeddings and local storage, and a custom setup using OpenAI embeddings and a specific similarity evaluation.

```python
from gptcache import Cache, Config
from gptcache.adapter.api import init_similar_cache, put, get
from gptcache.embedding import OpenAI as OpenAIEmbedding
from gptcache.similarity_evaluation import SearchDistanceEvaluation

# Quick setup with defaults (ONNX + SQLite + FAISS)
init_similar_cache(data_dir="./my_cache")

put("What is machine learning?", "Machine learning is a subset of AI...")
result = get("Explain ML to me")
print(result)

# Custom setup with OpenAI embeddings
custom_cache = Cache()
openai_embedding = OpenAIEmbedding(api_key="your-api-key")

init_similar_cache(
    cache_obj=custom_cache,
    embedding=openai_embedding,
    evaluation=SearchDistanceEvaluation(max_distance=2.0),
    config=Config(similarity_threshold=0.75)
)

put("How does Python work?", "Python interprets code line by line...", cache_obj=custom_cache)
```

--------------------------------

### GPTCache Server Configuration Example (YAML)

Source: https://github.com/zilliztech/gptcache/blob/main/examples/README.md

An example YAML configuration file for the GPTCache server, specifying settings for embedding, storage, evaluation, and other parameters.

```yaml
embedding:
    onnx
embedding_config:
    # Set embedding model params here
storage_config:
    data_dir:
        gptcache_data
    manager:
        sqlite,faiss
    vector_params:
        # Set vector storage related params here
evaluation: 
    distance
evaluation_config:
    # Set evaluation metric kws here
pre_function:
    get_prompt
post_function:
    first
config:
    similarity_threshold: 0.8
    # Set other config here
```

--------------------------------

### Dev Install GPTCache

Source: https://github.com/zilliztech/gptcache/blob/main/README.md

This section details the steps for installing GPTCache in development mode. It involves cloning the repository, navigating to the directory, and then installing the required dependencies and the package itself.

```bash
# clone GPTCache repo
git clone -b dev https://github.com/zilliztech/GPTCache.git
cd GPTCache

# install the repo
pip install -r requirements.txt
python setup.py install
```

--------------------------------

### Import Required Modules and Setup Session

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/langchain/baby_agi.ipynb

Imports necessary LangChain and GPTCache modules and initializes a session for tracking agent interactions.

```python
import os
from collections import deque
from typing import Dict, List, Optional, Any
from langchain import LLMChain, OpenAI, PromptTemplate
from langchain.embeddings import OpenAIEmbeddings
from langchain.llms import BaseLLM
from langchain.vectorstores.base import VectorStore
from pydantic import BaseModel, Field
from langchain.chains.base import Chain
from langchain.experimental import BabyAGI
from gptcache.adapter.langchain_models import LangChainLLMs
from gptcache.session import Session
session = Session(name="baby_agi")
```

--------------------------------

### Initialize and Use OpenAI Adapter with GPTCache

Source: https://github.com/zilliztech/gptcache/blob/main/docs/usage.md

Demonstrates how to initialize GPTCache, set the OpenAI API key, and make a chat completion request using the OpenAI adapter. This example shows a basic interaction for generating responses.

```python
import openai
from gptcache import Cache
from gptcache.manager import get_data_manager

cache = Cache()
cache.init(data_manager=get_data_manager())
os.environ["OPENAI_API_KEY"] = "API KEY"
cache.set_openai_key()

question = "what's github"
answer = openai.ChatCompletion.create(
      model='gpt-3.5-turbo',
      messages=[
        {
          'role': 'user',
          'content': question
        }
      ],
    )
print(answer)
```

--------------------------------

### Initialize GPTCache Client

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Shows how to instantiate a GPTCache client to perform basic put and get operations for caching key-value pairs.

```python
from gptcache import Client

client = Client()
client.put("Hi", "Hi back")
ans = client.get("Hi")
```

--------------------------------

### Start GPTCache Server (Docker)

Source: https://github.com/zilliztech/gptcache/blob/main/examples/README.md

Commands to pull the GPTCache Docker image and run the server in a container, with options for specifying ports and configuration files.

```shell
docker pull zilliz/gptcache:latest
```

```shell
docker run -p 8000:8000 -it zilliz/gptcache:latest
```

```shell
docker run -p 8000:8000 -it gptcache:v0 gptcache_server -s 0.0.0.0 -p 8000 -f gptcache.yml
```

--------------------------------

### Setup LangChain with GPTCache Session

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/langchain/sqlite.ipynb

Initializes a GPTCache session and prepares the LangChain environment for use with a SQLite database. This allows LLM interactions to be cached within a specific session context.

```python
from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
from gptcache.adapter.langchain_models import LangChainLLMs
from gptcache.session import Session

session = Session(name="sqlite-example")
db = SQLDatabase.from_uri("sqlite:///./Chinook.db")
```

--------------------------------

### GPTCache Put and Get API

Source: https://context7.com/zilliztech/gptcache/llms.txt

Illustrates the basic key-value storage and retrieval using GPTCache's `put` and `get` functions. It shows how to initialize an exact-match cache and a similar-match cache, demonstrating storing and retrieving data based on exact keys or semantic similarity.

```python
from gptcache import cache, Config, Cache
from gptcache.adapter.api import put, get, init_similar_cache
from gptcache.processor.pre import get_prompt
from gptcache.processor.post import nop

# Basic exact-match cache
cache.init(pre_embedding_func=get_prompt)

put("hello", "Hello! How can I help you today?")
result = get("hello")
print(result)  # Output: Hello! How can I help you today?

# Similar-match cache with multiple results
similar_cache = Cache()
init_similar_cache(
    cache_obj=similar_cache,
    post_func=nop,  # Return all matches instead of just the best
    config=Config(similarity_threshold=0)
)

# Store multiple Q&A pairs
put("What is Python?", "Python is a programming language.", cache_obj=similar_cache)
put("Explain Python", "Python is a high-level, interpreted language.", cache_obj=similar_cache)
put("Tell me about Python", "Python is popular for web development and data science.", cache_obj=similar_cache)

# Retrieve top-k similar results
results = get("Python programming", cache_obj=similar_cache, top_k=3)
print(results)  # Returns list of all similar answers
```

--------------------------------

### Initialize OpenAI API call

Source: https://github.com/zilliztech/gptcache/blob/main/docs/index.rst

Standard setup for making an OpenAI ChatCompletion request in Python.

```python
import os
import time
import openai

def response_text(openai_resp):
    return openai_resp['choices'][0]['message']['content']

question = 'what‘s chatgpt'
openai.api_key = os.getenv("OPENAI_API_KEY")
start_time = time.time()
response = openai.ChatCompletion.create(
  model='gpt-3.5-turbo',
  messages=[
    {
      "role": "user",
      "content": question
    }
  ]
)
```

--------------------------------

### Configure Similarity Evaluation Strategies

Source: https://github.com/zilliztech/gptcache/blob/main/examples/README.md

Examples of initializing GPTCache with different similarity evaluation strategies like ExactMatch, SearchDistance, ONNX models, and Numpy norms.

```python
from gptcache import cache
from gptcache.similarity_evaluation.exact_match import ExactMatchEvaluation
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
from gptcache.similarity_evaluation.onnx import OnnxModelEvaluation
from gptcache.similarity_evaluation.np import NumpyNormEvaluation

# Example using ExactMatch
cache.init(similarity_evaluation=ExactMatchEvaluation())
cache.set_openai_key()
```

--------------------------------

### Configure Cache with YAML for Server

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of configuring the GPTCache server using a YAML file.

```APIDOC
## Configure Cache with YAML for Server

### Description
This snippet shows how to initialize the GPTCache server's cache configuration using a YAML file.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache.adapter.api import init_similar_cache_from_config

init_similar_cache_from_config(config_dir="cache_config_template.yml")
```

### Response
N/A

### Response Example
N/A

**Config File Template:** https://github.com/zilliztech/GPTCache/blob/main/cache_config_template.yml
```

--------------------------------

### Configure GPTCache with Redis Distributed Eviction Manager

Source: https://github.com/zilliztech/gptcache/blob/main/docs/horizontal-scaling-usage.md

Demonstrates how to initialize a GPTCache instance using a Redis-backed data manager. This setup enables horizontal scaling by sharing cache metadata across multiple nodes via Redis.

```python
from gptcache import Cache
from gptcache.embedding import Onnx
from gptcache.manager import manager_factory

onnx = Onnx()
data_manager = manager_factory("redis,faiss",
                               eviction_manager="redis",
                               scalar_params={"url": "redis://localhost:6379"},
                               vector_params={"dimension": onnx.dimension},
                               eviction_params={"maxmemory": "100mb",
                                                "policy": "allkeys-lru",
                                                "ttl": 1}
                               )

cache = Cache()
cache.init(data_manager=data_manager)
question = "What is github?"
answer = "Online platform for version control and code collaboration."
embedding = onnx.to_embeddings(question)
cache.import_data([question], [answer], [embedding])
```

--------------------------------

### GPTCache Example: Custom Cache Initialization for OpenAI

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

This Python code demonstrates how to initialize a GPTCache with custom components, including embedding, data management, and similarity evaluation, for use with OpenAI's ChatCompletion API. It shows how to configure different processors and managers to tailor the cache to specific needs. The example iterates through a list of questions, queries the OpenAI API with caching enabled, and prints the response time and answer.

```python
import time

from gptcache import Cache, Config
from gptcache.adapter import openai
from gptcache.adapter.api import init_similar_cache
from gptcache.embedding import Onnx
from gptcache.manager import manager_factory
from gptcache.processor.post import random_one
from gptcache.processor.pre import last_content
from gptcache.similarity_evaluation import OnnxModelEvaluation

openai_complete_cache = Cache()
encoder = Onnx()
sqlite_faiss_data_manager = manager_factory(
    "sqlite,faiss",
    data_dir="openai_complete_cache",
    scalar_params={
        "sql_url": "sqlite:///./openai_complete_cache.db",
        "table_name": "openai_chat",
    },
    vector_params={
        "dimension": encoder.dimension,
        "index_file_path": "./openai_chat_faiss.index",
    },
)
onnx_evaluation = OnnxModelEvaluation()
cache_config = Config(similarity_threshold=0.75)

init_similar_cache(
    cache_obj=openai_complete_cache,
    pre_func=last_content,
    embedding=encoder,
    data_manager=sqlite_faiss_data_manager,
    evaluation=onnx_evaluation,
    post_func=random_one,
    config=cache_config,
)

questions = [
    "what's github",
    "can you explain what GitHub is",
    "can you tell me more about GitHub",
    "what is the purpose of GitHub",
]

for question in questions:
    start_time = time.time()
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": question}],
        cache_obj=openai_complete_cache,
    )
    print(f"Question: {question}")
    print("Time consuming: {:.2f}s".format(time.time() - start_time))
    print(f'Answer: {response["choices"][0]["message"]["content"]}\n')

```

--------------------------------

### Implement Exact Match Caching with GPTCache

Source: https://github.com/zilliztech/gptcache/blob/main/README.md

Demonstrates how to initialize GPTCache for exact query matching. This setup intercepts OpenAI API calls and serves identical responses from the cache to avoid redundant network requests.

```python
import time
from gptcache import cache
from gptcache.adapter import openai

def response_text(openai_resp):
    return openai_resp['choices'][0]['message']['content']

cache.init()
cache.set_openai_key()

question = "what's github"
for _ in range(2):
    start_time = time.time()
    response = openai.ChatCompletion.create(
      model='gpt-3.5-turbo',
      messages=[{'role': 'user', 'content': question}]
    )
    print(f'Question: {question}')
    print("Time consuming: {:.2f}s".format(time.time() - start_time))
    print(f'Answer: {response_text(response)}\n')
```

--------------------------------

### Initialize Similar Cache

Source: https://context7.com/zilliztech/gptcache/llms.txt

Simplifies the setup of semantic similarity caching with sensible defaults for embeddings, vector stores, and evaluation metrics.

```APIDOC
## Initialize Similar Cache

The `init_similar_cache` function streamlines the process of setting up a cache that leverages semantic similarity. It comes with sensible defaults, including ONNX embeddings, FAISS vector store, and distance-based evaluation, making it easy to get started with advanced caching capabilities.

### Description
Initializes a cache instance configured for semantic similarity search, allowing retrieval of semantically related data.

### Method
`init_similar_cache(cache_obj=None, data_dir=None, embedding=None, evaluation=None, config=None, post_func=None)`

### Endpoint
N/A (Configuration function)

### Parameters
- **cache_obj** (Cache object) - Optional - An existing Cache object to configure. If None, a new one is created.
- **data_dir** (string) - Optional - Directory to store cache data, including vector indexes.
- **embedding** (Embedding object) - Optional - The embedding function to use (e.g., OpenAI, Huggingface). Defaults to ONNX.
- **evaluation** (Evaluation object) - Optional - The similarity evaluation metric (e.g., SearchDistanceEvaluation). Defaults to a distance-based evaluation.
- **config** (Config object) - Optional - Configuration settings like `similarity_threshold`.
- **post_func** (function) - Optional - Function to process retrieval results.

### Request Example (Default Setup)
```python
from gptcache.adapter.api import init_similar_cache, put, get

# Quick setup with defaults (ONNX + SQLite + FAISS)
init_similar_cache(data_dir="./my_cache")

put("What is machine learning?", "Machine learning is a subset of AI...")
result = get("Explain ML to me")
print(result)
```

### Request Example (Custom Setup)
```python
from gptcache import Cache, Config
from gptcache.adapter.api import init_similar_cache
from gptcache.embedding import OpenAI as OpenAIEmbedding
from gptcache.similarity_evaluation import SearchDistanceEvaluation

# Custom setup with OpenAI embeddings
custom_cache = Cache()
openai_embedding = OpenAIEmbedding(api_key="your-api-key")

init_similar_cache(
    cache_obj=custom_cache,
    embedding=openai_embedding,
    evaluation=SearchDistanceEvaluation(max_distance=2.0),
    config=Config(similarity_threshold=0.75)
)

put("How does Python work?", "Python interprets code line by line...", cache_obj=custom_cache)
```

### Response
This function does not return a value but configures the provided or newly created `Cache` object for similar-match retrieval.
```

--------------------------------

### OpenAI API Original Usage Example

Source: https://github.com/zilliztech/gptcache/blob/main/README.md

This Python code snippet shows the basic usage of the OpenAI API for getting a response. It defines a helper function to extract the content from the API response and sets up a sample question.

```python
import os
import time

import openai


def response_text(openai_resp):
    return openai_resp['choices'][0]['message']['content']


question = 'what‘s chatgpt'

```

--------------------------------

### Initialize OpenAI Audio Adapter

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Demonstrates initializing the GPTCache with a file-based pre-embedding function and performing audio transcription using OpenAI's Whisper model.

```python
cache.init(pre_embedding_func=get_file_bytes)

openai.Audio.transcribe(
    model="whisper-1",
    file=audio_file
)
```

--------------------------------

### Add Embedding Functionality in Python

Source: https://github.com/zilliztech/gptcache/blob/main/docs/contributing.md

This example shows how to implement a new embedding function. It emphasizes lazy importing of third-party libraries and automatic installation to ensure smooth integration. The output dimension must be correctly specified.

```python
from gptcache.embedding import BaseEmbedding

class NewEmbedding(BaseEmbedding):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        # Lazy import third-party libraries here
        try:
            import some_embedding_library
        except ImportError:
            raise ImportError("Please install 'some_embedding_library' to use NewEmbedding.")
        self.model = some_embedding_library.load_model(**kwargs)
        self.output_dim = self.model.get_output_dimension()

    def get_embedding(self, text, **kwargs):
        # Implement embedding generation logic using the loaded model
        return self.model.encode(text, **kwargs)
```

--------------------------------

### Configure GPTCache Settings

Source: https://context7.com/zilliztech/gptcache/llms.txt

Demonstrates how to initialize the cache with custom configuration parameters such as similarity thresholds, auto-flush intervals, and template handling for prompt extraction.

```python
from gptcache import cache, Config
from gptcache.embedding import Onnx
from gptcache.manager import manager_factory
from gptcache.similarity_evaluation import SearchDistanceEvaluation

onnx = Onnx()
config = Config(
    similarity_threshold=0.8,
    auto_flush=20,
    enable_token_counter=True,
    context_len=5,
    template="Question: {question}\nAnswer:",
    prompts=["System:", "Assistant:"],
)
cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=manager_factory("sqlite,faiss", vector_params={"dimension": onnx.dimension}),
    similarity_evaluation=SearchDistanceEvaluation(),
    config=config
)
```

--------------------------------

### GPTCache API (Put and Get)

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Simplified API for putting and getting data from the cache.

```APIDOC
## GPTCache API (Put and Get)

### Description
This section introduces a simplified API for interacting with GPTCache, allowing easy insertion and retrieval of data.

### Method
`put()` and `get()`

### Endpoint
N/A (Library functions)

### Parameters
#### `put(key, value)`
- **key** (string) - Required - The key to identify the data.
- **value** (string) - Required - The data to be stored.

#### `get(key)`
- **key** (string) - Required - The key of the data to retrieve.

### Request Example
```python
from gptcache.adapter.api import put, get
from gptcache.processor.pre import get_prompt
from gptcache import cache

cache.init(pre_embedding_func=get_prompt)
put("hello", "foo")
print(get("hello"))
```

### Response
#### Success Response (200)
- **get() return value** (string) - The data associated with the provided key.

#### Response Example
```json
{
  "get('hello')": "foo"
}
```
```

--------------------------------

### Configure Context Processing Interfaces

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Demonstrates initializing GPTCache with either summarization or selective context processing strategies to handle dialog history.

```python
import transformers
from gptcache.processor.context.summarization_context import SummarizationContextProcess
from gptcache.processor.context.selective_context import SelectiveContextProcess
from gptcache import cache

summarizer = transformers.pipeline("summarization", model="facebook/bart-large-cnn")
context_process = SummarizationContextProcess(summarizer, None, 512)
cache.init(
    pre_embedding_func=context_process.pre_process,
)

context_processor = SelectiveContextProcess()
cache.init(
    pre_embedding_func=context_process.pre_process,
)
```

--------------------------------

### GPTCache Initialization and Configuration

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Demonstrates how to initialize the GPTCache with custom embedding functions, data managers, and similarity evaluations.

```APIDOC
## GPTCache Initialization and Configuration

### Description
This section shows how to initialize the GPTCache with various components such as pre-embedding functions, embedding functions, data managers, and similarity evaluation metrics.

### Method
`cache.init()`

### Endpoint
N/A (Library function)

### Parameters
#### Request Body
- **pre_embedding_func** (function) - Optional - A function to preprocess the input before embedding.
- **embedding_func** (function) - Required - The function used to generate embeddings for the input data.
- **data_manager** (object) - Required - An instance of a data manager for storing and retrieving data.
- **similarity_evaluation** (object) - Optional - An object that defines how similarity is evaluated.

### Request Example
```python
from gptcache import cache
from gptcache.embedding import Onnx
from gptcache.manager import manager_factory
from gptcache.processor.pre import get_prompt
from gptcache.similarity_evaluation import SearchDistanceEvaluation

onnx = Onnx()
data_manager = manager_factory("sqlite,faiss,local", 
                               data_dir='./', 
                               vector_params={'dimension': onnx.dimension},
                               object_params={'path': './images'}
                               )
cache.init(
    pre_embedding_func=get_prompt,
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation()
    )
```

### Response
#### Success Response (200)
N/A (Initialization function, no direct response)

#### Response Example
N/A
```

--------------------------------

### Install Gptcache and Vertex AI dependencies

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/vertex/vertexai_caching.ipynb

Installs the necessary Python packages to enable Vertex AI integration and Gptcache functionality within a notebook environment.

```bash
!pip install google-cloud-aiplatform
!pip install -q gptcache langchain
```

--------------------------------

### Initialize Cache with Next Cache in Python

Source: https://github.com/zilliztech/gptcache/blob/main/docs/feature.md

Demonstrates how to initialize a backup cache and chain it with the main cache. This allows for fallback mechanisms or tiered caching strategies.

```python
bak_cache = Cache()
bak_cache.init()
cache.init(next_cache=bak_cache)
```

--------------------------------

### Install GPTCache with pip

Source: https://github.com/zilliztech/gptcache/blob/main/README.md

This command installs the GPTCache library and its basic functionalities using pip. Ensure you have Python 3.8.1 or higher. If you encounter issues, upgrade pip.

```bash
pip install gptcache
```

--------------------------------

### Initialize GPTCache with Temperature Support

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/temperature/chat.ipynb

Sets up the GPTCache environment using ONNX embeddings, SQLite/FAISS data management, and the temperature_softmax post-processing function.

```python
import time
from gptcache import cache, Config
from gptcache.manager import manager_factory
from gptcache.embedding import Onnx
from gptcache.processor.post import temperature_softmax
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
from gptcache.adapter import openai

onnx = Onnx()
data_manager = manager_factory("sqlite,faiss", vector_params={"dimension": onnx.dimension})

cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
    post_process_messages_func=temperature_softmax
)
cache.set_openai_key()
question = 'what is github'
```

--------------------------------

### Basic GPTCache API Put and Get

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Demonstrates the basic usage of the GPTCache API for putting and getting data. It initializes the cache with a pre-embedding function and then stores and retrieves a key-value pair.

```python
from gptcache.adapter.api import put, get
from gptcache.processor.pre import get_prompt
from gptcache import cache

cache.init(pre_embedding_func=get_prompt)
put("hello", "foo")
print(get("hello"))
```

--------------------------------

### Create GPTCache Data Managers with Various Backends

Source: https://github.com/zilliztech/gptcache/blob/main/docs/usage.md

Provides examples of creating GPTCache data managers with different combinations of cache storage (SQLite, MySQL) and vector storage (FAISS, Milvus). It also demonstrates setting cache size and eviction policies like LRU.

```python
## create user defined data manager
data_manager = get_data_manager()
## create data manager with sqlite and faiss 
data_manager = get_data_manager(CacheBase("sqlite"), VectorBase("faiss", dimension=128))
## create data manager with mysql and milvus, max cache size is 100
data_manager = get_data_manager(CacheBase("mysql"), VectorBase("milvus", dimension=128), max_size=100)
## create data manager with mysql and milvus, max cache size is 100, eviction policy is LRU
data_manager = get_data_manager(CacheBase("mysql"), VectorBase("milvus", dimension=128), max_size=100, eviction='LRU')
```

--------------------------------

### Initialize OpenAI Image Adapter

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Sets up the GPTCache for image generation tasks using the OpenAI adapter, including API key configuration and request formatting.

```python
from gptcache.adapter import openai

cache.init()
cache.set_openai_key()

prompt1 = 'a cat sitting besides a dog'
size1 = '256x256'

openai.Image.create(
    prompt=prompt1,
    size=size1,
    response_format='b64_json'
    )
```

--------------------------------

### Llama Adapter Integration

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of integrating the Llama adapter with GPTCache.

```APIDOC
## Llama Adapter Integration

### Description
This snippet shows how to integrate the Llama adapter with GPTCache for running Llama models.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache.adapter.llama_cpp import Llama

llm = Llama('./models/7B/ggml-model.bin')
answer = llm(prompt=question)
```

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Initialize GPTCache with LangChain for VertexAI

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/vertex/vertexai_caching.ipynb

This snippet demonstrates how to wrap a VertexAI LLM with GPTCache using the LangChain adapter. It initializes the cache with a prompt-based pre-embedding function and shows how to measure performance gains between a cold cache read and a cache hit.

```python
import time
from langchain.llms import VertexAI
from gptcache.adapter.langchain_models import LangChainLLMs
from gptcache import Cache
from gptcache.processor.pre import get_prompt

llm = VertexAI()
llm_cache = Cache()
llm_cache.init(pre_embedding_func=get_prompt)

cached_llm = LangChainLLMs(llm=llm)

question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"

# First request (Cache miss)
before = time.time()
answer = cached_llm(prompt=question, cache_obj=llm_cache)
print(f"Read through Time Spent = {time.time() - before}")

# Second request (Cache hit)
before = time.time()
answer = cached_llm(prompt=question, cache_obj=llm_cache)
print(f"Cache Hit Time Spent = {time.time() - before}")
```

--------------------------------

### Dolly Model Adaptation

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of adapting the Dolly model with GPTCache.

```APIDOC
## Dolly Model Adaptation

### Description
This snippet demonstrates how to adapt the Dolly model for use with GPTCache.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache.adapter.dolly import Dolly

llm = Dolly.from_model(model="databricks/dolly-v2-3b")
llm(question)
```

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Session Support for LangChainLLMs

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of using sessions with LangChainLLMs in GPTCache.

```APIDOC
## Session Support for LangChainLLMs

### Description
This snippet shows how to enable session management for LangChainLLMs within GPTCache, allowing for conversational context.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from langchain import OpenAI
from gptcache.adapter.langchain_models import LangChainLLMs
from gptcache.session import Session

session = Session(name="sqlite-example")
llm = LangChainLLMs(llm=OpenAI(temperature=0), session=session)
```

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Initialize Data Manager via Factory Method

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

Demonstrates how to use the manager_factory to create a data manager by specifying store names and their respective parameters. This approach simplifies the initialization process for common storage configurations.

```python
from gptcache.manager import manager_factory

data_manager = manager_factory("sqlite,faiss", data_dir="./workspace", scalar_params={}, vector_params={"dimension": 128})
```

--------------------------------

### LangChain Embedding Support

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of using LangChain embeddings with GPTCache.

```APIDOC
## LangChain Embedding Support

### Description
This snippet demonstrates how to integrate LangChain's embedding models with GPTCache for generating embeddings.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache.embedding import LangChain
from langchain.embeddings.openai import OpenAIEmbeddings

test_sentence = 'Hello, world.'
embeddings = OpenAIEmbeddings(model="your-embeddings-deployment-name")
encoder = LangChain(embeddings=embeddings)
embed = encoder.to_embeddings(test_sentence)
```

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Timm Image Embedding

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of using the Timm library for image embeddings.

```APIDOC
## Timm Image Embedding

### Description
This section demonstrates how to use the Timm library, which provides access to various pre-trained image models, for generating image embeddings.

### Method
`encoder.to_embeddings()`

### Endpoint
N/A (Library function)

### Parameters
#### Request Body
- **image_tensor** (object) - Required - The preprocessed image tensor to be embedded.

### Request Example
```python
import requests
from PIL import Image
from gptcache.embedding import Timm

url = 'https://raw.githubusercontent.com/zilliztech/GPTCache/main/docs/GPTCache.png'
image = Image.open(requests.get(url, stream=True).raw)  # Read image url as PIL.Image      
encoder = Timm(model='resnet18')
image_tensor = encoder.preprocess(image)
embed = encoder.to_embeddings(image_tensor)
```

### Response
#### Success Response (200)
- **embed** (list) - A list of embedding vectors for the image.

#### Response Example
```json
{
  "embed": [
    0.987, 0.654, ... // Embedding vector components
  ]
}
```
```

--------------------------------

### Summarization Context Process

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of optimizing summarization context process in GPTCache.

```APIDOC
## Summarization Context Process

### Description
This snippet demonstrates how to optimize the summarization context process in GPTCache by initializing the cache with a `SummarizationContextProcess`.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache import cache
from gptcache.processor.context.summarization_context import SummarizationContextProcess

context_process = SummarizationContextProcess()
cache.init(
    pre_embedding_func=context_process.pre_process,
)
```

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Implement Beginner-Level GPTCache Integration

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

This snippet demonstrates the simplest configuration for GPTCache using ONNX embedding, SQLite/FAISS data management, and distance-based similarity evaluation. It initializes the cache with a pre-processing function and performs sequential OpenAI API queries.

```python
import time

from gptcache.adapter import openai
from gptcache.adapter.api import init_similar_cache
from gptcache.processor.pre import last_content

init_similar_cache(pre_func=last_content)

questions = [
    "what's github",
    "can you explain what GitHub is",
    "can you tell me more about GitHub",
    "what is the purpose of GitHub",
]

for question in questions:
    start_time = time.time()
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo", messages=[{"role": "user", "content": question}],
    )
    print(f"Question: {question}")
    print("Time consuming: {:.2f}s".format(time.time() - start_time))
    print(f'Answer: {response["choices"][0]["message"]["content"]}\n')
```

--------------------------------

### Llama - Get Prompt

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

Retrieves the 'prompt' parameter from Llama model request parameters.

```python
# Example for get_prompt in Llama
# Assumes 'data' contains a 'prompt' key.
```

--------------------------------

### Initialize Data Manager via Component Objects

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

Demonstrates how to manually combine CacheBase and VectorBase objects using the get_data_manager method. This provides more granular control over the initialization of individual storage components.

```python
from gptcache.manager import get_data_manager, CacheBase, VectorBase

data_manager = get_data_manager(CacheBase('sqlite'), VectorBase('faiss', dimension=128))
```

--------------------------------

### Use OpenAI Adapter with Temperature and Sessions

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Shows how to use the GPTCache OpenAI adapter, supporting temperature parameters and session-based cache isolation.

```python
from gptcache.adapter import openai
from gptcache.session import Session

session = Session(name="my-session")
question = "what do you think about chatgpt"
openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": question}
    ],
    session=session
)
```

--------------------------------

### OpenAI Moderation - Get Input

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

Retrieves the input parameter for OpenAI moderation requests.

```python
# Example for get_openai_moderation_input
# Assumes 'data' contains the input text for moderation.
```

--------------------------------

### POST /cache/init

Source: https://github.com/zilliztech/gptcache/blob/main/examples/README.md

Initializes the GPTCache instance with a specific similarity evaluation strategy.

```APIDOC
## POST /cache/init

### Description
Initializes the cache with a chosen similarity evaluation engine to determine how request similarity is calculated.

### Method
POST

### Endpoint
/cache/init

### Parameters
#### Request Body
- **similarity_evaluation** (Object) - Required - The evaluation strategy instance (e.g., ExactMatchEvaluation, SearchDistanceEvaluation).

### Request Example
```python
from gptcache import cache
from gptcache.similarity_evaluation.exact_match import ExactMatchEvaluation

cache.init(similarity_evaluation=ExactMatchEvaluation())
```

### Response
#### Success Response (200)
- **status** (string) - Initialization success status.
```

--------------------------------

### Session Layer Implementation

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of using the session layer with the OpenAI adapter in GPTCache.

```APIDOC
## Session Layer Implementation

### Description
This snippet illustrates how to utilize the session layer in GPTCache with the OpenAI adapter, enabling stateful interactions.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache.adapter import openai
from gptcache.session import Session

session = Session(name="my-session")
question = "what do you think about chatgpt"
openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": question}
    ],
    session=session
)
```

### Response
N/A

### Response Example
N/A

**Details:** https://github.com/zilliztech/GPTCache/tree/main/examples#How-to-run-with-session
```

--------------------------------

### Configure GPTCache via YAML

Source: https://context7.com/zilliztech/gptcache/llms.txt

Demonstrates initializing the cache using a YAML configuration file for cleaner deployment and environment management.

```python
from gptcache.adapter.api import init_similar_cache_from_config
config = init_similar_cache_from_config("cache_config.yml")
```

```yaml
embedding: onnx
embedding_config:
  model: GPTCache/paraphrase-albert-onnx
storage_config:
  manager: sqlite,faiss
  data_dir: ./gptcache_data
  vector_params:
    dimension: 768
evaluation: distance
evaluation_config:
  max_distance: 4.0
  positive: false
config:
  similarity_threshold: 0.8
  auto_flush: 20
```

--------------------------------

### Temperature Parameter Support

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of using the temperature parameter with OpenAI adapter in GPTCache.

```APIDOC
## Temperature Parameter Support

### Description
This snippet demonstrates how to use the `temperature` parameter when making a chat completion request using the OpenAI adapter in GPTCache. A higher temperature increases randomness, while a lower temperature leads to more deterministic output.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache.adapter import openai

openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    temperature = 1.0,  # Change temperature here
    messages=[{"role": "user", "content": question}]
)
```

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Pgvector as Vector Store

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Configuration example for using pgvector as a vector store with GPTCache.

```APIDOC
## Pgvector as Vector Store

### Description
This snippet demonstrates how to configure pgvector as a vector store using the `manager_factory` in GPTCache.

### Method
N/A (Code Example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
from gptcache.manager import manager_factory

data_manager = manager_factory("sqlite,pgvector", vector_params={"dimension": 10})
```

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Langchain LLM - Get Prompt

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

Retrieves the 'prompt' parameter from Langchain LLM request parameters.

```python
# Example for get_prompt in Langchain LLM
# Assumes 'data' contains a 'prompt' key.
```

--------------------------------

### Simple GPTCache Server

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Instructions for running a basic GPTCache server for caching operations.

```APIDOC
## Simple GPTCache Server

### Description
This section provides instructions on how to set up and run a simple GPTCache server that allows basic caching operations via HTTP requests.

### Method
HTTP PUT and GET requests

### Endpoint
`http://localhost:8000`

### Parameters
#### PUT Request
- **Query Parameter `prompt`** (string) - Required - The data to be cached.

#### GET Request
- **Query Parameter `prompt`** (string) - Required - The key to retrieve cached data.

### Request Example
```bash
# To put data into the cache:
curl -X PUT -d "receive a hello message" "http://localhost:8000?prompt=hello"

# To get data from the cache:
curl -X GET  "http://localhost:8000?prompt=hello"
```

### Response
#### Success Response (200)
- **PUT**: Confirmation of data stored.
- **GET**: The cached data associated with the prompt.

#### Response Example
```json
{
  "message": "Data stored successfully."
}

# For GET request, the response would be the cached value, e.g.:
"receive a hello message"
```
```

--------------------------------

### Stability AI Integration

Source: https://github.com/zilliztech/gptcache/blob/main/docs/release_note.md

Example of using the Stability AI API for image generation with GPTCache.

```APIDOC
## Stability AI Image Generation

### Description
This snippet demonstrates how to use the Stability AI API for generating images, potentially integrated with GPTCache for caching purposes.

### Method
`stability_api.generate()`

### Endpoint
N/A (External API call)

### Parameters
#### Query Parameters
- **prompt** (string) - Required - The text prompt for image generation.
- **width** (integer) - Optional - The desired width of the generated image.
- **height** (integer) - Optional - The desired height of the generated image.

#### Request Body
N/A

### Request Example
```python
import os
from gptcache.adapter.stability import StabilityInference

api_key = os.getenv('STABILITY_KEY', 'key-goes-here')

stability_api = StabilityInference(
    key=os.environ['STABILITY_KEY'], # API Key reference.
    verbose=False, # Print debug messages.
    engine='stable-diffusion-xl-beta-v2-2-2', # Set the engine to use for generation.
    )

answers = stability_api.generate(
    prompt='a cat sitting besides a dog',
    width=256,
    height=256
    )
```

### Response
#### Success Response (200)
- **answers** (list) - A list of generated image data or references.

#### Response Example
```json
{
  "answers": [
    // Image data or URLs
  ]
}
```
```

--------------------------------

### Initialize Exact Match Cache

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/langchain/baby_agi.ipynb

Shows how to initialize GPTCache for exact match scenarios. This is the simplest form of caching where the prompt must match exactly.

```python
from gptcache import cache
cache.init(pre_embedding_func=get_content_func)
cache.set_openai_key()
```

--------------------------------

### OpenAI Audio - Get File Bytes

Source: https://github.com/zilliztech/gptcache/blob/main/docs/configure_it.md

Extracts the file bytes from OpenAI audio request parameters.

```python
# Example for get_file_bytes in OpenAI Audio
# Assumes 'data' contains file content or a reference to it.
```

--------------------------------

### Initialize SQLDatabaseChain with GPTCache

Source: https://github.com/zilliztech/gptcache/blob/main/docs/bootcamp/langchain/sqlite.ipynb

Sets up the SQLDatabaseChain using a cached LLM instance. This allows the chain to leverage GPTCache for performance improvements when querying databases.

```python
llm = LangChainLLMs(llm=OpenAI(temperature=0), session=session)
db_chain = SQLDatabaseChain(llm=llm, database=db, verbose=True)
db_chain.run("How many employees are there?")
```
Tokens:
Raw