### Install PyTiDB Dependencies (Bash) Source: https://github.com/pingcap/pytidb/blob/main/examples/quickstart/README.md Sets up a Python virtual environment and installs all required packages for the PyTiDB quickstart demo using pip. Ensures all dependencies are met. ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` -------------------------------- ### Install Dependencies and Setup Environment Source: https://github.com/pingcap/pytidb/blob/main/examples/rag/README.md Installs the necessary Python packages for the RAG application and sets up a virtual environment. This step ensures all required libraries are available for running the example. ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` -------------------------------- ### Set Up Python Virtual Environment and Install Dependencies Source: https://github.com/pingcap/pytidb/blob/main/examples/basic/README.md Creates a Python virtual environment and installs the necessary packages listed in `reqs.txt` for running the PyTiDB examples. ```Bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` -------------------------------- ### Clone PyTiDB Repository (Bash) Source: https://github.com/pingcap/pytidb/blob/main/examples/quickstart/README.md Clones the official PyTiDB GitHub repository to your local machine. This is the first step to access the example code and necessary files. ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/quickstart/ ``` -------------------------------- ### Setup Python Environment and Install Dependencies Source: https://github.com/pingcap/pytidb/blob/main/examples/hybrid_search/README.md Creates a Python virtual environment, activates it, and installs all necessary packages listed in the 'reqs.txt' file. ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` -------------------------------- ### Install Dependencies and Setup Virtual Environment Source: https://github.com/pingcap/pytidb/blob/main/examples/fulltext_search/README.md Sets up a Python virtual environment named '.venv' and installs all necessary project dependencies listed in 'reqs.txt' using pip. ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` -------------------------------- ### Install PyTiDB Dependencies Source: https://github.com/pingcap/pytidb/blob/main/examples/text2sql/README.md Sets up a Python virtual environment named '.venv', activates it, and installs all necessary packages listed in 'reqs.txt'. This ensures all project dependencies are met. ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` -------------------------------- ### Set Up Python Virtual Environment and Install Dependencies Source: https://github.com/pingcap/pytidb/blob/main/examples/vector_search/README.md Creates a Python virtual environment, activates it, and installs all required packages listed in 'reqs.txt'. ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` -------------------------------- ### Run PyTiDB Quickstart Demo (Bash) Source: https://github.com/pingcap/pytidb/blob/main/examples/quickstart/README.md Executes the main Python script for the PyTiDB quickstart demo. This script connects to TiDB, performs data operations, and demonstrates semantic search. ```bash python main.py ``` -------------------------------- ### Clone Repository and Navigate Source: https://github.com/pingcap/pytidb/blob/main/examples/rag/README.md Clones the PyTiDB repository from GitHub and navigates into the specific RAG example directory. This is the initial step to get the project files locally. ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/rag/ ``` -------------------------------- ### Clone PyTiDB Repository Source: https://github.com/pingcap/pytidb/blob/main/examples/text2sql/README.md Clones the pytidb repository and navigates into the text2sql examples directory. This is the first step to set up the project locally. ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/text2sql/ ``` -------------------------------- ### Run Streamlit Application Source: https://github.com/pingcap/pytidb/blob/main/examples/vector_search/README.md Starts the Streamlit web application, which provides the user interface for the vector search example. ```bash streamlit run app.py ``` -------------------------------- ### PyTiDB Quickstart Demo Expected Output (Plain Text) Source: https://github.com/pingcap/pytidb/blob/main/examples/quickstart/README.md Shows the expected output when the PyTiDB quickstart demo script is successfully executed. It details the connection status, table operations, data insertion, query results, and semantic search distances. ```plain text === Connect to TiDB === Connected to TiDB === Create embedding function === Embedding function created === Create table === Table created === Truncate table === Table truncated === Insert data === Inserted 3 chunks === Query data === ID: 1, Text: PyTiDB is a Python library for developers to connect to TiDB., User ID: 2 ID: 2, Text: LlamaIndex is a framework for building AI applications., User ID: 2 ID: 3, Text: OpenAI is a company and platform that provides AI models service and tools., User ID: 3 === Semantic search === ID: 2, Text: LlamaIndex is a framework for building AI applications., User ID: 2, Distance: 0.5720575919048316 ID: 3, Text: OpenAI is a company and platform that provides AI models service and tools., User ID: 3, Distance: 0.6032368346515378 ID: 1, Text: PyTiDB is a Python library for developers to connect to TiDB., User ID: 2, Distance: 0.6203520237350386 === Delete a row === Deleted chunk #1 === Drop table === Table dropped ``` -------------------------------- ### Install Dependencies Source: https://github.com/pingcap/pytidb/blob/main/examples/image_search/README.md Sets up a Python virtual environment and installs all necessary packages listed in `reqs.txt`. Ensure Python 3.10+ is installed. ```bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r reqs.txt ``` -------------------------------- ### Run Streamlit Application Source: https://github.com/pingcap/pytidb/blob/main/examples/text2sql/README.md Starts the Streamlit development server, making the web application accessible via a browser. Users can then interact with the application by providing API keys and connection strings. ```bash streamlit run main.py ``` -------------------------------- ### Clone Repository Source: https://github.com/pingcap/pytidb/blob/main/examples/image_search/README.md Clones the pyTiDB repository to your local machine. This is the first step to access the example code and setup files. ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/image_search/ ``` -------------------------------- ### Run the Streamlit Application Source: https://github.com/pingcap/pytidb/blob/main/examples/rag/README.md Starts the Streamlit web server to launch the RAG application's user interface. Users can then interact with the application through their web browser. ```bash streamlit run main.py ``` -------------------------------- ### Clone and Navigate PyTiDB Repository Source: https://github.com/pingcap/pytidb/blob/main/examples/basic/README.md Clones the PyTiDB repository from GitHub and navigates into the basic examples directory. Essential for setting up the project locally. ```Bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/basic/ ``` -------------------------------- ### Run Streamlit Application Source: https://github.com/pingcap/pytidb/blob/main/examples/fulltext_search/README.md Starts the Streamlit application, which serves as the user interface for the full-text search demo, typically accessible via a local web browser. ```bash streamlit run app.py ``` -------------------------------- ### Install Dependencies Source: https://github.com/pingcap/pytidb/blob/main/examples/memory/README.md Creates a Python virtual environment and installs all required packages listed in the 'reqs.txt' file. This ensures the project has all necessary libraries to run. ```bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r reqs.txt ``` -------------------------------- ### Clone Repository and Navigate Source: https://github.com/pingcap/pytidb/blob/main/examples/memory/README.md Clones the pytidb repository and navigates into the memory example directory. This is the initial step to set up the project locally. ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/memory/ ``` -------------------------------- ### Install pytidb Source: https://github.com/pingcap/pytidb/blob/main/README.md Installs the TiDB Python SDK. Additional packages can be installed for model support or pandas integration. ```bash pip install pytidb # To use built-in embedding functions and rerankers: pip install "pytidb[models]" # To convert query results to pandas DataFrame: pip install pandas ``` -------------------------------- ### Clone Repository and Navigate Directory Source: https://github.com/pingcap/pytidb/blob/main/examples/fulltext_search/README.md Clones the pytidb repository from GitHub and navigates into the specific full-text search example directory. ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/fulltext_search/ ``` -------------------------------- ### Clone and Install TiDB Python SDK Source: https://github.com/pingcap/pytidb/blob/main/CONTRIBUTING.md This snippet demonstrates how to clone the TiDB Python SDK repository from GitHub, navigate into the project directory, install the 'uv' package manager, and then install all development dependencies using 'uv sync --dev'. ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb pip install uv uv sync --dev ``` -------------------------------- ### Configure TiDB Cloud Connection (Bash) Source: https://github.com/pingcap/pytidb/blob/main/examples/quickstart/README.md Creates a .env file to configure environment variables for connecting to TiDB Cloud. This includes host, port, username, password, database name, and OpenAI API key. ```bash cat > .env < .env < .env < .env < .env < .env < .env < .env < .env < .env < EOF ``` -------------------------------- ### Configure TiDB Connection via .env Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Creates a `.env` file to store TiDB connection details and API keys for OpenAI and Jina AI. This method is used to manage sensitive configuration information securely and load it into the environment. ```bash # Check if the .env file is existing. if [ -f .env ]; then exit 0 fi # Create .env file with your configuration. replace the value with your saved key above. cat > .env < SQLExecuteResult`: Executes SQL statements that do not return a result set (e.g., INSERT, UPDATE, DELETE, CREATE TABLE). Supports parameterized queries using `:param_name` syntax. Example (raw SQL): `db.execute("INSERT INTO chunks(text, user_id) VALUES ('inserted from raw sql', 5)")` Example (parameterized SQL): `db.execute("INSERT INTO chunks(text, user_id) VALUES (:text, :user_id)", {"text": "inserted from dynamic sql", "user_id": 6})` - `db.query(sql: str, params: Optional[Dict] = None) -> SQLQueryResult`: Executes SQL statements that return a result set (e.g., SELECT, SHOW). Supports parameterized queries. The `SQLQueryResult` object provides helper methods for data retrieval: - `to_pandas()`: Converts the result to a pandas DataFrame. Example: `db.query("SELECT id, text, user_id FROM chunks").to_pandas()` - `to_list()`: Converts the result into a list of dictionaries. Example: `db.query("SELECT id, text, user_id FROM chunks WHERE user_id = :user_id", {"user_id": 3}).to_list()` - `to_rows()`: Returns a list of tuples, where each tuple represents a row. Example: `db.query("SHOW TABLES;").to_rows()` - `scalar()`: Returns the first column of the first row, useful for aggregate functions. Example: `db.query("SELECT COUNT(*) FROM chunks;").scalar()` ``` -------------------------------- ### Table Management Operations Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Provides methods for managing database tables, including listing all available table names, truncating a table to remove all its data, and dropping a table entirely. ```python db.table_names() ``` ```python table.truncate() table.rows() ``` ```python db.drop_table("chunks") ``` -------------------------------- ### PyTiDB Join Structured and Unstructured Data Source: https://github.com/pingcap/pytidb/blob/main/README.md Demonstrates how to join structured data (User table) with unstructured data (Chunk table) using PyTiDB's session and select capabilities. This allows for querying based on relationships between different data types. ```Python from pytidb import Session from pytidb.sql import select # Create a table to store user data: class User(TableModel): __tablename__ = "users" id: int = Field(primary_key=True) name: str = Field(max_length=20) with Session(engine) as session: query = ( select(Chunk).join(User, Chunk.user_id == User.id).where(User.name == "Alice") ) chunks = session.exec(query).all() [(c.id, c.text, c.user_id) for c in chunks] ``` -------------------------------- ### Join Chunk and User Tables Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Executes a SQL query to join the 'chunks' table with the 'users' table on 'user_id' and 'id' respectively, filtering by user name. It retrieves chunk details for users named 'Alice'. ```python from pytidb import Session from pytidb.sql import select db_engine = db.db_engine with Session(db_engine) as db_session: query = ( select(Chunk).join(User, Chunk.user_id == User.id).where(User.name == "Alice") ) chunks = db_session.exec(query).all() [(c.id, c.text, c.user_id) for c in chunks] ``` -------------------------------- ### TiDB Client Filtering Operators Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Describes the available filter operators for flexible querying in the TiDB client. These operators allow for precise data selection based on various conditions like equality, range, and array membership. ```APIDOC TiDB Client Filtering Operators: - `$eq`: Equal to. Example: `{"field": {"$eq": "hello"}}` - `$gt`: Greater than. Example: `{"field": {"$gt": 1}}` - `$gte`: Greater than or equal. Example: `{"field": {"$gte": 1}}` - `$lt`: Less than. Example: `{"field": {"$lt": 1}}` - `$lte`: Less than or equal. Example: `{"field": {"$lte": 1}}` - `$in`: In array. Example: `{"field": {"$in": [1, 2, 3]}}` - `$nin`: Not in array. Example: `{"field": {"$nin": [1, 2, 3]}}` - `$and`: Logical AND. Example: `{"$and": [{"field1": 1}, {"field2": 2}]}` - `$or`: Logical OR. Example: `{"$or": [{"field1": 1}, {"field2": 2}]}` ``` -------------------------------- ### PyTiDB Advanced Filtering Operators Source: https://github.com/pingcap/pytidb/blob/main/README.md Provides a reference for PyTiDB's advanced filtering operators used in queries, including logical and comparison operators. These operators allow for flexible data filtering based on various conditions. ```APIDOC PyTiDB Filtering Operators: - `$eq`: Equal to Example: `{"field": {"$eq": "hello"}}` - `$gt`: Greater than Example: `{"field": {"$gt": 1}}` - `$gte`: Greater than or equal Example: `{"field": {"$gte": 1}}` - `$lt`: Less than Example: `{"field": {"$lt": 1}}` - `$lte`: Less than or equal Example: `{"field": {"$lte": 1}}` - `$in`: In array Example: `{"field": {"$in": [1, 2, 3]}}` - `$nin`: Not in array Example: `{"field": {"$nin": [1, 2, 3]}}` - `$and`: Logical AND Example: `{"$and": [{"field1": 1}, {"field2": 2}]}` - `$or`: Logical OR Example: `{"$or": [{"field1": 1}, {"field2": 2}]}` ``` -------------------------------- ### Full-text Search Source: https://github.com/pingcap/pytidb/blob/main/README.md Executes a full-text search by tokenizing the query and matching exact keywords. Results can be returned as pydantic models. ```python df = ( table.search("", search_type="fulltext") .limit(2) .to_pydantic() ) # Output: A list of pydantic model instances. ``` -------------------------------- ### Perform Full-Text Search Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Conducts a full-text search by tokenizing the query and retrieving records that match the keywords. This search type is useful for keyword-based retrieval and relies on the FTS index for efficiency. ```python res = ( table.search( "A library for my artificial intelligence software", search_type="fulltext" ) .limit(3) .to_pandas() ) res ``` -------------------------------- ### Bulk Insert Data Source: https://github.com/pingcap/pytidb/blob/main/README.md Inserts multiple records into a table in bulk. The SDK automatically handles embedding generation for vector fields. ```python table.bulk_insert([ Chunk(id=2, text="bar", user_id=2), # 👈 The text field is embedded and saved to text_vec automatically. Chunk(id=3, text="baz", user_id=3), Chunk(id=4, text="qux", user_id=4), ]) ``` -------------------------------- ### Perform Hybrid Search with Reranking Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Combines vector and full-text search for more relevant results, then applies a reranker model (e.g., Jina AI) to further refine the order of results based on semantic relevance. The final results are returned as a pandas DataFrame. ```python from pytidb.rerankers import Reranker jinaai = Reranker(model_name="jina_ai/jina-reranker-m0") res = ( table.search( "A library for my artificial intelligence software", search_type="hybrid" ) .rerank(jinaai, "text") .limit(3) .to_pandas() ) res ``` -------------------------------- ### Create Full-Text Search Index Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Adds a full-text search (FTS) index to a specified column in the table. This enhances the performance and relevance of full-text search queries on that column. ```python if not table.has_fts_index("text"): table.create_fts_index("text") ``` -------------------------------- ### Define Table Schema with Vector Field Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Defines a database table schema using `TableModel` and `Field` from `pytidb.schema`. It includes a `VectorField` which automatically generates embeddings for a specified source text field using a configured embedding model. ```python from pytidb.schema import TableModel, Field from pytidb.embeddings import EmbeddingFunction # Define your embedding model. text_embed = EmbeddingFunction("openai/text-embedding-3-small") class Chunk(TableModel, table=True): __tablename__ = "chunks" __table_args__ = {"extend_existing": True} id: int = Field(primary_key=True) text: str = Field() text_vec: list[float] = text_embed.VectorField( source_field="text" ) user_id: int = Field() ``` -------------------------------- ### Retrieve and Update Chunk Data Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Shows how to retrieve a specific chunk by its text content, access its properties, and then update only the 'text' field of that chunk. The vector field is automatically updated. ```python old_chunk = table.query({"text": "PyTiDB is a Python library for developers to connect to TiDB."}) chunk_id = old_chunk[0].id (old_chunk[0].text, old_chunk[0].text_vec) ``` ```python table.update( values={ "text": "foo" # 👈 Only provide the fields you want to update. }, filters={"id": chunk_id}, ) ``` ```python new_chunk = table.get(chunk_id) (new_chunk.text, new_chunk.text_vec) # 👈 The vector field is updated automatically. ``` -------------------------------- ### Delete Data by Filter Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Demonstrates deleting records from the 'chunks' table based on a filter condition, specifically removing chunks associated with 'user_id' equal to 2. It also shows how to list rows before and after deletion. ```python table.rows() ``` ```python table.delete(filters={"user_id": 2}) ``` ```python table.rows() ``` -------------------------------- ### Insert Data with Auto Embedding Source: https://github.com/pingcap/pytidb/blob/main/docs/quickstart.ipynb Inserts single or multiple records into the TiDB table. The SDK automatically generates vector embeddings for the `text` field when inserting data, provided a `VectorField` is defined in the schema. ```python from asyncio import sleep table.truncate() await sleep(3) table.insert( Chunk( text="TiDB is a distributed database that supports OLTP, OLAP, HTAP and AI workloads.", user_id=1, ), ) table.bulk_insert( [ Chunk( text="PyTiDB is a Python library for developers to connect to TiDB.", user_id=2, ), Chunk( text="LlamaIndex is a framework for building AI applications.", user_id=2 ), Chunk( text="OpenAI is a company and platform that provides AI models service and tools.", user_id=3, ), ] ) table.rows() ``` -------------------------------- ### Define Table with Embedding Source: https://github.com/pingcap/pytidb/blob/main/README.md Defines a table schema using `TableModel` and automatically generates vector embeddings for a specified text field using an `EmbeddingFunction`. ```python from pytidb.schema import TableModel, Field, FullTextField from pytidb.embeddings import EmbeddingFunction text_embed = EmbeddingFunction("openai/text-embedding-3-small") class Chunk(TableModel): __tablename__ = "chunks" id: int = Field(primary_key=True) text: str = FullTextField() text_vec: list[float] = text_embed.VectorField( source_field="text" ) # 👈 Defines the vector field. user_id: int = Field() table = db.create_table(schema=Chunk, mode="exist_ok") ``` -------------------------------- ### Hybrid Search Source: https://github.com/pingcap/pytidb/blob/main/README.md Combines full-text search with vector search for more relevant results. Results can be converted to a pandas DataFrame. ```python df = ( table.search("", search_type="hybrid") .limit(2) .to_pandas() ) ```