### Start DocsGPT Services Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/use_milvus_in_docsgpt.md Run the setup script to start the DocsGPT services. Access the UI at http://localhost:5173/ to interact with your documents. ```shell ./setup.sh ``` -------------------------------- ### Download Example Data Files Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/tutorials/contextual_retrieval_with_milvus.md Use these wget commands to download the necessary JSON and JSONL files for contextual retrieval examples. Ensure you have wget installed and internet connectivity. ```shell $ wget https://raw.githubusercontent.com/anthropics/anthropic-cookbook/refs/heads/main/skills/contextual-embeddings/data/codebase_chunks.json $ wget https://raw.githubusercontent.com/anthropics/anthropic-cookbook/refs/heads/main/skills/contextual-embeddings/data/evaluation_set.jsonl ``` -------------------------------- ### Install Milvus Go SDK Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/getstarted/install_SDKs/install-go.md Install the Milvus GO SDK and its dependencies using the `go get` command. Ensure you have Go version 1.17 or later installed. ```bash $ go get -u github.com/milvus-io/milvus/client/v2 ``` -------------------------------- ### Run Milvus Example Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/adminGuide/clouds/aws/eks.md Execute the Python script to test the Milvus installation. ```shell python3 hello_milvus.py ``` -------------------------------- ### Start Milvus Docker Container Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/getstarted/run-milvus-docker/install_standalone-docker.md Execute the downloaded installation script to start the Milvus Docker container. ```shell bash standalone_embed.sh start ``` -------------------------------- ### Start a Minikube Kubernetes Cluster Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/fragments/create_a_k8s_cluster_using_minikube.md Run this command after installing Minikube to initiate a local Kubernetes cluster. This is intended for testing environments only. ```bash $ minikube start ``` -------------------------------- ### Create an Index Interactively Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/tools/cli_commands.md This interactive example guides you through creating an index for a field in a collection. It prompts for necessary details like collection name, field, index name, index type, and metric type. ```shell milvus_cli > create index Collection name (car, car2): car The name of the field to create an index for (vector): vector Index name: vectorIndex Index type (FLAT, IVF_FLAT, IVF_SQ8, IVF_PQ, HNSW, AUTOINDEX, DISKANN, GPU_IVF_FLAT, GPU_IVF_PQ, SPARSE_INVERTED_INDEX, SCANN, STL_SORT, Trie, INVERTED): IVF_FLAT Vector Index metric type (L2, IP, HAMMING, TANIMOTO, COSINE): L2 Index params nlist: 2 Timeout []: ``` -------------------------------- ### Start Milvus Standalone Service Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/getstarted/run-milvus-docker/install_standalone-binary.md Start the Milvus standalone service after installation. This command assumes Milvus is installed as a systemd service. ```shell systemctl start milvus ``` -------------------------------- ### Set up Sample SQLite Database Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/integrate_with_vanna.md Create and populate a sample SQLite database for demonstration purposes. This includes defining table schemas and inserting initial data. ```python import sqlite3 sqlite_path = "./my-database.sqlite" sql_connect = sqlite3.connect(sqlite_path) c = sql_connect.cursor() init_sqls = """ CREATE TABLE IF NOT EXISTS Customer ( ID INTEGER PRIMARY KEY AUTOINCREMENT, Name TEXT NOT NULL, Company TEXT NOT NULL, City TEXT NOT NULL, Phone TEXT NOT NULL ); CREATE TABLE IF NOT EXISTS Company ( ID INTEGER PRIMARY KEY AUTOINCREMENT, Name TEXT NOT NULL, Industry TEXT NOT NULL, Location TEXT NOT NULL, EmployeeCount INTEGER NOT NULL ); CREATE TABLE IF NOT EXISTS User ( ID INTEGER PRIMARY KEY AUTOINCREMENT, Username TEXT NOT NULL UNIQUE, Email TEXT NOT NULL UNIQUE ); INSERT INTO Customer (Name, Company, City, Phone) VALUES ('John Doe', 'ABC Corp', 'New York', '123-456-7890'); INSERT INTO Customer (Name, Company, City, Phone) VALUES ('Jane Smith', 'XYZ Inc', 'Los Angeles', '098-765-4321'); INSERT INTO Company (Name, Industry, Location, EmployeeCount) VALUES ('ABC Corp', 'cutting-edge technology', 'New York', 100); INSERT INTO User (Username, Email) VALUES ('johndoe123', 'johndoe123@example.com'); """ for sql in init_sqls.split(";"): c.execute(sql) sql_connect.commit() # Connect to the SQLite database vn_milvus.connect_to_sqlite(sqlite_path) ``` -------------------------------- ### Install Python and Dependencies Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/build_rag_on_arm.md Installs Python 3 and pip, then creates and activates a virtual environment. Finally, it installs necessary Python dependencies for the RAG application. ```bash sudo apt update sudo apt install python-is-python3 python3-pip python3-venv -y ``` ```bash python -m venv venv source venv/bin/activate ``` ```shell pip install --upgrade pymilvus openai requests langchain-huggingface huggingface_hub tqdm ``` -------------------------------- ### Crawl4AI Setup and Verification Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/build_RAG_with_milvus_and_crawl4ai.md Perform post-installation setup for Crawl4AI and verify its installation and basic functionality with a health check. This includes browser setup and database initialization. ```shell # Run post-installation setup $ crawl4ai-setup # Verify installation $ crawl4ai-doctor ``` ```shell [36m[INIT].... → Running post-installation setup... [0m [36m[INIT].... → Installing Playwright browsers... [0m [32m[COMPLETE] ● Playwright installation completed successfully. [0m [36m[INIT].... → Starting database initialization... [0m [32m[COMPLETE] ● Database initialization completed successfully. [0m [32m[COMPLETE] ● Post-installation setup completed! [0m [0m [36m[INIT].... → Running Crawl4AI health check... [0m [36m[INIT].... → Crawl4AI 0.4.247 [0m [36m[TEST].... ℹ Testing crawling capabilities... [0m [36m[EXPORT].. ℹ Exporting PDF and taking screenshot took 0.80s [0m [32m[FETCH]... ↓ https://crawl4ai.com... | Status: [32mTrue [0m | Time: 4.22s [0m [36m[SCRAPE].. ◆ Processed https://crawl4ai.com... | Time: 14ms [0m [32m[COMPLETE] ● https://crawl4ai.com... | Status: [32mTrue [0m | Total: [33m4.23s [0m [0m [32m[COMPLETE] ● ✅ Crawling test passed! [0m [0m ``` -------------------------------- ### Create and List Partitions in Go Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/collections/manage-partitions.md Illustrates creating a partition and listing all partitions for a collection using the Milvus Go SDK. Ensure context is properly managed. ```go import ( "fmt" client "github.com/milvus-io/milvus/client/v2/milvusclient" ) ctx, cancel := context.WithCancel(context.Background()) deferr cancel() err = client.CreatePartition(ctx, milvusclient.NewCreatePartitionOption("my_collection", "partitionA")) if err != nil { fmt.Println(err.Error()) // handle error } partitionNames, err := client.ListPartitions(ctx, milvusclient.NewListPartitionOption("my_collection")) if err != nil { fmt.Println(err.Error()) // handle error } fmt.Println(partitionNames) // Output // ["_default", "partitionA"] ``` -------------------------------- ### Create and List Partitions in Java Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/collections/manage-partitions.md Demonstrates creating a partition named 'partitionA' and then listing all partitions for 'my_collection'. Requires Milvus client library. ```java import io.milvus.v2.service.partition.request.CreatePartitionReq; CreatePartitionReq createPartitionReq = CreatePartitionReq.builder() .collectionName("my_collection") .partitionName("partitionA") .build(); client.createPartition(createPartitionReq); ListPartitionsReq listPartitionsReq = ListPartitionsReq.builder() .collectionName("my_collection") .build(); List partitionNames = client.listPartitions(listPartitionsReq); System.out.println(partitionNames); // Output: // [_default, partitionA] ``` -------------------------------- ### Example Output for List Collections Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/collections/view-collections.md This JSON output shows a sample result when listing collections, indicating the presence of a collection named 'quick_setup'. ```json ["quick_setup"] ``` -------------------------------- ### MilvusClient with Partitions Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/using_milvusclient.md Demonstrates initializing MilvusClient with partitions, inserting data into specific partitions, adding new partitions, and searching within partitions. ```python from pymilvus import MilvusClient client = MilvusClient( collection_name="qux", uri="http://localhost:19530", vector_field="float_vector", partitions = ["zaz"], overwrite=True, ) data = [ { "float_vector": [1,2,3], "id": 1, "text": "foo" }, ] client.insert_data(data, partition="zaz") client.add_partitions(["zoo"]) data = [ { "float_vector": [4,5,6], "id": 2, "text": "bar" }, ] client.insert_data(data, partition="zoo") res = client.search_data( data = [1,3,5], top_k = 2, ) # [[ # {'data': {'id': 1, 'internal_pk_3bd4': 441363276234227849, 'text': 'foo'}, 'score': 5.0}, # {'data': {'id': 2, 'internal_pk_3bd4': 441363276234227866, 'text': 'bar'}, 'score': 14.0} # ]] res = client.search_data( data = [1,3,5], top_k = 2, partitions=["zaz"] ) # [[ # {'data': {'id': 1, 'internal_pk_3bd4': 441363276234227849, 'text': 'foo'}, 'score': 5.0} # ]] res = client.search_data( data = [1,3,5], top_k = 2, partitions=["zoo"] ) # [[ # {'data': {'id': 2, 'internal_pk_3bd4': 441363276234227866, 'text': 'bar'}, 'score': 14.0} # ]] ``` -------------------------------- ### Start Milvus Standalone with Docker Compose Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/adminGuide/configure-docker.md After modifying the installation and configuration files, use this command to start the Milvus standalone service in detached mode. ```bash $ sudo docker compose up -d ``` -------------------------------- ### Get Dataset Size Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/AIMon_milvus_integration.md Calculates and prints the number of examples in the loaded dataset. ```python len(train_split) ``` -------------------------------- ### Example Output for Describe Collection Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/collections/view-collections.md This plaintext output details the schema and properties of the 'quick_setup' collection, including fields, dimensions, and consistency level. ```plaintext { 'collection_name': 'quick_setup', 'auto_id': False, 'num_shards': 1, 'description': '', 'fields': [ { 'field_id': 100, 'name': 'id', 'description': '', 'type': , 'params': {}, 'is_primary': True }, { 'field_id': 101, 'name': 'vector', 'description': '', 'type': , 'params': {'dim': 768} } ], 'functions': [], 'aliases': [], 'collection_id': 456909630285026300, 'consistency_level': 2, 'properties': {}, 'num_partitions': 1, 'enable_dynamic_field': True } ``` -------------------------------- ### Create and List Partitions in JavaScript Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/collections/manage-partitions.md Shows how to create a partition and then list all partitions for a collection using the Milvus JavaScript SDK. ```javascript await client.createPartition({ collection_name: "my_collection", partition_name: "partitionA" }) res = await client.listPartitions({ collection_name: "my_collection" }) console.log(res) // Output // ["_default", "partitionA"] ``` -------------------------------- ### Provide Examples for LangExtract Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/langextract_milvus_demo.md Supplies n-shot examples to LangExtract to guide the model's extraction process for movie descriptions. This helps ensure consistency and accuracy in the extracted data. ```python # Provide examples to guide the model - n-shot examples for movie descriptions # Unify attribute keys to ensure consistency in extraction results examples = [ lx.data.ExampleData( text="A space marine battles alien creatures on a distant planet. The sci-fi action movie features futuristic weapons and intense combat scenes.", extractions=[ lx.data.Extraction( extraction_class="genre", extraction_text="sci-fi action", attributes={"primary_genre": "sci-fi", "secondary_genre": "action"}, ), lx.data.Extraction( extraction_class="character", extraction_text="space marine", attributes={"role": "protagonist", "type": "military"}, ), lx.data.Extraction( extraction_class="theme", extraction_text="battles alien creatures", attributes={"theme_type": "conflict", "setting": "space"}, ), ], ), lx.data.ExampleData( text="A detective investigates supernatural murders in Victorian London. The horror thriller film combines period drama with paranormal elements.", extractions=[ lx.data.Extraction( extraction_class="genre", extraction_text="horror thriller", attributes={"primary_genre": "horror", "secondary_genre": "thriller"}, ), lx.data.Extraction( extraction_class="character", extraction_text="detective", attributes={"role": "protagonist", "type": "detective"}, ), lx.data.Extraction( extraction_class="theme", extraction_text="supernatural murders", attributes={"theme_type": "investigation", "setting": "victorian"}, ), ], ), lx.data.ExampleData( text="Two friends embark on a road trip adventure across America. The comedy drama explores friendship and self-discovery through humorous situations.", extractions=[ lx.data.Extraction( extraction_class="genre", extraction_text="comedy drama", attributes={"primary_genre": "comedy", "secondary_genre": "drama"}, ), lx.data.Extraction( extraction_class="character", extraction_text="two friends", attributes={"role": "protagonist", "type": "friends"}, ), lx.data.Extraction( extraction_class="theme", extraction_text="friendship and self-discovery", attributes={"theme_type": "personal_growth", "setting": "america"}, ), ], ), ] ``` -------------------------------- ### List and Describe Databases Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/manage_databases.md Demonstrates how to list all available databases and describe a specific database using Java, NodeJS, Go, and cURL. ```java import io.milvus.v2.service.database.response.*; ListDatabasesResp listDatabasesResp = client.listDatabases(); DescribeDatabaseResp descDBResp = client.describeDatabase(DescribeDatabaseReq.builder() .databaseName("default") .build()); ``` ```javascript await client.describeDatabase({ db_name: 'default' }); ``` ```go // List all existing databases databases, err := cli.ListDatabase(ctx, milvusclient.NewListDatabaseOption()) if err != nil { // handle err } log.Println(databases) db, err := cli.DescribeDatabase(ctx, milvusclient.NewDescribeDatabaseOption("default")) if err != nil { // handle err } log.Println(db) ``` ```bash export CLUSTER_ENDPOINT="http://localhost:19530" export TOKEN="root:Milvus" curl --request POST \ --url "${CLUSTER_ENDPOINT}/v2/vectordb/databases/describe" \ --header "Authorization: Bearer ${TOKEN}" \ --header "Content-Type: application/json" \ --header "Request-Timeout: 10" \ -d '{ "dbName": "default" }' ``` -------------------------------- ### Milvus Quick Start: Create, Insert, and Search Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/aiTools/agents_overview.md This snippet demonstrates the basic workflow of using MilvusClient to create a collection with a schema and index, insert data, and perform a search. Ensure Milvus is running and accessible at the specified URI. ```python from pymilvus import MilvusClient, DataType client = MilvusClient(uri="http://localhost:19530") # 1. Define schema schema = client.create_schema(auto_id=True) schema.add_field("id", DataType.INT64, is_primary=True) schema.add_field("vector", DataType.FLOAT_VECTOR, dim=768) schema.add_field("text", DataType.VARCHAR, max_length=512) # 2. Define index index_params = client.prepare_index_params() index_params.add_index(field_name="vector", index_type="AUTOINDEX", metric_type="COSINE") # 3. Create collection (auto-indexes and auto-loads) client.create_collection(collection_name="docs", schema=schema, index_params=index_params) # 4. Insert client.insert(collection_name="docs", data=[ {"vector": [0.1] * 768, "text": "first doc"}, {"vector": [0.2] * 768, "text": "second doc"}, ]) # 5. Search results = client.search( collection_name="docs", data=[[0.15] * 768], limit=5, output_fields=["text"], ) ``` -------------------------------- ### Specify Milvus Server URI Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/langchain/milvus_hybrid_search_retriever.md Defines the Uniform Resource Identifier (URI) for the Milvus server. Ensure your Milvus server is installed and running, following the provided installation guide. ```python URI = "http://localhost:19530" ``` -------------------------------- ### Example Process Output Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/reference/coordinator_ha.md This is an example output from `ps aux|grep milvus`, showing multiple Milvus processes, including coordinators. ```shell > ps aux|grep milvus root 12813 0.7 0.2 410709648 82432 ?? S 5:18PM 0:33.28 ./bin/milvus run rootcoord root 12816 0.5 0.2 409487968 62352 ?? S 5:18PM 0:22.69 ./bin/milvus run proxy root 17739 0.1 0.3 410289872 91792 s003 SN 6:01PM 0:00.30 ./bin/milvus run rootcoord ... ``` -------------------------------- ### Get Snapshot Metadata for Spark Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/snapshot-use-cases.md Retrieve snapshot metadata to locate data files in object storage, essential for offline data processing with Spark. This example shows how to get metadata and construct an S3 path. ```python # Get snapshot metadata snapshot_info = client.describe_snapshot( snapshot_name=s"analytics_snapshot_20260321", include_collection_info=True ) # Locate data files in S3 s3_path = f"s3a://{snapshot_info.s3_location}/binlogs/" ``` -------------------------------- ### Get Milvus Manifest Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md Renders Milvus cluster chart templates and saves them to a YAML manifest file for installation. ```shell helm template my-release zilliztech/milvus > milvus_manifest.yaml ``` -------------------------------- ### Example Describe User Output Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/adminGuide/grant_roles.md This is an example of the output format when describing a user's roles. ```bash {'user_name': 'user_1', 'roles': 'role_a'} ``` -------------------------------- ### Milvus Pod Status Output Example Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md Example output of `kubectl get pods` showing the status of various Milvus cluster components, including etcd, MinIO, datanode, mixcoord, proxy, querynode, and streamingnode. All should be in 'Running' state. ```shell NAME READY STATUS RESTARTS AGE my-release-etcd-0 1/1 Running 0 2m36s my-release-etcd-1 1/1 Running 0 2m36s my-release-etcd-2 1/1 Running 0 2m36s my-release-milvus-datanode-58955c65b9-j4j7s 1/1 Running 0 92s my-release-milvus-mixcoord-686f84968f-jcv5d 1/1 Running 0 92s my-release-milvus-proxy-646f48fc7c-4lctb 1/1 Running 0 92s my-release-milvus-querynode-0-d89d7677b-x7j7q 1/1 Running 0 91s my-release-milvus-streamingnode-556bdcc87c-2qwcc 1/1 Running 0 92s my-release-minio-0 1/1 Running 0 2m36s my-release-minio-1 1/1 Running 0 2m36s my-release-minio-2 1/1 Running 0 2m35s my-release-minio-3 1/1 Running 0 2m35s ``` -------------------------------- ### Basic Milvus Workflow Example Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/aiTools/python_sdk.md A complete example demonstrating the basic workflow: connecting to Milvus, defining collection parameters, and dropping the collection if it already exists. ```python from pymilvus import MilvusClient, DataType # Connect to Milvus client = MilvusClient( uri="YOUR_MILVUS_URI", token="YOUR_MILVUS_TOKEN" ) COLLECTION_NAME = "my_collection" DIMENSION = 768 # Drop the collection if it already exists if client.has_collection(COLLECTION_NAME): client.drop_collection(COLLECTION_NAME) ``` -------------------------------- ### Get Query Segment Information (Python) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/load_balance.md Use this snippet to retrieve segment information for a specified collection in Python. Ensure the pymilvus library is installed. ```python from pymilvus import utility utility.get_query_segment_info("book") ``` -------------------------------- ### Load Example Data Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/llamaindex_milvus_hybrid_search.md Download and load sample documents from a specified URL into the './data/paul_graham/' directory using SimpleDirectoryReader. ```shell mkdir -p 'data/paul_graham/' wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt' ``` -------------------------------- ### Run Standard Analyzer with Configuration (Python) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/schema/analyzer/tokenizer/standard-tokenizer.md Demonstrates how to run the standard analyzer with a given text and configuration in Python. ```python result = client.run_analyzer(sample_text, analyzer_params) print("English analyzer output:", result) ``` -------------------------------- ### HDF5 to Milvus Migration Configuration (Data Directory) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/adminGuide/backup/h2m.md Example YAML configuration for migrating HDF5 data using 'data_dir'. Ensure MilvusDM is installed and configured. ```yaml H2M: milvus_version: 2.0.0 data_path: data_dir: '/Users/zilliz/HDF5_data' dest_host: '127.0.0.1' dest_port: 19530 mode: 'append' # 'skip/append/overwrite' dest_collection_name: 'test_binary' dest_partition_name: collection_parameter: dimension: 512 index_file_size: 1024 metric_type: 'HAMMING' ``` -------------------------------- ### HDF5 to Milvus Migration Configuration (Data Path) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/adminGuide/backup/h2m.md Example YAML configuration for migrating HDF5 data using 'data_path'. Ensure MilvusDM is installed and configured. ```yaml H2M: milvus-version: 2.0.0 data_path: - /Users/zilliz/float_1.h5 - /Users/zilliz/float_2.h5 data_dir: dest_host: '127.0.0.1' dest_port: 19530 mode: 'overwrite' # 'skip/append/overwrite' dest_collection_name: 'test_float' dest_partition_name: 'partition_1' collection_parameter: dimension: 128 index_file_size: 1024 metric_type: 'L2' ``` -------------------------------- ### Connect to Milvus Server with Authentication (Go) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/getstarted/connect-to-milvus-server.md Initialize a Milvus client in Go with username and password. Ensure the Go SDK is installed. ```go import "github.com/milvus-io/milvus/client/v2/milvusclient" c, err := milvusclient.New(ctx, &milvusclient.ClientConfig{ Address: "localhost:19530", Username: "root", Password: "Milvus", }) ``` -------------------------------- ### Clone PII Masker Repository Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/RAG_with_pii_and_milvus.md Clone the PII Masker repository from GitHub to get started. This step is necessary to access the tool's code and models. ```shell git clone https://github.com/HydroXai/pii-masker-v1.git cd pii-masker-v1/pii-masker ``` -------------------------------- ### Search with Keyword Match (Go) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/keyword-match.md Go examples demonstrate setting a TEXT_MATCH filter for search operations using the milvusclient. ```go filter := "TEXT_MATCH(text, 'keyword1 keyword2')" resultSets, err := client.Search(ctx, milvusclient.NewSearchOption( "my_collection", // collectionName 10, // limit []entity.Vector{entity.FloatVector(queryVector)}). WithANNSField("embeddings"). WithFilter(filter). WithOutputFields("id", "text")) if err != nil { fmt.Println(err.Error()) // handle error } ``` -------------------------------- ### Get Milvus values.yaml for Helm Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/adminGuide/use-pulsar-v2.md Retrieves the current values.yaml file for a Milvus instance installed via Helm. This is the first step before modifying configurations for Pulsar version. ```bash namespace=default release=my-release helm -n ${namespace} get values ${release} -o yaml > values.yaml cat values.yaml ``` -------------------------------- ### Upgrade Milvus with Specific Command-Line Flags Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/adminGuide/configure-helm.md Configure and start Milvus by directly setting parameters using the `--set` flag during the upgrade process. This example disables compaction. ```bash # For instance, upgrade the Milvus cluster with compaction disabled helm upgrade my-release milvus/milvus --set dataCoord.enableCompaction=false ``` -------------------------------- ### Connect to Milvus Server (Go) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/getstarted/connect-to-milvus-server.md Initialize a Milvus client in Go using the server address. Ensure the Go SDK is installed. ```go import "github.com/milvus-io/milvus/client/v2/milvusclient" c, err := milvusclient.New(ctx, &milvusclient.ClientConfig{ Address: "localhost:19530", }) ``` -------------------------------- ### Markdown Front-Matter Example Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/CONTRIBUTING.md Pages in Milvus documentation must start with a front-matter block containing essential metadata like ID, title, and summary. The 'id' field is mandatory. ```markdown --- id: page_id.md title: Title of Page summary: Short description of the page for SEO purposes related_key: keyword group: major_page.md --- ``` -------------------------------- ### Example: Data Exploration with MilvusClient (Python) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/boolean/random-sampling.md Instantiate MilvusClient to perform data exploration using random sampling. Ensure Milvus is running at the specified URI. ```python from pymilvus import MilvusClient client = MilvusClient(uri="http://localhost:19530") ``` -------------------------------- ### Example Highlighted Output for Filtering Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/text-highlighter.md Illustrates the JSON output when query term highlighting is enabled for filtering. The 'highlight' field contains the annotated text, starting from the first matched term. ```json { ..., "highlight": { "text": [ "{text} {filtering} works in Milvus." ] } } ``` -------------------------------- ### Install NLWeb and Milvus Client Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/NLWeb_with_milvus.md Clone the NLWeb repository, set up a virtual environment, and install dependencies including the Milvus Python client. ```bash git clone https://github.com/microsoft/NLWeb cd NLWeb python -m venv .venv source .venv/bin/activate # or .venv\Scripts\activate on Windows cd code pip install -r requirements.txt pip install pymilvus # Add Milvus Python client ``` -------------------------------- ### Exact Match Count with MATCH_EXACT Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/boolean/struct-array-operators.md Use MATCH_EXACT to find elements where the count of items matching a predicate is exactly k. This example checks for exactly 3 chunks where the text starts with 'Red'. ```python MATCH_EXACT(chunks, $[text] LIKE 'Red%', 3) ``` -------------------------------- ### MilvusClient Connection Examples Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/aiTools/python_sdk.md Shows how to instantiate `MilvusClient` for both local, unauthenticated Milvus instances and for Zilliz Cloud or authenticated Milvus deployments using URI and token. ```python # Local Milvus client = MilvusClient(uri="http://localhost:19530") # Zilliz Cloud or authenticated Milvus client = MilvusClient( uri="YOUR_MILVUS_URI", token="YOUR_MILVUS_TOKEN" ) ``` -------------------------------- ### Initialize MilvusClient and Create Schema (Python) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/schema/analyzer/analyzer-overview.md Sets up a Milvus client and creates a new collection schema. Ensure the Milvus server is running and accessible at the specified URI. ```python from pymilvus import MilvusClient, DataType # Set up a Milvus client client = MilvusClient(uri="http://localhost:19530") # Create a new schema schema = client.create_schema(auto_id=True, enable_dynamic_field=False) ``` -------------------------------- ### Install Dependencies Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/use_milvus_with_sambanova.md Set up a Python virtual environment and install the required dependencies listed in the `requirements.txt` file. This ensures all necessary libraries are available for the application. ```shell python3 -m venv enterprise_knowledge_env source enterprise_knowledge_env/bin/activate pip install -r requirements.txt ``` -------------------------------- ### Generate Dense and Sparse Embeddings with BGE M3 Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/embeddings/embeddings.md Utilize the BGE M3 model to embed text into both dense and sparse vectors. Ensure `pymilvus[model]` is installed. This example prepares a corpus and a query for embedding. ```python from pymilvus.model.hybrid import BGEM3EmbeddingFunction from pymilvus import ( utility, FieldSchema, CollectionSchema, DataType, Collection, AnnSearchRequest, RRFRanker, connections, ) # 1. prepare a small corpus to search docs = [ "Artificial intelligence was founded as an academic discipline in 1956.", "Alan Turing was the first person to conduct substantial research in AI.", "Born in Maida Vale, London, Turing was raised in southern England.", ] query = "Who started AI research?" # BGE-M3 model can embed texts as dense and sparse vectors. # It is included in the optional `model` module in pymilvus, to install it, # simply run "pip install pymilvus[model]". bge_m3_ef = BGEM3EmbeddingFunction(use_fp16=False, device="cpu") docs_embeddings = bge_m3_ef(docs) query_embeddings = bge_m3_ef([query]) ``` -------------------------------- ### cURL: Query Entities in a Partition Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/get-and-scalar-query.md Shows how to perform a query with a filter and limit on a specific partition using the `/v2/vectordb/entities/get` endpoint. Note: The example uses the 'get' endpoint for query, which might be a typo in the source and should likely be '/v2/vectordb/entities/query'. ```bash # Use query curl --request POST \ --url "${CLUSTER_ENDPOINT}/v2/vectordb/entities/get" \ --header "Authorization: Bearer ${TOKEN}" \ --header "Content-Type: application/json" \ --header "Request-Timeout: 10" \ -d '{ "collectionName": "my_collection", "partitionNames": ["partitionA"], "filter": "color like \"red%\"", "limit": 3, "outputFields": ["vector", "color"], "id": [0, 1, 2] }' ``` -------------------------------- ### Create Index and Load Collection (Java) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/schema/nullable-and-default.md Shows how to create an index with specified parameters (field, name, type, metric) and then load the collection using the Java SDK. Imports for necessary classes are included. ```java import io.milvus.v2.common.IndexParam; import io.milvus.v2.service.collection.request.LoadCollectionReq; import io.milvus.v2.service.index.request.CreateIndexReq; import java.util.Collections; IndexParam indexParam = IndexParam.builder() .fieldName("embedding") .indexName("embedding_index") .indexType(IndexParam.IndexType.AUTOINDEX) .metricType(IndexParam.MetricType.COSINE) .build(); client.createIndex(CreateIndexReq.builder() .collectionName("my_collection") .indexParams(Collections.singletonList(indexParam)) .build()); client.loadCollection(LoadCollectionReq.builder() .collectionName("my_collection") .build()); ``` -------------------------------- ### Paginated Filtered Query with QueryIterator (Node.js) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/get-and-scalar-query.md Utilize `queryIterator` in Node.js to fetch entities based on filter expressions across multiple pages. This example queries for entities where 'color' starts with 'red', accumulating results. Requires `@zilliz/milvus2-sdk-node`. ```javascript import { MilvusClient, DataType } from "@zilliz/milvus2-sdk-node"; const iterator = await milvusClient.queryIterator({ collection_name: 'my_collection', batchSize: 10, expr: 'color like "red%"', output_fields: ['color'], }); const results = []; for await (const value of iterator) { results.push(...value); page += 1; } ``` -------------------------------- ### Copy Environment Example Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/dify_with_milvus.md Copy the example environment file to create a new environment configuration file for Dify. ```shell cp .env.example .env ``` -------------------------------- ### Milvus Hybrid Search with RRF Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/search-query-get/elasticsearch-queries-to-milvus.md Implements a hybrid search in Milvus by combining dense and sparse vector searches, then reranking the results using the RRF strategy. This example shows the setup for both search requests and the hybrid search execution. ```python search_params_dense = { "data": [[1.25, 2, 3.5]], "anns_field": "vector", "param": { "metric_type": "IP", "params": {"nprobe": 10}, }, "limit": 100 } req_dense = ANNSearchRequest(**search_params_dense) search_params_sparse = { "data": ["shoes"], "anns_field": "text_sparse", "param": { "metric_type": "BM25", } } req_sparse = ANNSearchRequest(**search_params_sparse) res = client.hybrid_search( collection_name="my_collection", reqs=[req_dense, req_sparse], reranker=RRFRanker(), limit=10 ) ``` -------------------------------- ### Prepare Sample Documents and Imports Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/full_text_search_with_milvus_and_haystack.md Imports necessary Haystack and Milvus components and defines sample documents for indexing. ```python from haystack import Pipeline from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder from haystack.components.writers import DocumentWriter from haystack.utils import Secret from milvus_haystack import MilvusDocumentStore, MilvusSparseEmbeddingRetriever from haystack.document_stores.types import DuplicatePolicy from milvus_haystack.function import BM25BuiltInFunction from milvus_haystack import MilvusDocumentStore from milvus_haystack.milvus_embedding_retriever import MilvusHybridRetriever from haystack.utils import Secret from haystack.components.builders import PromptBuilder from haystack.components.generators import OpenAIGenerator from haystack import Document documents = [ Document(content="Alice likes this apple", meta={"category": "fruit"}), Document(content="Bob likes swimming", meta={"category": "sport"}), Document(content="Charlie likes white dogs", meta={"category": "pets"}), ] ``` -------------------------------- ### Run Lindera Tokenizer with Korean Stop Tags (Node.js) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/schema/analyzer/tokenizer/lindera-tokenizer.md This Node.js example shows how to use the Milvus SDK to apply the Lindera tokenizer with Korean stop tags. It details the client setup and the structure of the analyzer parameters object. ```javascript import { MilvusClient } from "@zilliz.com/milvus2-sdk-node"; const client = new MilvusClient({ uri: "http://localhost:19530", }); const analyzer_params = { tokenizer: { type: "lindera", dict_kind: "ko-dic", filter: [ { kind: "korean_stop_tags", tags: [ "SP", "SSC", "SSO", "SC", "SE", "SF", "JKS", "JKC", "JKG", "JKO", "JKB", "JKV", "JKQ", "JX", "JC", "UNK", "EP", "ETM", ], }, ], }, }; const sample_text = "서울에서 맛있는 음식을 먹었습니다"; const result = await client.run_analyzer(sample_text, analyzer_params); console.log("Analyzer output:", result); ``` -------------------------------- ### Initialize Milvus Client in Go Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/insert-and-delete/insert-update-delete.md Sets up the context and client for interacting with Milvus. This is a prerequisite for performing operations like insert, update, or delete. ```go import ( "context" "fmt" "github.com/milvus-io/milvus/client/v2/column" "github.com/milvus-io/milvus/client/v2/milvusclient" ) ctx, cancel := context.WithCancel(context.Background()) def cancel() // Use defer cancel() to ensure cleanup milvusAddr := "localhost:19530" ``` -------------------------------- ### Apply SiliconFlow Ranker to Standard Vector Search (Node.js) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/embeddings-reranking/reranking/siliconflow-ranker.md Example demonstrating how to configure and use the SiliconFlow ranker within a Node.js application for standard vector search. Ensure Milvus SDK is installed and API keys are configured. ```javascript const milvus = require('milvus-sdk-node'); async function run() { // Connect to Milvus const client = new milvus.MilvusClient({ host: 'localhost', port: '19530' }); const collectionName = 'siliconflow_ranker_example'; // Load collection await client.collectionApi.loadCollection({ collection_name: collectionName }); // Define reranker parameters const rerankParams = { reranker: 'model', provider: 'siliconflow', model_name: 'BAAI/bge-reranker-v2-m3', queries: ['example search query'], max_client_batch_size: 64, max_chunks_per_doc: 5, overlap_tokens: 50, credential: 'your-siliconflow-api-key' }; // Define search parameters const searchParams = { collection_name: collectionName, vector_top_k: 10, vectors: [ { vector: Array(768).fill(0.1), // Assuming 'embeddings' is the name of the vector field // If using query vectors, specify field_name here } ], search_params: JSON.stringify({ metric_type: 'L2', params: { offset: 0, ignore_growing_segments: true } }), output_fields: ['content'], rerank_params: rerankParams }; // Perform search // const searchResult = await client.search(searchParams); // Release collection await client.collectionApi.releaseCollection({ collection_name: collectionName }); // Close connection client.close(); } run().catch(console.error); ``` -------------------------------- ### Apply SiliconFlow Ranker to Standard Vector Search (Python) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/embeddings-reranking/reranking/siliconflow-ranker.md Example demonstrating how to configure and use the SiliconFlow ranker within a Python application for standard vector search. Ensure necessary libraries are installed and API keys are configured. ```python from milvus import CollectionSchema, FieldSchema, DataType, Collection, connections # Connect to Milvus connections.connect("default", host="localhost", port="19530") # Define schema fields = [ FieldSchema(name="pk", dtype=DataType.INT64, is_primary=True, auto_increment=False), FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=768), FieldSchema(name="content", dtype=DataType.VARCHAR, max_length=500) ] schema = CollectionSchema(fields, "SiliconFlow Ranker Example") # Create collection collection = Collection(name="siliconflow_ranker_example", schema=schema) # Insert data # ... (data insertion logic here) # Define ranker parameters ranker_params = { "reranker": "model", "provider": "siliconflow", "model_name": "BAAI/bge-reranker-v2-m3", "queries": ["example search query"], "max_client_batch_size": 64, "max_chunks_per_doc": 5, "overlap_tokens": 50, "credential": "your-siliconflow-api-key" } # Perform search with reranking search_params = { "metric_type": "L2", "params": {"offset": 0, "ignore_growing_segments": True} } results = collection.search( data=[[0.1]*768], # Example search vector anns_field="embeddings", param=search_params, limit=10, expr=None, output_fields=["content"], rerank_params=ranker_params ) # Process results for hit in results[0]: print(f"Hit: {hit.entity.get('content')}, Score: {hit.score}") # Release collection collection.release() ``` -------------------------------- ### Create and Load Collection (Python) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/schema/nullable-and-default.md Demonstrates creating an index and loading a collection using the Python SDK. Ensure the collection and index names are correctly specified. ```python client.load_collection(collection_name="my_collection") ``` -------------------------------- ### Initialize OpenAI Client Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/integrations/build_RAG_with_milvus_and_embedAnything.md Sets up the OpenAI client by configuring the API key as an environment variable. Ensure your OPENAI_API_KEY is correctly set. ```python import os from openai import OpenAI os.environ["OPENAI_API_KEY"] = "sk-***********" openai_client = OpenAI() ``` -------------------------------- ### Load Documents into Milvus Vector Store Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/tutorials/full_text_search_with_milvus.md Initializes and loads prepared documents into a Milvus vector store. This setup includes both 'dense' (OpenAI embeddings) and 'sparse' (BM25) vector fields. Ensure Milvus and OpenAI libraries are installed and configured. ```python vectorstore = Milvus.from_documents( documents=docs, embedding=OpenAIEmbeddings(), builtin_function=BM25BuiltInFunction(), vector_field=["dense", "sparse"], connection_args={ "uri": URI, }, # Strong consistency waits for all loads to complete, adding latency with large datasets # consistency_level="Strong", # drop_old=True, ) ``` -------------------------------- ### Apply SiliconFlow Ranker to Standard Vector Search (Go) Source: https://github.com/milvus-io/milvus-docs/blob/v3.0.x/site/en/userGuide/embeddings-reranking/reranking/siliconflow-ranker.md Example demonstrating how to configure and use the SiliconFlow ranker within a Go application for standard vector search. Ensure Milvus Go SDK is installed and API keys are configured. ```go package main import ( "context" "fmt" "log" "github.com/milvus-io/milvus-sdk-go/v2/client" "github.com/milvus-io/milvus-sdk-go/v2/entity" "github.com/milvus-io/milvus-sdk-go/v2/param" ) func main() { // Connect to Milvus ctx := context.Background() milvusClient, err := client.NewMilvusServiceClient(ctx, client.Config{ Address: "localhost:19530", }) if err != nil { log.Fatalf("Failed to connect to Milvus: %v", err) } collectionName := "siliconflow_ranker_example" // Load collection err = milvusClient.LoadCollection(ctx, collectionName, 0) if err != nil { log.Fatalf("Failed to load collection: %v", err) } // Define reranker parameters rerankParams := param.RerankParams{ Reranker: "model", Provider: "siliconflow", ModelName: "BAAI/bge-reranker-v2-m3", Queries: []string{"example search query"}, MaxClientBatchSize: 64, MaxChunksPerDoc: 5, OverlapTokens: 50, Credential: "your-siliconflow-api-key", } // Define search parameters searchParams := param.SearchBuildParams{ CollectionName: collectionName, SearchParams: param.SearchParam{ MetricType: entity.MetricType(entity.L2), Params: map[string]interface{}{ "offset": 0, "ignore_growing_segments": true }, }, Vectors: []param.VectorParam{ { FieldName: "embeddings", Vectors: []entity.Vector{entity.Vector(make([]float32, 768))}, // Example vector TopK: 10, }, }, OutputFields: []string{"content"}, RerankParams: &rerankParams, } // Perform search // searchResult, err := milvusClient.Search(ctx, searchParams) // if err != nil { // log.Fatalf("Search failed: %v", err) // } // Process search results // fmt.Printf("Search results: %+v\n", searchResult) // Release collection err = milvusClient.ReleaseCollection(ctx, collectionName, 0) if err != nil { log.Fatalf("Failed to release collection: %v", err) } // Close connection defer milvusClient.Close(ctx) } ```