### Install teradatagenai Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/Embedding_Based.ipynb Install or upgrade the teradatagenai library. This is the first step before using any of its functionalities. ```python # %pip install teradatagenai --upgrade ``` -------------------------------- ### Install Libraries in User Environment Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/TextAnalytics/teradata_text_analytics_with_byollm_huggingface.ipynb Installs the 'transformers', 'torch', and 'pandas' libraries into the specified user environment. Ensure the environment is active before running. ```python demo=get_env('notebook_demo') demo.install_lib(['transformers', 'torch', 'pandas']) ``` -------------------------------- ### Install Teradata Generative AI Package on Windows Source: https://github.com/teradata/teradatagenai/blob/main/README.md Standard installation command for the teradatagenai package on Windows systems. ```bash python -m pip install teradatagenai ``` -------------------------------- ### Initialize File-Based Vector Store Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/FileBased-VectorStore-GettingStarted-VCL.ipynb This snippet demonstrates how to initialize a file-based vector store. Ensure you have the necessary libraries installed. ```python from teradataml.analytics.vector_stores.file_based_vector_store import FileBasedVectorStore vector_store = FileBasedVectorStore(index_name="my_index", embedding_model="all-MiniLM-L6-v2", index_path="./my_index_data") ``` -------------------------------- ### Load Example Data Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/Content_Based.ipynb Load the 'amazon_reviews_25' and 'amazon_reviews_10' datasets into teradataml DataFrames. This data will be used for collection creation and demonstration. ```python # Load example datasets load_data('byom', 'amazon_reviews_25') load_data('byom', 'amazon_reviews_10') # Create DataFrame for the main dataset input_data = DataFrame(in_schema('vs_user', 'amazon_reviews_25')) print("✓ Example data loaded") input_data.head(1) ``` -------------------------------- ### Install Teradata Generative AI with NV-Ingest Support Source: https://github.com/teradata/teradatagenai/blob/main/README.md Use this command to install the teradatagenai package with support for NVIDIA NV-Ingest integration. Requires Python 3.11 or later. ```bash pip install teradatagenai[nv-ingest-client] ``` -------------------------------- ### Load example data with embeddings Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/Embedding_Based.ipynb Load example datasets that contain pre-computed embeddings. These datasets are used to create or interact with embedding-based collections. Ensure the dataset names match those in your environment. ```python # Load example datasets with embeddings (1024 dimensions) # Note: Replace with actual dataset names that have embeddings in your environment load_data('amazon', 'amazon_reviews_embedded') load_data('amazon', 'amazon_reviews_embedded_10_alter') # Create DataFrame for the main dataset input_data = DataFrame(in_schema('vs_user', 'amazon_reviews_embedded')) print("✓ Example data with embeddings loaded") ``` -------------------------------- ### Create User Environment Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/TextAnalytics/teradata_text_analytics_with_byollm_huggingface.ipynb Creates a new user environment for the notebook. This is a prerequisite for installing libraries and deploying models. ```python create_env('notebook_demo') ``` -------------------------------- ### Bash Deployment Script Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/teradata_multiformat_document_rag.ipynb An example of a bash script for automated deployment of the CloudSync Pro application. It includes logging and error handling. ```bash #!/bin/bash ################################################################################ # CloudSync Pro Deployment Script # Version: 3.2.1 # Description: Automated deployment script for CloudSync Pro application # Author: DevOps Team - TechVision Solutions ################################################################################ set -e # Exit on error set -u # Exit on undefined variable # Configuration APP_NAME="cloudsync-pro" APP_VERSION="3.2.1" DEPLOYMENT_ENV=${1:-production} TIMESTAMP=$(date +%Y%m%d_%H%M%S) LOG_FILE="/var/log/deployments/${APP_NAME}_${TIMESTAMP}.log" # Color codes for output RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' NC='\033[0m' # No Color ################################################################################ # Functions ################################################################################ log_info() { echo -e "${GREEN}[INFO]${NC} $(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE" } log_warn() { echo -e "${YELLOW}[WARN]${NC} $(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE" } log_error() { echo -e "${RED}[ERROR]${NC} $(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE" } # Function to perform health checks health_check() { log_info "Performing health check for ${APP_NAME} version ${APP_VERSION} on environment ${DEPLOYMENT_ENV}..." # Simulate health check commands sleep 1 if [ "$RANDOM" -lt 16384 ]; then log_warn "Health check partially failed: Service XYZ is not responding." else log_info "Health check passed." fi } # Function to deploy the application deploy_application() { log_info "Deploying ${APP_NAME} version ${APP_VERSION} to ${DEPLOYMENT_ENV}..." # Simulate deployment steps sleep 2 log_info "Application deployed successfully." } # Function to restart the application restart_application() { log_info "Restarting ${APP_NAME}..." # Simulate restart command sleep 1 log_info "Application restarted." } ################################################################################ # Main Execution ################################################################################ log_info "Starting deployment process for ${APP_NAME} v${APP_VERSION}..." # Perform initial health check health_check # Deploy the application deploy_application # Perform post-deployment health check health_check # Restart the application if needed (e.g., after configuration changes) # restart_application log_info "Deployment process completed." exit 0 ``` -------------------------------- ### Azure OpenAI Integration - Setup Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/TextAnalytics/teradata_text_analytics_with_azure_openai.ipynb Sets up the connection to Azure OpenAI. This requires your Azure OpenAI endpoint and API key. Ensure these are kept secure. ```python from teradataml.azure_openai import AzureOpenAI # Replace with your actual Azure OpenAI endpoint and API key openai_endpoint = "YOUR_AZURE_OPENAI_ENDPOINT" openai_api_key = "YOUR_AZURE_OPENAI_API_KEY" aio = AzureOpenAI(endpoint=openai_endpoint, api_key=openai_api_key) ``` -------------------------------- ### Output of Initializing Collection Pipeline Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/Content_Embedding_Based_Ingestor_Pipeline.ipynb This output indicates the start of the collection pipeline initialization process. ```text 🏗️ Creating Content-Based Collection using Ingestor API... 🚀 Initializing collection pipeline... ``` -------------------------------- ### Load Example Data Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/ContentBased-VectorStore-GettingStarted-VCL.ipynb Loads the 'amazon_reviews_10' dataset into the database for use in creating a vector store. This operation may skip loading if the table already exists. ```python # Loading example data load_data('byom', 'amazon_reviews_10') amazon_reviews_updated = DataFrame('amazon_reviews_10') ``` -------------------------------- ### Load Data for Vector Store Operations Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/ContentBased-VectorStore-GettingStarted-AI-Factory.ipynb Loads example data into the database. This is a prerequisite for adding datasets to an existing vector store. ```python # Loading example data load_data('byom', 'amazon_reviews_10') amazon_reviews_10 = DataFrame('amazon_reviews_10') ``` -------------------------------- ### Display Environment and Installed Packages Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/TextAnalytics/teradata_text_analytics_with_byollm_huggingface.ipynb This snippet displays the details of the Teradata Python environment, including installed files, libraries, and models. It's useful for verifying the environment setup and available resources. ```text Result: ================================================ Environment Name: td_gen_ai_env Base Environment: python_3.11 Description: This env 'td_gen_ai_env' is created with base env 'python_3.11'. ############ Files installed in User Environment ############ File Size Timestamp 0 summarize_text.py 1515 2025-09-02T08:32:17Z 1 entity_recognition.py 2122 2025-09-01T09:39:45Z 2 td_sample_inference_script_03.py 4714 2025-09-02T08:39:17Z 3 mask_pii.py 1877 2025-09-01T09:37:13Z 4 td_sample_inference_script.py 3811 2025-09-02T06:40:32Z 5 sentence_similarity.py 2795 2025-09-01T09:48:28Z 6 recognize_pii.py 1616 2025-09-01T09:40:39Z 7 analyze_sentiment.py 1579 2025-09-01T09:19:24Z 8 classify_text.py 1717 2025-09-01T09:23:31Z 9 embeddings.py 2347 2025-09-01T09:32:16Z 10 td_sample_embeddings_script.py 3393 2025-09-02T06:40:34Z 11 detect_language.py 2209 2025-09-01T09:26:17Z 12 td_sample_embeddings_script_03.py 3927 2025-09-02T08:39:19Z 13 extract_key_phrases.py 2156 2025-09-01T09:34:27Z ############ Libraries installed in User Environment ############ name version 0 certifi 2025.8.3 1 charset-normalizer 3.4.3 2 filelock 3.19.1 3 fsspec 2025.7.0 4 hf-xet 1.1.9 5 huggingface-hub 0.34.4 6 idna 3.10 7 Jinja2 3.1.6 8 joblib 1.5.2 9 MarkupSafe 3.0.2 10 mpmath 1.3.0 11 networkx 3.5 12 numpy 2.3.2 13 nvidia-cublas-cu12 12.8.4.1 14 nvidia-cuda-cupti-cu12 12.8.90 15 nvidia-cuda-nvrtc-cu12 12.8.93 16 nvidia-cuda-runtime-cu12 12.8.90 17 nvidia-cudnn-cu12 9.10.2.21 18 nvidia-cufft-cu12 11.3.3.83 19 nvidia-cufile-cu12 1.13.1.3 20 nvidia-curand-cu12 10.3.9.90 21 nvidia-cusolver-cu12 11.7.3.90 22 nvidia-cusparse-cu12 12.5.8.93 23 nvidia-cusparselt-cu12 0.7.1 24 nvidia-nccl-cu12 2.27.3 25 nvidia-nvjitlink-cu12 12.8.93 26 nvidia-nvtx-cu12 12.8.90 27 packaging 25.0 28 pillow 11.3.0 29 pip 25.2 30 proxy-client 1.0.5 31 PyYAML 6.0.2 32 regex 2025.9.1 33 requests 2.32.5 34 safetensors 0.6.2 35 scikit-learn 1.7.1 36 scipy 1.16.1 37 sentence-transformers 5.1.0 38 sentencepiece 0.2.1 39 setuptools 80.9.0 40 sympy 1.14.0 41 threadpoolctl 3.6.0 42 tokenizers 0.22.0 43 torch 2.8.0 44 tqdm 4.67.1 45 transformers 4.56.0 46 triton 3.4.0 47 typing_extensions 4.15.0 48 urllib3 2.5.0 ############ Models installed in User Environment ############ Model Size \ 0 models--papluca--xlm-roberta-base-language-det... 6144 1 models--sentence-transformers--all-MiniLM-L6-v2 6144 2 models--Helsinki-NLP--opus-mt-en-fr 6144 3 models--facebook--bart-large-mnli 6144 4 models--ml6team--keyphrase-extraction-kbir-kpc... 6144 5 distilbert-base-uncased-emotion 6144 6 models--bhadresh-savani--distilbert-base-uncas... 6144 7 models--lakshyakh93--deberta_finetuned_pii 6144 8 models--tner--roberta-large-ontonotes5 6144 9 models--facebook--bart-large-cnn 6144 Timestamp 0 2025-09-01T09:24:41Z 1 2025-09-01T09:11:23Z 2 2025-09-01T09:53:41Z 3 2025-09-01T09:22:16Z 4 2025-09-01T09:33:23Z 5 2025-09-02T06:32:50Z 6 2025-09-01T09:17:49Z 7 2025-09-01T09:35:54Z 8 2025-09-01T09:38:11Z 9 2025-09-01T09:51:27Z ================================================ ``` -------------------------------- ### Prepare Sample Document Path Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/FileBased-VectorStore-GettingStarted-VCL.ipynb Sets up the path to a sample PDF document for use in vector store creation. ```python # Preparing the data import teradatagenai base_dir = os.path.dirname(teradatagenai.__file__) sample_doc = os.path.join(base_dir, 'example-data', 'SQL_Fundamentals.pdf') ``` -------------------------------- ### Manual Configuration Setup Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Authentication_Guide.ipynb Prompts the user for necessary configuration values like base URLs and IDP type. Use this when environment variables are not available or preferred. ```python def manual_configure(): """Prompt user for configuration values""" config = {} print("🔧 Manual Configuration Setup") print("=" * 60) print("\nPlease provide the following configuration details:\n") # VectorStore Base URL config['VS_BASE_URL'] = input("Enter VectorStore Base URL (e.g., https://your-site.teradata.com): ").strip().rstrip('/') if config['VS_BASE_URL'].endswith('/data-insights'): config['VS_BASE_URL'] = config['VS_BASE_URL'][:-len('/data-insights')] # Keycloak/IDP Base URL config['KEYCLOAK_BASE_URL'] = input("Enter Keycloak/IDP Base URL (e.g., https://keycloak.your-site.com): ").strip().rstrip('/') # ModelOps Base URL config['MODELOPS_BASE_URL'] = input("Enter ModelOps Base URL (e.g., https://modelops.your-site.com): ").strip().rstrip('/') # IDP Type print("\nSelect IDP Type:") print(" 1. Keycloak (default)") print(" 2. PingFederate") idp_choice = input("Enter choice (1 or 2) [default: 1]: ").strip() or "1" config['IDP_TYPE'] = 'keycloak' if idp_choice == '1' else 'ping' # Site ID (for PingFederate) if config['IDP_TYPE'] == 'ping': config['SITE_ID'] = input("Enter Site ID (for PingFederate): ").strip().upper() config['IDP_CLIENT_ID'] = f"onetd-dev-{config['SITE_ID']}" config['IDP_BASE_URL'] = input("Enter PingFederate Base URL: ").strip().rstrip('/') else: config['IDP_CLIENT_ID'] = 'vectorstore' config['IDP_BASE_URL'] = config['KEYCLOAK_BASE_URL'] config['SITE_ID'] = '' # SSL Verification print("\nEnable SSL Certificate Verification?") print(" 1. Yes (recommended for production)") print(" 2. No (only for development/testing)") ssl_choice = input("Enter choice (1 or 2) [default: 1]: ").strip() or "1" config['SSL_VERIFY'] = ssl_choice == '1' config['REALM'] = "teradata" print("\n" + "=" * 60) print("✅ Configuration completed!\n") return config ``` -------------------------------- ### Environment Setup and TeradataGenAI Imports Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/File_Content_Based_Ingestor_pdf_without_create_context.ipynb Sets up the environment by configuring auto-reloading, adding Python paths, and importing necessary TeradataML and TeradataGenAI modules for file-based vector store operations. Ensures all imports are successful and modules are ready for use. ```python # 1. Environment Setup and Auto-Reload Configuration print("🔧 Setting up file-based vector store environment...") import os import tempfile from pathlib import Path print("✅ Auto-reload configured - modules will refresh on code changes") print("📂 Python paths added for development") # Now import the modules that we want to auto-reload print("📚 Importing TeradataGenAI modules...") # TeradataML imports from teradataml import set_auth_token # TeradataGenAI file-based imports import teradatagenai from teradatagenai import ( Collection, CollectionManager, LocalConfig, S3Config, AzureBlobConfig, GCPConfig, BasicIngestor, NVIngestor, UnstructuredIngestor, ExtractionSchema, ColumnInfo, HNSW, FLAT, SearchParams, Ingestor, CollectionType, TeradataAI ) from teradatasqlalchemy.types import VARCHAR, INTEGER print("✅ All imports successful!") print("🔄 Modules will now auto-reload when you modify source code") print("📚 Environment ready for file-based vector store creation") # Get the base directory of the teradatagenai package base_dir = os.path.dirname(teradatagenai.__file__) ``` -------------------------------- ### Connect to Vantage and Set Environment Variables Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/GetNimEndpoints.ipynb Establishes a connection to Vantage by setting necessary environment variables for host, username, password, base URL, and authentication mechanism, then calls create_context(). ```python # Connect to Vantage using create_context. os.environ['TD_HOST'] = getpass(prompt='hostname: ') os.environ['TD_USERNAME'] = getpass(prompt='username: ') os.environ['TD_PASSWORD'] = getpass(prompt='password: ') os.environ['TD_BASE_URL'] = getpass(prompt='base_url: ') os.environ['TD_AUTH_MECH'] = getpass(prompt='auth_mech: ') create_context() ``` -------------------------------- ### Install Teradata Generative AI Package on macOS/Linux Source: https://github.com/teradata/teradatagenai/blob/main/README.md Standard installation command for the teradatagenai package on macOS and Linux systems. ```bash pip install teradatagenai ``` -------------------------------- ### Environment Setup and Module Imports Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/Content_Embedding_Based_Ingestor_Pipeline.ipynb Sets up the environment by configuring auto-reloading and importing necessary modules from TeradataML and TeradataGenAI. This includes classes for collections, ingestors, and indexing. ```python # 1. Environment Setup and Auto-Reload Configuration print("🔧 Setting up environment with auto-reload...") import os import pandas as pd print("✅ Auto-reload configured - modules will refresh on code changes") print("📂 Python paths added for development") # Now import the modules that we want to auto-reload print("📚 Importing TeradataML and TeradataGenAI modules...") # TeradataML imports from teradataml import create_context, DataFrame, in_schema # TeradataGenAI file-based imports from teradatagenai import ( Collection, CollectionManager, LocalConfig, S3Config, AzureBlobConfig, GCPConfig, BasicIngestor, NVIngestor, UnstructuredIngestor, ExtractionSchema, ColumnInfo, HNSW, FLAT, SearchParams, Ingestor, CollectionType, TeradataAI, load_data, ContentBasedIndex, EmbeddingBasedIndex ) from teradatasqlalchemy.types import VARCHAR print("✅ All imports successful!") print("🔄 Modules will now auto-reload when you modify source code") print("📚 Environment ready for vector store validation") ``` -------------------------------- ### Vector Data Example Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/FileBased-VectorStore-GettingStarted-VCL.ipynb This is an example of vector data, likely representing embeddings. Each line contains a series of floating-point numbers. ```Text -0.0888671875,-0.462890625,0.447265625,-0.10693359375,0.5234375,-0.1591796875,0.36328125,2.52723693847656e-05,0.2578125,-0.0322265625,0.70703125,-0.66796875,0.01416015625,0.1943359375,-0.10400390625,-0.043701171875,-0.3984375,0.4609375,-0.23828125,0.2373046875,-0.14453125,-0.125,-0.00482177734375,0.59765625,-0.1806640625,0.56640625,0.30859375,-0.421875,0.115234375,-0.53125,-0.09033203125,0.2333984375,0.349609375,0.3125,-0.376953125,0.58203125,-0.2890625,0.11474609375,0.388671875,0.23046875,-0.076171875,-0.1572265625,-0.008544921875,-0.625,-0.03125,-0.3203125,0.216796875,0.16015625,-0.061767578125,-0.1416015625,0.126953125,0.1650390625,-0.126953125,0.02490234375,0.1005859375,-0.1708984375,-0.19140625,0.038818359375,1.015625,-0.07470703125,-0.333984375,0.328125,0.046142578125,-0.068359375,-0.1796875,-0.1982421875,-0.0135498046875,-0.01513671875,-0.1201171875,0.48046875,0.0673828125,-0.06494140625,0.1376953125,0.095703125,-0.62109375,0.083984375,0.1728515625,0.5703125,-0.21484375,0.1435546875,0.0830078125,-0.142 ``` -------------------------------- ### Create a Metadata-Based Vector Store Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/Metadata-Based-VectorStore-GettingStarted-VCL.ipynb Initializes a new metadata-based vector store. Ensure you have the necessary permissions and that the service is operational. ```python vs_meta = VectorStore(name=f'cricket_test_example_{uuid.uuid4().hex[:8]}') ``` -------------------------------- ### Configure BYOM Install Location Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/TextAnalytics/teradata_text_analytics_with_onnx.ipynb This snippet sets the installation location for Bring Your Own Model (BYOM). Ensure the path provided is correct for your environment. ```python # Setting up BYOM install location. configure.byom_install_location = getpass.getpass("BYOM Install Location: ") ``` -------------------------------- ### Setup CSV Document Management Demo Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/File_Content_Based_Ingestor_csv.ipynb Initializes the document management scenario by cleaning up any existing collection and setting up the extraction schema for CSV data. This prepares the environment for subsequent document operations. ```python # 8.1: Setup - Document Management Demo print("📊 CSV Document Management Scenario\n") doc_mgmt_collection = "csv_doc_mgmt_demo" # Clean up existing collection try: Collection(name=doc_mgmt_collection).destroy() except: pass # Setup extraction schema doc_schema = ExtractionSchema( data_columns=[ ColumnInfo(name="Country", datatype=VARCHAR(100)), ColumnInfo(name="Product", datatype=VARCHAR(200)), ColumnInfo(name="Amount", datatype=VARCHAR(500)) ] ) print("✅ Setup complete - Ready for document management operations\n") ``` -------------------------------- ### Prepare Sample Documents for Vector Store Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/FileBased-VectorStore-GettingStarted-AI-Factory.ipynb Prepares the paths to sample PDF documents for use in creating a vector store. Ensure the `teradatagenai` library is installed and accessible. ```python # Preparing the data import teradatagenai base_dir = os.path.dirname(teradatagenai.__file__) sample_doc = os.path.join(base_dir, 'example-data', 'SQL_Fundamentals.pdf') doc_to_update = os.path.join(base_dir, 'example-data', 'LLM_handbook.pdf') ``` -------------------------------- ### Another Vector Data Example Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/FileBased-VectorStore-GettingStarted-VCL.ipynb This is another example of vector data, similar to the previous one, showing a different set of numerical values. ```Text -0.0071091346130606,-0.0370299978746233,0.0357800401404588,-0.0085543982431883,0.0418735840945108,-0.0127339444168009,0.0290615173193246,2.0217211886816e-06,0.0206243026137143,-0.00257803782671428,0.0565605874709437,-0.0534356931355324,0.00113277419658658,0.015546349318671,-0.00832003116803246,-0.00349597553774134,-0.0318739222211948,0.0368737531578528,-0.0190618554460086,0.0189837330876234,-0.0115621090410216,-0.00999966187331601,-0.000385729144527326,0.0478108833317922,-0.014452636301277,0.0453109678634632,0.0246866652497489,-0.0337488588224415,0.0092184382894632,-0.042498562961593,-0.00722631815063852,0.0186712436540822,0.0279678043019307,0.02499915468329,-0.0301552303367186,0.0465609255976277,-0.0231242180820433,0.00917937711027055,0.031092698637342,0.0184368765789264,-0.00609354395405194,-0.0125776997000303,-0.000683570635871211,-0.04999830936658,-0.002499915468329,-0.0256241335503723,0.0173431635615325,0.0128120667751861,-0.00494123916786904,-0.0113277419658658,0.0101559065900866,0.0132026785671125 ``` -------------------------------- ### Initialize and Configure Audio RAG Pipeline Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/teradata_audio_image_rag.ipynb Imports necessary LangChain components, sets up AWS credentials, initializes Teradata Vector Store for audio, configures embeddings, and builds the RAG chain with a prompt template and LLM. ```python # Import necessary components from langchain_core.runnables import RunnablePassthrough from langchain_core.prompts import PromptTemplate from langchain_core.output_parsers import StrOutputParser from langchain.chat_models import init_chat_model # AWS Configuration (already set earlier) aws_access_key = os.environ['AWS_ACCESS_KEY_ID'] aws_secret_key = os.environ['AWS_SECRET_ACCESS_KEY'] aws_region = os.environ['AWS_REGION'] # Create TeradataVectorStore instance for audio vs_audio = TeradataVectorStore(td_vs_audio_name) retriever_audio = vs_audio.as_retriever(search_kwargs={"top_k": 5}) # Update vector store with embedding configuration embedding_endpoint = os.environ['EMBEDDING_ENDPOINT'] model_name = os.environ['EMBEDDING_MODEL_NAME'] vs_audio.update( embeddings_model=model_name, embeddings_dims=2048, embeddings_base_url=os.environ['EMBEDDINGS_BASE_URL'] ) print("🔑 Configuring Audio RAG Pipeline") print("=" * 60) # Initialize the Bedrock LLM for images with Model Name llm_audio = init_chat_model( model_name = model_name, model_provider="bedrock_converse", region_name=aws_region, aws_access_key_id=aws_access_key, aws_secret_access_key=aws_secret_key ) print("✅ LLM initialized: AWS Bedrock (Claude)") print(f"✅ Using retriever from vector store: {td_vs_audio_name}") # Create the Prompt Template for audio content prompt_audio = PromptTemplate.from_template( """You are an AI assistant analyzing audio transcriptions. Use the following context retrieved from audio transcriptions to answer the question comprehensively. Context: {context} Question: {question} Answer:""" ) print("✅ Prompt template configured") # Build RAG chain for audio rag_chain_audio = ( { "context": retriever_audio, "question": RunnablePassthrough() } | prompt_audio | llm_audio | StrOutputParser() ) print("✅ Audio RAG chain built successfully") print("=" * 60) print("🎉 Audio RAG pipeline ready!") print(" You can now ask questions about your audio content.\n") ``` -------------------------------- ### Get Matching Objects Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/Metadata-Based-VectorStore-GettingStarted-VCL.ipynb Retrieve a list of objects that match the defined pattern using the get() method. This allows you to find all data entries conforming to the specified pattern. ```python ptn.get() ``` ```python ptn2.get() ``` -------------------------------- ### Collection Creation and Ingestion Process Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/File-Embedding-Based-Ingestor-Parquet.ipynb This output details the steps involved in creating a collection, uploading files, ingesting data, and creating an index, showing progress and completion messages. ```text Output: Creating collection... Collection initialized successfully Starting create collection operation... Create Collection completed. Completed: ｜⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿｜ 100% - 100/100 Create Collection completed successfully Processing files... Files uploaded successfully and ingestion in progress Starting ingest operation... Ingest completed. Completed: ｜⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿｜ 100% - 100/100 Warning: Ingest completed with warnings Creating index with default settings... Updating collection with index configuration for file based VS... Collection update request is accepted and in progress Starting update operation... Update completed. Completed: ｜⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿⫿｜ 100% - 100/100 Update completed successfully Pipeline completed successfully! Operations: create_collection, ingest, create_index ✓ Ingestion completed - embeddings uploaded to collection ``` -------------------------------- ### Create a Vector Store with Access Management Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/VectorStoreManagement-GettingStarted.ipynb Initializes and creates a Vector Store, setting up access management configurations. The creation process differs based on the platform (e.g., 'nim' or other). ```python # Initialize and create a Vectore Store vs = VectorStore(name=f"acces_management_vs_{uuid.uuid4().hex[:8]}") # Create an input DataFrame as amazon_reviews_25 is in 'oaf' database. input_data = DataFrame(in_schema(os.environ['TD_USERNAME'], 'amazon_reviews_25')) # Python platform_value = VSManager.health().to_pandas()["platform"].iloc[0].lower() if platform_value == 'nim': vs.create(embeddings_model= 'nvidia/llama-3.2-nv-embedqa-1b-v2', embeddings_base_url= getpass(prompt='embeddings base url: '), completions_base_url=getpass(prompt='completions base url: '), chat_completion_model='mistralai/mistral-7b-instruct-v0.3', chat_completion_max_tokens=8192, embeddings_dims=2048, search_algorithm= 'HNSW', seed=10, top_k=10, ef_construction=32, num_connpernode=32, maxnum_connpernode=32, metric='EUCLIDEAN', apply_heuristics=False, ef_search=32, object_names= input_data, key_columns= ['rev_id', 'aid'], data_columns= ['rev_text'], vector_column= 'VectorIndex') else: vs.create(embeddings_model= 'amazon.titan-embed-text-v1', search_algorithm= 'HNSW', seed=10, top_k=10, ef_construction=32, num_connpernode=32, maxnum_connpernode=32, metric='EUCLIDEAN', apply_heuristics=False, ef_search=32, object_names= input_data, key_columns= ['rev_id', 'aid'], data_columns= ['rev_text'], vector_column= 'VectorIndex') ``` -------------------------------- ### Vector Data Example (Normalized) Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/FileBased-VectorStore-GettingStarted-AI-Factory.ipynb Presents a normalized version of vector data, often used for consistency in machine learning models. This data is similar to the previous example but may have undergone scaling or preprocessing. ```text 0.00839698407799006,-0.0316046476364136,0.00906774308532476,0.040695384144783,0.0266934130340815,0.0158333126455545,0.00262705655768514,0.00583390472456813,-0.0211564023047686,0.0363029614090919,-0.00708945374935865,0.00187332718633115,-0.032001506537199,-0.0299422461539507,-0.000617778103332967,-0.0107081532478333,-0.0153144998475909,-0.0174647271633148,0.0178465899080038,-0.0103112971410155,0.0532038919627666,0.0301561690866947,-0.0166570171713829,0.0374625436961651,-0.0324283540248871,0.00649066874757409,-0.065528467297554,0.0206685774028301,-0.0062997373752296,0.00318585592322052,0.00179335591383278,-0.0535697601735592,0.00954057369381189,-0.0424957387149334,0.0197069216519594,0.000960654986556619,-0.0137280691415071,-0.0102123320102692,-0.0274101551622152,0.0197678990662098,-0.00273201894015074,0.0381942838430405,0.000903675449080765,0.0348384864628315,-0.0428616069257259,0.0123705565929413,0.0478348173201084,-0.00467532081529498,-0.0216902103275061,-0.0152995055541396,0.000824703776743263,-0.01877725496 ``` -------------------------------- ### Environment Setup and Auto-Reload Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/File_Content_Based_Ingestor_pdf.ipynb Sets up the environment by configuring auto-reloading for modules and importing necessary TeradataGenAI and TeradataML libraries. This ensures that changes in source code are automatically reflected and prepares the environment for creating file-based vector stores. ```python # 1. Environment Setup and Auto-Reload Configuration print("🔧 Setting up file-based vector store environment...") import sys import os import tempfile from pathlib import Path print("✅ Auto-reload configured - modules will refresh on code changes") print("📂 Python paths added for development") # Now import the modules that we want to auto-reload print("📚 Importing TeradataGenAI modules...") # TeradataML imports import teradatagenai from teradataml import create_context # TeradataGenAI file-based imports from teradatagenai import \ Collection, CollectionManager, LocalConfig, S3Config, AzureBlobConfig, GCPConfig, BasicIngestor, NVIngestor, UnstructuredIngestor, ExtractionSchema, ColumnInfo, HNSW, FLAT, SearchParams, Ingestor, CollectionType, TeradataAI from teradatasqlalchemy.types import VARCHAR, INTEGER print("✅ All imports successful!") print("🔄 Modules will now auto-reload when you modify source code") print("📚 Environment ready for file-based vector store creation") # Get the base directory of the teradatagenai package base_dir = os.path.dirname(teradatagenai.__file__) ``` -------------------------------- ### Setup and Imports for CSV File-Based Vector Store Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/Collection/File_Embedding_Based_Ingestor_csv.ipynb Imports necessary libraries for Teradata GenAI, Teradataml, and standard Python modules for environment setup, file handling, and data manipulation. Configures auto-reloading for modules during development. ```python # 1. Environment Setup and Auto-Reload Configuration print("🔧 Setting up CSV file-based vector store environment (with embeddings)...") import sys import os import tempfile from pathlib import Path import pandas as pd from datetime import datetime print("✅ Auto-reload configured - modules will refresh on code changes") print("📂 Python paths added for development") # Now import the modules that we want to auto-reload print("📚 Importing TeradataGenAI modules...") # Teradataml imports from teradataml import create_context, remove_context # TeradataGenAI file-based imports import teradatagenai from teradatagenai import ( Collection, CollectionManager, LocalConfig, S3Config, AzureBlobConfig, GCPConfig, BasicIngestor, NVIngestor, UnstructuredIngestor, ExtractionSchema, ColumnInfo, HNSW, FLAT, SearchParams, Ingestor, CollectionType, TeradataAI ) # Get the directory one level above the current notebook file (notebooks directory) base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) if '__file__' in dir() else os.path.dirname(os.getcwd()) from teradatasqlalchemy.types import VARCHAR, INTEGER, CLOB, VECTOR print("✅ All imports successful!") print(f"📂 Base directory: {base_dir}") print("📚 Environment ready for CSV file-based vector store creation with embeddings") print("🔄 Modules will now auto-reload when you modify source code") ``` -------------------------------- ### Similarity Search with Input Vector (Second Example) Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/EmbedddingBased-VectorStore-GettingStarted-VCL.ipynb Presents another instance of similarity search using a provided vector, illustrating variations in input data or query parameters. This example is useful for understanding the behavior with different semantic contexts. ```python 2 0.178799 oaf amazon_reviews_embedded A2KU9IU07LOJS1 000100039X This book has been a classic for many years. It has so much wisdom in it that it can be read numerous times and new things will come out each time. My favorite chapter is the one on children. -0.0621318183839321,0.0329974703490734,-0.00635804887861013,-0.0347680635750294,0.00643853051587939,-0.029134351760149,0.0181083679199219,-0.0149695836007595,-0.029134351760149,0.0121527267619967,0.0342851765453815,-0.010462611913681,0.015532954595983,-0.050220537930727,-0.0479670539498329,-0.0386311821639538,0.0278466455638409,-0.0362167358398438,-0.0106235751882195,0.00309854280203581,0.029456278309226,-0.0226958207786083,0.038148295134306,-0.00424540601670742,-0.0434600822627544,0.00756527343764901,0.00989924091845751,-0.0212471503764391,0.0521520972251892,-0.0470012724399567,-0.0247883424162865,0.00619708560407162,0.0453916415572166,-0.00989924091845751,0.00655925320461392,0.0515082441270351,0.0630975961685181,-0.00239432859234512,-0.0169816240668297,-0.0136818774044514,0.00348083069548011,0.0102614080533385,-0.00236414792016149,0.0181888490915298,0.0453916415572166,0.031548798084259,0.0231787096709013,-0.00259553268551826,-0.00800792220979929,0.0339632481336594,0.0114283915609121,0.00110662239603698,-0.0 5 ``` -------------------------------- ### GET /api/v1/sync/jobs Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/sample_data/technical_documentation.html Returns a list of all synchronization jobs associated with the authenticated user. ```APIDOC ## GET /api/v1/sync/jobs ### Description Returns a list of all synchronization jobs for the authenticated user. ### Method GET ### Endpoint /api/v1/sync/jobs ### Response #### Success Response (200) - **jobs** (array) - A list of synchronization job objects ``` -------------------------------- ### Get Document by ID Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/FileBased-VectorStore-GettingStarted-VCL.ipynb Retrieve a specific document from the vector store using its unique ID. ```python document = vector_store.get("doc_id_123") print(document) ``` -------------------------------- ### List Available Base Environments Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/TextAnalytics/teradata_text_analytics_with_byollm_huggingface.ipynb Prints a list of available base environments (e.g., Python, R versions) to confirm successful CCP authentication. ```python # Listing the available base environments to confirm that the CCP authentication was successful print(list_base_envs()) ``` -------------------------------- ### Loading Data into a Vector Store Source: https://github.com/teradata/teradatagenai/blob/main/Workflows/VectorStore/EmbedddingBased-VectorStore-GettingStarted-VCL.ipynb Demonstrates how to load data from a specified source path into a vector store. Ensure the source path is correctly configured. ```python from langchain_community.document_loaders import TextLoader from langchain_community.vectorstores import TeradataVectorStore from langchain_community.embeddings import OpenAIEmbeddings loader = TextLoader(file_path="/teradata/teradatagenai/Workflows/VectorStore/EmbedddingBased-VectorStore-GettingStarted-VCL.ipynb") documents = loader.load() embeddings = OpenAIEmbeddings() vector_store = TeradataVectorStore.from_documents(documents=documents, embedding=embeddings) ```