### Chunk Match Web UI - Basic Setup and Run

Source: https://github.com/jparkerweb/chunk-match/blob/main/webui/README.md

This snippet demonstrates the essential commands to clone the repository, navigate to the web UI directory, install dependencies, and start the development server. It's the primary setup guide for running the application locally.

```javascript
git clone https://github.com/jparkerweb/chunk-match.git
cd chunk-match/webui
npm install
npm start
```

--------------------------------

### Start Chunk Match Web UI Server

Source: https://github.com/jparkerweb/chunk-match/blob/main/webui/README.md

Starts the development server for the Chunk Match Web UI. After installation, this command launches the application, typically accessible at http://localhost:3000.

```bash
npm start
```

--------------------------------

### Install Chunk Match Web UI Dependencies

Source: https://github.com/jparkerweb/chunk-match/blob/main/webui/README.md

Installs the necessary Node.js packages for the Chunk Match Web UI. Ensure Node.js and npm are installed and updated.

```bash
git clone https://github.com/jparkerweb/chunk-match.git
cd chunk-match/webui
npm install
```

--------------------------------

### Install and Import Libraries

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

Installs necessary Python packages including openai, langchain, numpy, and pandas, and then imports them for use in the project. These libraries are essential for working with LLMs, data manipulation, and numerical operations.

```bash
pip install openai langchain numpy pandas
```

--------------------------------

### Install chunk-match with npm

Source: https://github.com/jparkerweb/chunk-match/blob/main/README.md

Installs the chunk-match library using npm. This is the first step to using the library in a NodeJS project.

```bash
npm install chunk-match
```

--------------------------------

### Generate Code Example with Custom Settings

Source: https://github.com/jparkerweb/chunk-match/blob/main/webui/public/index.html

This section displays a code example that incorporates the user's customized chunk match settings. It allows users to copy the generated code for use in their own projects, demonstrating how to apply the tuned parameters programmatically.

```JavaScript
Copy Code Close
```

--------------------------------

### Implement sinpi in OpenCL

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

This code example illustrates the implementation of the 'sinpi' function within the OpenCL framework, designed for parallel computing on heterogeneous systems. It calculates the sine of pi times the input.

```OpenCL
sinpi(x)
```

--------------------------------

### Implement sinpi in CUDA

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

This example shows how to use the 'sinpi' function in CUDA, NVIDIA's parallel computing platform and programming model. It calculates the sine of pi times the input for GPU acceleration.

```CUDA
sinpi(x)
```

--------------------------------

### Fixed Point Iteration for Dottie Number (Python)

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Demonstrates the fixed point iteration method to find the Dottie number, which is the unique real root of the equation cos(x) = x. The iteration starts with an initial value and converges to the Dottie number.

```python
x_n+1 = cos(x_n)
Initial value x_0 = -1
This iteration converges to the Dottie number.
```

--------------------------------

### Get Word Embeddings using OpenAI

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

Defines a Python function to generate embeddings for a given text using OpenAI's 'text-embedding-ada-002' model. It preprocesses the text by replacing newline characters and then calls the OpenAI API to get the embedding vector.

```python
import openai

def get_embedding(text, model="text-embedding-ada-002"):
   text = text.replace("\n", " ")
   return openai.Embedding.create(input = [text], model=model)['data'][0]['embedding']
```

--------------------------------

### No Code Found

Source: https://github.com/jparkerweb/chunk-match/blob/main/webui/public/sample.txt

The provided text discusses Large Language Models (LLMs) but does not contain any executable code snippets.

--------------------------------

### Taylor Series Expansion for Sine and Cosine

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Demonstrates the Taylor series expansions for sine and cosine functions around a point, illustrating how these series approximate the functions. The expansion for sine is shown, and the derivative of this series yields the expansion for cosine.

```mathematics
sin(x) = x - x^3/3! + x^5/5! - x^7/7! + ... = sum_{n=0}^{inf} (-1)^n * x^(2n+1) / (2n+1)!
cos(x) = 1 - x^2/2! + x^4/4! - x^6/6! + ... = sum_{n=0}^{inf} (-1)^n * x^(2n) / (2n)!
```

--------------------------------

### Inference with Sparse and Non-Sparse Subsets

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

This pseudocode illustrates the inference process using pre-defined sparse (S) and non-sparse (R) chunk subsets. It shows how a hybrid retriever first attempts to retrieve information from R using a user query (Q). If successful, a large language model (LLM) generates an initial response. The description implies a fallback to searching S if this initial retrieval is insufficient, though that part is not explicitly coded here.

```pseudocode
chunks_R <- H.retrieve(R, Q) {Retrieve information from subset R}
initial_response <- LLM.generate(chunks_R) {Generate initial response}
```

--------------------------------

### Prompt Engineering and LLM Interaction (Python)

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

This code demonstrates how to construct a prompt for a Large Language Model (LLM) by incorporating relevant context retrieved via RAG. It uses Langchain's PromptTemplate to format the prompt and then sends it to an OpenAI LLM instance to generate an answer.

```Python
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate

llm = OpenAI(temperature=0)

context = ""
for index, row in df[0:5].iterrows():
    context = context + " " + row.text_chunks

template = """
You are a sports journalist! Given the following context sections, answer thequestion using only the given context. If you are unsure and the answer is notexplicitly written in the context, say "Sorry, I am a bad journalist!"
Context sections:{context}
Question:{users_question}
Answer:"""

prompt = PromptTemplate(template=template, input_variables=["context", "users_question"])

prompt_text = prompt.format(context = context, users_question = users_question)

res = llm(prompt_text)
print(res)
```

--------------------------------

### Import Libraries for Web Scraping and Text Chunking

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

This snippet imports necessary Python libraries for web scraping (requests, BeautifulSoup) and text processing for LLM prompts (langchain.text_splitter). It sets up the foundation for fetching data from a website and preparing it for LLM input.

```Python
import requests
from bs4 import BeautifulSoup
from langchain.text_splitter import RecursiveCharacterTextSplitter
```

--------------------------------

### Initialize OpenAI API Key

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

Sets the OpenAI API key by retrieving it from environment variables. This is a crucial step for authenticating requests to the OpenAI API, enabling the use of their models for tasks like generating embeddings.

```python
import os
import openai

openai.api_key = os.getenv("OPENAI_API_KEY")
```

--------------------------------

### Football Data Snippets

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

These text snippets provide a definition of football, mention its various forms, and touch upon its historical evolution and the codification of its rules. This information serves as a data source for LLM applications.

```English
sen_5 = "Football refers to a family of team sports involving kicking a ball to score a goal, with the term usually denoting the most popular form in a particular region."
sen_6 = "Common forms of football include soccer, Australian rules football, Gaelic football, American football, Canadian football, rugby league, and rugby union."
sen_7 = "Modern football codes evolved from traditional games played worldwide, with contemporary rules being codified in English public schools in the 19th century."
sen_8 = "The spread of football rules was facilitated by the British Empire, and by the late 19th century, distinct regional codes like Gaelic football had developed."
```

--------------------------------

### Recommend Outfits with Flask and Vector Similarity

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-8.txt

This Flask web application allows users to upload an image of an outfit. The system analyzes the image, generates descriptions for each clothing item, creates vector embeddings for these descriptions, and then calculates cosine similarity against a pre-existing wardrobe embedding database to find matching items.

```python
from flask import Flask, request, render_template, jsonify
import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

app = Flask(__name__)

# Load the embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Load the wardrobe embeddings (assuming saved in CSV)
# In a real app, you'd load this once on startup or use a vector database
try:
    wardrobe_df = pd.read_csv("wardrobe_embeddings.csv")
    # Convert embedding strings back to numpy arrays
    wardrobe_df['embedding'] = wardrobe_df['embedding'].apply(lambda x: np.array(eval(x)))
    wardrobe_embeddings = np.vstack(wardrobe_df['embedding'].values)
    wardrobe_descriptions = wardrobe_df['description'].values
    print("Wardrobe embeddings loaded successfully.")
except FileNotFoundError:
    print("Error: wardrobe_embeddings.csv not found. Please run the embedding generation script first.")
    wardrobe_embeddings = None
    wardrobe_descriptions = None
except Exception as e:
    print(f"Error loading wardrobe embeddings: {e}")
    wardrobe_embeddings = None
    wardrobe_descriptions = None

# Placeholder for image analysis and description generation
def analyze_outfit_image(image_file):
    # In a real scenario, this would involve image analysis to describe each item.
    # For demonstration, we'll return hardcoded descriptions.
    print(f"Analyzing image: {image_file.filename}")
    # Example: return ["A red t-shirt", "Blue jeans"]
    return ["Description of item 1", "Description of item 2"]

@app.route('/')
def index():
    # Render the HTML form for uploading images
    return render_template('upload.html') # Assumes you have an upload.html file

@app.route('/recommend', methods=['POST'])
def recommend():
    if 'file' not in request.files:
        return jsonify({'error': 'No file part in the request'})
    
    file = request.files['file']
    
    if file.filename == '':
        return jsonify({'error': 'No selected file'})

    if not wardrobe_embeddings or not wardrobe_descriptions:
        return jsonify({'error': 'Wardrobe data not available'})

    try:
        # Analyze the uploaded image to get descriptions of clothing items
        outfit_descriptions = analyze_outfit_image(file)
        
        if not outfit_descriptions:
            return jsonify({'error': 'Could not analyze outfit image'})

        # Generate embeddings for the outfit descriptions
        outfit_embeddings = model.encode(outfit_descriptions)

        # Calculate cosine similarity between outfit items and wardrobe items
        # Reshape outfit_embeddings if it's a single item for similarity calculation
        if outfit_embeddings.ndim == 1:
            outfit_embeddings = outfit_embeddings.reshape(1, -1)

        similarity_scores = cosine_similarity(outfit_embeddings, wardrobe_embeddings)

        # Find the best match for each item in the outfit
        recommendations = []
        for i, outfit_desc in enumerate(outfit_descriptions):
            best_match_index = np.argmax(similarity_scores[i])
            best_match_description = wardrobe_descriptions[best_match_index]
            best_match_score = similarity_scores[i][best_match_index]
            
            recommendations.append({
                "outfit_item": outfit_desc,
                "recommended_wardrobe_item": best_match_description,
                "similarity_score": float(best_match_score) # Ensure score is JSON serializable
            })

        return jsonify({'recommendations': recommendations})

    except Exception as e:
        print(f"An error occurred during recommendation: {e}")
        return jsonify({'error': f'An internal error occurred: {str(e)}'}), 500

if __name__ == '__main__':
    # Ensure you have an 'upload.html' file in a 'templates' folder
    # Example upload.html:
    # <!DOCTYPE html>
    # <html>
    # <head><title>Upload Outfit</title></head>
    # <body>
    #     <h1>Upload an Outfit Image</h1>
    #     <form action="/recommend" method="post" enctype="multipart/form-data">
    #         <input type="file" name="file" accept="image/*">
    #         <input type="submit" value="Get Recommendations">
    #     </form>
    # </body>
    # </html>
    app.run(debug=True)

```

--------------------------------

### Sine and Cosine as Solutions to Differential Equations

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Illustrates how sine and cosine functions can be defined as solutions to a system of two-dimensional differential equations. The initial conditions for these solutions are provided, relating to the unit circle's phase space trajectory.

```mathematical-notation
d^2x/dt^2 = -x
with initial conditions x(0) = cos(t0) and dx/dt(0) = -sin(t0)
This system's solution is x(t) = cos(t + t0).
```

```mathematical-notation
d^2y/dt^2 = -y
with initial conditions y(0) = sin(t0) and dy/dt(0) = cos(t0)
This system's solution is y(t) = sin(t + t0).
```

--------------------------------

### Web Scraping and Text Extraction (Python)

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

This snippet demonstrates how to scrape content from a given URL using the 'requests' library and parse the HTML to extract all text content using 'BeautifulSoup'. It includes setting up request headers to mimic a browser.

```Python
import requests
from bs4 import BeautifulSoup

url = 'https://olympics.com/en/news/fifa-world-cup-winners-list-champions-record'
headers = { 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0' }

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
text = soup.get_text()
```

--------------------------------

### Usage: Match text chunks with a query

Source: https://github.com/jparkerweb/chunk-match/blob/main/README.md

Demonstrates how to use the `matchChunks` function from the 'chunk-match' library. It shows how to prepare documents, define a query, set matching options, and process the results.

```javascript
import { matchChunks } from 'chunk-match';

const documents = [
    {
        document_name: "doc1.txt",
        document_text: "Your document text here..."
    },
    {
        document_name: "doc2.txt",
        document_text: "Another document text..."
    }
];

const query = "What are the key points?";

const options = {
    maxResults: 5,
    minSimilarity: 0.5,
    chunkingOptions: {
        maxTokenSize: 500,
        similarityThreshold: 0.5,
        dynamicThresholdLowerBound: 0.4,
        dynamicThresholdUpperBound: 0.8,
        numSimilaritySentencesLookahead: 3,
        combineChunks: true,
        combineChunksSimilarityThreshold: 0.8,
        onnxEmbeddingModel: "nomic-ai/nomic-embed-text-v1.5",
        dtype: 'q8',
        chunkPrefixDocument: "search_document",
        chunkPrefixQuery: "search_query"
    }
};

const results = await matchChunks(documents, query, options);
console.log(results);
```

--------------------------------

### Implement sinpi in ARM

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

This snippet demonstrates the 'sinpi' function as implemented for ARM architecture, likely within a specific library or compiler. It computes the sine of pi times the input for ARM-based systems.

```ARM
sinpi(x)
```

--------------------------------

### Retrieve Information Chunks

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

Retrieves relevant information chunks from a subset 'S' of a corpus 'H' based on a query 'Q'. This is a key step in preparing context for the LLM.

```Python
chunks_S = H.retrieve(S, Q)
```

--------------------------------

### Generate Text Embeddings

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

This snippet shows how to use the FlagModel from Hugging Face to encode input text into numerical embeddings. It iterates through a list of texts, generates an embedding for each, and stores them in a list, which is then converted to a NumPy array.

```Python
from FlagEmbedding import FlagModel
import numpy as np

# Assuming all_input_text is a list of strings
# Example: all_input_text = ["text1", "text2", ...]

embedding_model = FlagModel('BAAI/bge-large-zh-v1.5')

embeddings = []
for input_text in all_input_text:
    emb = embedding_model.encode(input_text)
    embeddings.append(emb)

embeddings_array = np.array(embeddings)
print("Shape: " + str(embeddings_array.shape), "\n")
print("Array:", embeddings_array[0])
```

--------------------------------

### Antiderivatives of Sine and Cosine (Mathematical Notation)

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Provides the antiderivatives (integrals) of the sine and cosine functions. These are essential for calculating the area under the curves of these trigonometric functions over a given interval.

```mathematical-notation
∫ sin(x) dx = -cos(x) + C
∫ cos(x) dx = sin(x) + C
where C is the constant of integration.
```

--------------------------------

### Cricket Data Snippets

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

These text snippets describe the sport of Cricket, including how it is played, the roles of players, and how runs are scored. This data can be used as a trusted source for LLM-based information retrieval.

```English
sen_1 = "Cricket is played between two teams of eleven on a field with a 22-yard pitch and a wicket at each end."
sen_2 = "Two players from the batting team stand at either wicket with bats, while the bowler from the fielding team bowls the ball towards the striker's wicket."
sen_3 = "The striker tries to hit the ball and switch places with the nonstriker to score runs; runs are also scored when the ball crosses the boundary or is bowled illegally."
sen_4 = "The fielding team tries to dismiss batters by hitting the wicket with the ball, catching the ball, or preventing the batters from crossing the crease."
```

--------------------------------

### Encode Query for Vector Search

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

This snippet demonstrates how to encode a textual query into a numerical vector using the same embedding model used for the data source. This vector can then be used for similarity comparisons or vector searches.

```Python
from FlagEmbedding import FlagModel

# Assuming embedding_model is already initialized
# embedding_model = FlagModel('BAAI/bge-large-zh-v1.5')

query = "What is cricket?"
q_ = embedding_model.encode(query)

# The query embedding 'q_' can now be used for vector search.
```

--------------------------------

### Formula One Data Snippets

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

These text snippets describe Formula One (F1) racing, including its status as the highest class of open-wheel racing, the structure of the World Championship, and the technical aspects of F1 cars. This data can be used for RAG systems.

```English
sen_9 = "Formula One (F1) is the highest class of international racing for open-wheel single-seater cars, governed by the Fdration Internationale de l'Automobile (FIA)."
sen_10 = "The FIA Formula One World Championship, started in 1950, is a series of races called Grands Prix, held on circuits or closed public roads across various countries."
sen_11 = "Points from Grands Prix determine annual World Championships for drivers and constructors, with drivers requiring a Super Licence and races occurring on grade one tracks."
sen_12 = "F1 cars, known for their high speeds due to aerodynamic downforce, have undergone changes to reduce turbulence and improve overtaking, relying heavily on electronics, aerodynamics, suspension, and tyres."
```

--------------------------------

### Generate Final Response

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

Generates the final response using an LLM, providing it with the retrieved information chunks. This step synthesizes the information to produce a coherent answer.

```Python
final_response = LLM.generate(chunks_S)
```

--------------------------------

### Retrieve Top N Similar Items

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

Identifies and retrieves the top 'n' most similar items based on pre-calculated similarity scores. It sorts the scores, selects the indices of the highest scores, and fetches the corresponding content.

```Python
import numpy as np

n = 3
sorted_indices = np.argsort(score_list)[::-1]
top_n_indices = sorted_indices[:n]

retrieved_content = []
for i in top_n_indices:
    print(all_input_text[i], "\n")
    retrieved_content.append(all_input_text[i])
```

--------------------------------

### Generate Wardrobe Embeddings with Python

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-8.txt

This Python script processes images of clothing items, generates descriptive text prompts, and then uses a text embedding model to create vector representations for each item. These vectors are stored, typically in a CSV file for smaller wardrobes or a scalable database for larger ones.

```python
import pandas as pd
from sentence_transformers import SentenceTransformer

# Placeholder for image processing and description generation
def describe_clothing_item(image_path):
    # In a real scenario, this would involve image analysis (e.g., using a vision model)
    # to extract details like type, color, style, material, etc.
    # For demonstration, we'll use a placeholder description.
    print(f"Processing image: {image_path}")
    # Example: return "A blue denim jacket with a collared neck and button closure."
    return "A description of the clothing item."

# Load a pre-trained sentence transformer model
# Example: 'all-MiniLM-L6-v2' is a good general-purpose model
model = SentenceTransformer('all-MiniLM-L6-v2')

# List of image paths for wardrobe items
wardrobe_images = [
    "path/to/wardrobe/item1.jpg",
    "path/to/wardrobe/item2.png",
    # ... more images
]

wardrobe_data = []

for img_path in wardrobe_images:
    description = describe_clothing_item(img_path)
    if description:
        # Generate embedding for the description
        embedding = model.encode(description)
        wardrobe_data.append({
            "description": description,
            "embedding": embedding.tolist() # Convert numpy array to list for saving
        })

# Save embeddings to a CSV file
if wardrobe_data:
    df = pd.DataFrame(wardrobe_data)
    df.to_csv("wardrobe_embeddings.csv", index=False)
    print("Wardrobe embeddings saved to wardrobe_embeddings.csv")
else:
    print("No wardrobe data generated.")

```

--------------------------------

### Text Vectorization using OpenAI Embeddings

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

This section describes the process of converting text files into numerical embeddings using the OpenAI 'text-embedding-ada-002' model. The text is first divided into manageable chunks, and the size of these chunks is optimized to improve retrieval metrics. Extracted entities are used as metadata for question answering systems.

```Python
# Conceptual code for vectorization:
# from openai import OpenAI
# client = OpenAI()
# 
# def get_embedding(text, model="text-embedding-ada-002"):
#    text = text.replace("\n", " ")
#    return client.embeddings.create(input = [text], model=model).data[0].embedding
# 
# # Example usage:
# # text_chunks = split_text_into_chunks(text_file)
# # embeddings = [get_embedding(chunk) for chunk in text_chunks]
```

--------------------------------

### Create Sparse and Non-Sparse Chunk Subsets

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

This pseudocode describes the process of iterating through a corpus of text chunks (T) to identify and categorize them into a subset of sparse information chunks (S) and a subset of non-sparse information chunks (R). The `identifySparseInformation` function determines if a chunk's information is unique to that chunk within the corpus.

```pseudocode
Initialize S <- ∅
Initialize R <- ∅
for each chunk c in T do
  isSparse <- identifySparseInformation(c, T)
  if isSparse then
    S <- S ∪ {c}
  else
    R <- R ∪ {c}
  end if
end for

Function identifySparseInformation(chunk, corpus)
  unique <- True
  for each otherChunk in corpus do
    if otherChunk ≠ chunk then
      if information in chunk is found in otherChunk then
        unique <- False
        break
      end if
    end if
  end for
  return unique
```

--------------------------------

### Validating Prompt for RAG Response Adequacy

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

This prompt is used to validate the adequacy of responses generated by an initial RAG approach, ensuring a seamless switch between methodologies based on user query requirements. It evaluates if a provided answer effectively resolves a given query.

```English
You are an intelligent bot designed to assist users on an organizations
website by answering their queries. Youll be given a users question and
an associated answer. Your task is to determine if the provided answer
effectively resolves the query. If the answer is unsatisfactory, return 0.
Query: {query}
Answer: {answer}
Your Feedback:
```

--------------------------------

### Implement sinpi in Julia

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

This snippet details the use of the 'sinpi' function in Julia, a high-level, high-performance dynamic programming language for technical computing. It computes the sine of pi times the input argument.

```Julia
sinpi(x)
```

--------------------------------

### Golf Data Snippet

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

This text snippet defines golf as a precision sport where players use clubs to hit balls into holes on a course in the fewest strokes possible. This data can be integrated into LLM knowledge bases.

```English
sen_13 = "Golf is a precision sport in which players use various clubs to hit balls into a series of holes on a course in as few strokes as possib"
```

--------------------------------

### Generate and Display Word Embedding

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

Demonstrates how to use the `get_embedding` function to obtain the vector representation of the word 'cat'. The resulting embedding is then stored in a pandas DataFrame, and its shape is printed, showing the dimensionality of the vector.

```python
import pandas as pd

word = "cat"
word_embedding = get_embedding(text=word, model="text-embedding-ada-002")
df_word = pd.DataFrame({'embed': word_embedding})
print(df_word.shape)
```

--------------------------------

### Web Crawling and HTML Parsing

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

This snippet details the process of obtaining HTML pages from a specified URL using a web crawler. It utilizes Python libraries such as 'requests', 're', 'urllib', and 'BeautifulSoup' for fetching and parsing HTML content. The goal is to extract relevant text while discarding unnecessary elements like headers and footers.

```Python
import requests
import re
import urllib
from bs4 import BeautifulSoup
from collections import *

# Example usage (conceptual):
# url = "https://i-venture.org"
# response = requests.get(url)
# soup = BeautifulSoup(response.content, 'html.parser')
# # Further processing to extract text and clean it
```

--------------------------------

### Sine and Cosine for Complex Arguments

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Presents the formulas for sine and cosine when their arguments are complex numbers, expressing them in terms of real trigonometric and hyperbolic functions.

```mathematics
cos(x + iy) = cos(x)cosh(y) - i*sin(x)sinh(y)
sin(x + iy) = sin(x)cosh(y) + i*cos(x)sinh(y)
```

--------------------------------

### Validate LLM Response

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-6.txt

Validates an initial response from a Large Language Model (LLM) using a provided query and the initial response. This step determines if the LLM's output is satisfactory.

```Python
validation = LLM.validate(Q, initial_response)
```

--------------------------------

### Dot Product Formula

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-1.txt

This snippet provides the mathematical formula for the dot product of two vectors A and B. It is used as a comparison metric against cosine similarity.

```latex
Dot Product (A \cdot B) = \sum_{i=1}^{n} A_i B_i
```

--------------------------------

### Embedding Generation and Cosine Similarity (Python)

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

This snippet shows how to generate embeddings for text chunks using a specified model ('text-embedding-ada-002') and calculate the cosine similarity between user input embeddings and the document embeddings. It utilizes libraries like pandas and numpy.

```Python
import pandas as pd
import numpy as np
from numpy.linalg import norm

# Assuming get_embedding function is defined elsewhere
def get_embedding(text, model):
    # Placeholder for actual embedding function
    return [0.1] * 1536 # Example embedding

text_chunks=[]
for text in texts:
    text_chunks.append(text.page_content)
df = pd.DataFrame({'text_chunks': text_chunks})
df['ada_embedding'] = df.text_chunks.apply(lambda x: get_embedding(x, model='text-embedding-ada-002'))

users_question = "Who was the winner in Fifa 2022 world cup?"
question_embedding = get_embedding(text=users_question, model="text-embedding-ada-002")

cos_sim = []
for index, row in df.iterrows():
   A = row.ada_embedding
   B = question_embedding
   cosine = np.dot(A,B)/(norm(A)*norm(B))
   cos_sim.append(cosine)
df["cos_sim"] = cos_sim
```

--------------------------------

### Fourier Series Coefficients

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Provides the formulas for calculating the coefficients of a trigonometric series when it represents a given integrable function, commonly known as Fourier series coefficients.

```mathematics
a_k = (1/pi) * integral_{-pi}^{pi} f(x) * cos(kx) dx
b_k = (1/pi) * integral_{-pi}^{pi} f(x) * sin(kx) dx
```

--------------------------------

### Configure ONNX Embedding Model and Data Type

Source: https://github.com/jparkerweb/chunk-match/blob/main/README.md

Customize the `chunkit` function by specifying the ONNX embedding model and its data type (precision). This affects embedding quality, chunking performance, and model size. Options for `dtype` include `fp32`, `fp16`, `q8`, and `q4`, with `fp32` offering the highest precision but `q8` providing a good balance.

```JavaScript
const chunkit = require('chunk-match');

// Example configuration
const options = {
  onnxEmbeddingModel: 'Xenova/all-MiniLM-L6-v2', // Or other ONNX models like 'nomic-ai/nomic-embed-text-v1.5'
  dtype: 'fp16' // Options: 'fp32', 'fp16', 'q8', 'q4'
};

// Initialize chunkit with options
const chunkMatcher = new chunkit(options);

// Use chunkMatcher for semantic text matching...
```

--------------------------------

### Sine and Cosine Definitions using Vectors

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Defines sine and cosine functions using vector operations, specifically the cross product and dot product. This approach is useful in vector calculus and physics.

```mathematics
sin(θ) = ||a x b|| / (||a|| ||b||)
cos(θ) = (a · b) / (||a|| ||b||)
```

--------------------------------

### Calculate Cosine Similarity Scores

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-5.txt

Computes cosine similarity scores between a query vector and a list of embedding vectors. It iterates through each embedding, calculates the similarity, and returns a list of scores.

```Python
from sklearn.metrics.pairwise import cosine_similarity

def search(embeddings:List, q_:List)->List[float]:
    """
    Search for the cosine similarity scores between a query vector (q_) and a list of embedding vectors.
    Parameters:
        embeddings (List[List[float]]): A list of embedding vectors.
        q_ (List[float]): The query vector for which the cosine similarity scores are calculated.
    Returns:
        List[float]: A list of cosine similarity scores between the query vector and each embedding vector.
    """
    scores = []
    for vec in embeddings:
        scores.append(cosine_similarity([vec], [q_])[0][0])
    return scores
```

--------------------------------

### Euclidean Distance Formula

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-1.txt

This snippet shows the mathematical formula for calculating the Euclidean distance between two vectors A and B. It is presented as another comparison metric for vector similarity.

```latex
Euclidean Distance = \sqrt{\sum_{i=1}^{n} (A_i - B_i)^2}
```

--------------------------------

### Create Text Embeddings with Pandas

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

This snippet shows how to create a Pandas DataFrame from a list of sentences and then generate vector embeddings for each sentence using a specified model (e.g., 'text-embedding-ada-002'). The embeddings are stored as a new column in the DataFrame.

```Python
import pandas as pd

contexts = ["I have a dog. My dog's name is Jimmy",             "I have a cat. My cat's name is biscuit",            "My dog who likes to listen to music",            "My cat likes to play with cricket balls "]
df_context = pd.DataFrame({'context': contexts})
df_context['embedding'] = df_context.context.apply(lambda x: get_embedding(x, model='text-embedding-ada-002'))
print(df_context)
```

--------------------------------

### Calculate Cosine Similarity between Embeddings

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

This code calculates the cosine similarity between a question's embedding and the embeddings of multiple text contexts. It iterates through the context embeddings, computes the dot product and magnitudes, and stores the similarity scores in a new column.

```Python
import numpy as np
from numpy.linalg import norm

question = "What is my cat's name?"
question_embedding = get_embedding(text=question, model="text-embedding-ada-002")

cos_sim = []
for index, row in df_context.iterrows():
   x = row.embedding
   y = question_embedding
   # calculate the cosine similiarity
   cosine = np.dot(x,y)/(norm(x)*norm(y))
   cos_sim.append(cosine)
df_context["cos_sim"] = cos_sim
print(df_context)
```

--------------------------------

### Implement sinpi in R

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

This snippet shows the usage of the 'sinpi' function in the R programming language. It computes the sine of pi times the input value, useful for statistical and mathematical analysis.

```R
sinpi(x)
```

--------------------------------

### Text Chunking with Langchain (Python)

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-2.txt

This code uses Langchain's RecursiveCharacterTextSplitter to divide the extracted text into smaller, manageable chunks. It specifies the chunk size and overlap to control the granularity of the text segments for further processing.

```Python
from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 500,
    chunk_overlap  = 100,
    length_function = len,
)

texts = text_splitter.create_documents([text])
```

--------------------------------

### MMR Diversity Penalty

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-4.txt

MMR introduces a diversity term to penalize highly similar documents, promoting a more diverse selection. The parameter 'lambda' controls the balance between relevance and diversity.

```Python
max(sim(D_i,Q)) - lambda * max(sim(D_i,D_j))
```

--------------------------------

### Cosine Similarity Formula

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-1.txt

This snippet presents the mathematical formula for calculating cosine similarity between two vectors A and B. It defines the formula using the dot product and the magnitudes of the vectors.

```latex
Cosine Similarity = \frac{A \cdot B}{||A|| ||B||}
```

--------------------------------

### Maximal Marginal Relevance (MMR) Formula

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-4.txt

MMR is used to select documents that are relevant to a query while also being diverse among themselves. It balances relevance with the novelty of selected items.

```latex
MMR = \lambda \times sim(D_i, Q) - (1 - \lambda) \times \max_{D_j \in S} sim(D_i, D_j)
```

--------------------------------

### Sine and Cosine Definitions using Unit Circle

Source: https://github.com/jparkerweb/chunk-match/blob/main/example/text-10.txt

Defines sine and cosine functions using the unit circle. The x and y coordinates of the intersection point of a line through the origin with the unit circle define cosine and sine, respectively.

```mathematics
cos(θ) = x
sin(θ) = y
```