### Clone Repository and Install Dependencies

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Installs Node.js dependencies and clones the project repository. Ensure Node.js version is >=18.0.0.

```sh
git clone git@github.com:pinecone-io/semantic-search-example.git
cd semantic-search-example
npm install
```

--------------------------------

### Example Query Execution

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

This command shows how to execute a query against the populated Pinecone index using the project's CLI. It specifies the query text and the number of top results to retrieve.

```sh
npm start -- query --query="which city has the highest population in the world?" --topK=2
```

--------------------------------

### Indexer Progress Output Example

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Illustrates the expected output during the indexer's operation, showing the creation of the index, waiting for it to be ready, and the progress of the embedding and insertion process with a visual progress bar.

```sh
Creating index semantic-search
Waiting until index is ready...
Index ready after 45 seconds
Index is ready.
█████████████████████░░░░░░░░░░░░░░░░░░░ 52% | ETA: 291s | 52000/99999
```

--------------------------------

### Example Query Result

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

This is an example of the JSON output received after performing a semantic search query. It includes the text of the most similar documents and their corresponding relevance scores.

```js
[
  {
    text: 'Which country in the world has the largest population?',
    score: 0.79473877,
  },
  {
    text: 'Which cities are the most densely populated?',
    score: 0.706895828,
  },
];
```

--------------------------------

### Example Result for Advanced Query

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

This JSON output shows the results of a semantic search query using alternative phrasing. Despite the different vocabulary, the results are semantically relevant, demonstrating the power of understanding meaning over keywords.

```js
[
  {
    text: 'Which cities are the most densely populated?',
    score: 0.66688776,
  },
  {
    text: 'What are the most we dangerous cities in the world?',
    score: 0.556335568,
  },
];
```

--------------------------------

### Deleting the Pinecone Index

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

This command is used to clean up resources by deleting the Pinecone index after the example queries are complete. It's a good practice to remove indexes when they are no longer needed.

```sh
npm start -- delete
```

--------------------------------

### Command Line Interface for Loading Data

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Provides instructions on how to run the data loading script using npm. It specifies the command structure and arguments required for loading data from a CSV file into Pinecone.

```sh
npm start -- load --csvPath=<path-to-csv-file> --column=<column-name>
```

--------------------------------

### Build Project

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Builds the project using npm. This command compiles the TypeScript code and prepares the application for execution.

```sh
npm run build
```

--------------------------------

### Configure Pinecone Credentials

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Copies the environment variable template and fills in Pinecone API key, index name, cloud, and region. These are required for interacting with the Pinecone API.

```sh
cp .env.example .env
PINECONE_API_KEY=<your-api-key>
PINECONE_INDEX="semantic-search"
PINECONE_CLOUD="aws"
PINECONE_REGION="us-west-2"
```

--------------------------------

### Indexing Specific Columns from Test CSV

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Demonstrates how to index specific columns ('question1' and 'question2') from a 'test.csv' file using the provided command-line interface. This process involves running the loading script twice, once for each column.

```sh
npm start -- load --csvPath=test.csv --column=question1
npm start -- load --csvPath=test.csv --column=question2
```

--------------------------------

### Advanced Query Execution with Different Phrasing

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

This command demonstrates executing a query with significantly different wording than the expected results, highlighting the effectiveness of semantic search. It uses more complex language to ask about urban populations.

```sh
npm start -- query --query="which urban locations have the highest concentration of homo sapiens?" --topK=2
```

--------------------------------

### Load Embeddings into Pinecone

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Loads data from a CSV file, creates a Pinecone index, embeds the data in batches, and upserts the embeddings into the index. It uses environment variables for configuration and a progress bar for feedback.

```typescript
import cliProgress from 'cli-progress';
import { config } from 'dotenv';
import loadCSVFile from './csvLoader.js';

import { embedder } from './embeddings.js';
import { Pinecone } from '@pinecone-database/pinecone';
import { getEnv, validateEnvironmentVariables } from './utils/util.js';

import type { TextMetadata } from './types.js';

// Load environment variables from .env
config();

const progressBar = new cliProgress.SingleBar(
  {},
  cliProgress.Presets.shades_classic
);

let counter = 0;

export const load = async (csvPath: string, column: string) => {
  validateEnvironmentVariables();

  // Get a Pinecone instance
  const pinecone = new Pinecone();

  // Create a readable stream from the CSV file
  const { data, meta } = await loadCSVFile(csvPath);

  // Ensure the selected column exists in the CSV file
  if (!meta.fields?.includes(column)) {
    console.error(`Column ${column} not found in CSV file`);
    process.exit(1);
  }

  // Extract the selected column from the CSV file
  const documents = data.map((row) => row[column] as string);

  // Get index name, cloud, and region
  const indexName = getEnv('PINECONE_INDEX');
  const indexCloud = getEnv('PINECONE_CLOUD');
  const indexRegion = getEnv('PINECONE_REGION');

  // Create a Pinecone index with a dimension of 384 to hold the outputs
  // of our embeddings model. Use suppressConflicts in case the index already exists.
  await pinecone.createIndex({
    name: indexName,
    dimension: 384,
    spec: {
      serverless: {
        region: indexRegion,
        cloud: indexCloud,
      },
    },
    waitUntilReady: true,
    suppressConflicts: true,
  });

  // Select the target Pinecone index. Passing the TextMetadata generic type parameter
  // allows typescript to know what shape to expect when interacting with a record's
  // metadata field without the need for additional type casting.
  const index = pinecone.index<TextMetadata>(indexName);

  // Start the progress bar
  progressBar.start(documents.length, 0);

  // Start the batch embedding process
  await embedder.init();
  await embedder.embedBatch(documents, 100, async (embeddings) => {
    counter += embeddings.length;
    // Whenever the batch embedding process returns a batch of embeddings, insert them into the index
    await index.upsert(embeddings);
    progressBar.update(counter);
  });

  progressBar.stop();
  console.log(`Inserted ${documents.length} documents into index ${indexName}`);
};

```

--------------------------------

### Querying a Pinecone Index

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

This TypeScript code demonstrates how to query a Pinecone index for semantic similarity. It embeds a user's query using a provided embedder, then uses the Pinecone client to find the topK most similar vectors in the index. Results include metadata and scores.

```typescript
import {
  config
} from 'dotenv';
import { embedder } from './embeddings.js';
import { Pinecone } from '@pinecone-database/pinecone';
import { getEnv, validateEnvironmentVariables } from './utils/util.js';
import type { TextMetadata } from './types.js';

config();

export const query = async (query: string, topK: number) => {
  validateEnvironmentVariables();
  const pinecone = new Pinecone();

  // Target the index
  const indexName = getEnv('PINECONE_INDEX');
  const index = pinecone.index<TextMetadata>(indexName);

  await embedder.init();

  // Embed the query
  const queryEmbedding = await embedder.embed(query);

  // Query the index using the query embedding
  const results = await index.query({
    vector: queryEmbedding.values,
    topK,
    includeMetadata: true,
    includeValues: false,
  });

  // Print the results
  console.log(
    results.matches?.map((match) => ({
      // eslint-disable-next-line @typescript-eslint/ban-ts-comment
      // @ts-ignore
      text: match.metadata?.text,
      score: match.score,
    }))
  );
};
```

--------------------------------

### Load CSV File

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Loads and parses a CSV file into JavaScript objects using papaparse. It handles file reading, parsing with dynamic typing, and header detection.

```typescript
import fs from 'fs/promises';
import Papa from 'papaparse';

async function loadCSVFile(
  filePath: string
): Promise<Papa.ParseResult<Record<string, unknown>>> {
  try {
    // Get csv file absolute path
    const csvAbsolutePath = await fs.realpath(filePath);

    // Create a readable stream from the CSV file
    const data = await fs.readFile(csvAbsolutePath, 'utf8');

    // Parse the CSV file
    return await Papa.parse(data, {
      dynamicTyping: true,
      header: true,
      skipEmptyLines: true,
    });
  } catch (err) {
    console.error(err);
    throw err;
  }
}

export default loadCSVFile;
```

--------------------------------

### Embedder Class for Text Embeddings

Source: https://github.com/pinecone-io/semantic-search-example/blob/main/README.md

Generates text embeddings using the Xenova/all-MiniLM-L6-v2 model from the @xenova/transformers library. Supports embedding single strings or batches of strings.

```typescript
import type { PineconeRecord } from '@pinecone-database/pinecone';
import type { TextMetadata } from './types.js';
import { Pipeline } from '@xenova/transformers';
import { v4 as uuidv4 } from 'uuid';
import { sliceIntoChunks } from './utils/util.js';

class Embedder {
  private pipe: Pipeline | null = null;

  // Initialize the pipeline
  async init() {
    const { pipeline } = await import('@xenova/transformers');
    this.pipe = await pipeline('embeddings', 'Xenova/all-MiniLM-L6-v2');
  }

  // Embed a single string
  async embed(text: string): Promise<PineconeRecord<TextMetadata>> {
    const result = this.pipe && (await this.pipe(text));
    return {
      id: uuidv4(),
      metadata: {
        text,
      },
      values: Array.from(result.data),
    };
  }

  // Batch an array of string and embed each batch
  // Call onDoneBatch with the embeddings of each batch
  async embedBatch(
    texts: string[],
    batchSize: number,
    onDoneBatch: (embeddings: PineconeRecord<TextMetadata>[]) => void
  ) {
    const batches = sliceIntoChunks<string>(texts, batchSize);
    for (const batch of batches) {
      const embeddings = await Promise.all(
        batch.map((text) => this.embed(text))
      );
      await onDoneBatch(embeddings);
    }
  }
}

const embedder = new Embedder();

export { embedder };
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.