ruby-spacy

https://github.com/yohasebe/ruby-spacy
Admin
ruby-spacy is a wrapper module for using spaCy, a natural language processing library, from Ruby via...
Tokens:13,568
Snippets:95
Trust Score:9.4
License:MIT
Update:6 months ago
Show doc for...
Tokens:
Raw
### Generate Syntax Tree using OpenAI Query

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This example shows how to use the ruby-spacy library to query OpenAI for generating a syntax tree diagram from a given sentence. It utilizes token properties and a specified bracketing style for the output. An API key is required.

```Ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("The Beatles released 12 studio albums")

# default parameter values
# max_completion_tokens: 1000
# model: "gpt-5-mini"
res = doc.openai_query(
  access_token: api_key,
  prompt: "Generate a tree diagram from the text using given token data. Use the following bracketing style: [S [NP [Det the] [N cat]] [VP [V sat] [PP [P on] [NP the mat]]"
)
puts res
```

--------------------------------

### Configure Python Environment for ruby-spacy

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Instructions to install Python with shared library support and the necessary spaCy language models.

```shell
env CONFIGURE_OPTS="--enable-shared" pyenv install 3.10.6
pyenv global 3.10.6
pip install spacy
python -m spacy download en_core_web_sm
```

--------------------------------

### Visualizing Dependency Trees (SVG)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This example demonstrates how to generate an SVG visualization of dependency parses for English text using ruby-spacy. The 'displacy' method is used with the 'dep' style to create a visual representation of the grammatical relationships between words in a sentence. The output is saved to an SVG file.

```ruby
require "ruby-spacy"

lp = Spacy::Language.new("en_core_web_sm")

sentence = "Autonomous cars shift insurance liability toward manufacturers"
doc = nlp.read(sentence)

dep_svg = doc.displacy(style: "dep", compact: false)

File.open(File.join("test_dep.svg"), "w") do |file|
  file.write(dep_svg)
end
```

--------------------------------

### GET /doc/linguistic_summary

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Generates a structured JSON summary of spaCy's analysis for use as LLM context.

```APIDOC
## GET /doc/linguistic_summary

### Description
Generates a JSON summary of spaCy's analysis including tokens, entities, and noun chunks.

### Method
GET

### Endpoint
/doc/linguistic_summary

### Parameters
#### Query Parameters
- **sections** (array) - Optional - List of sections to include (e.g., [:text, :tokens, :entities])
- **token_attributes** (array) - Optional - List of attributes to include for tokens

### Response
#### Success Response (200)
- **summary** (object) - JSON object containing linguistic data

#### Response Example
{
  "text": "Apple Inc. acquired a startup.",
  "entities": [{"text": "Apple Inc.", "label": "ORG"}]
}
```

--------------------------------

### Visualizing Dependency Trees (Compact SVG)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This snippet illustrates how to generate a compact SVG visualization of dependency parses for English text with ruby-spacy. Similar to the previous example, it uses the 'displacy' method with 'dep' style but sets 'compact' to true for a more condensed visualization. The result is saved to a separate SVG file.

```ruby
require "ruby-spacy"

lp = Spacy::Language.new("en_core_web_sm")

sentence = "Autonomous cars shift insurance liability toward manufacturers"
doc = nlp.read(sentence)

dep_svg = doc.displacy(style: "dep", compact: true)

File.open(File.join("test_dep_compact.svg"), "w") do |file|
  file.write(dep_svg)
end
```

--------------------------------

### Generate Text Embeddings using OpenAI

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This example illustrates how to generate text embeddings for a given sentence using OpenAI's embedding models through the ruby-spacy library. It requires an OpenAI API key and uses the default 'text-embedding-3-small' model.

```Ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Vladimir Nabokov was a Russian-American novelist, poet, translator and entomologist.")

# default model: text-embedding-3-small
res = doc.openai_embeddings(access_token: api_key)

puts res
```

--------------------------------

### Efficient OpenAI API Usage with Ruby spaCy

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

This section covers the `Language#with_openai` method for block-based OpenAI API interactions in ruby-spacy. It shows examples for single document analysis, batch processing using `nlp.pipe`, and generating embeddings. This approach is efficient for multiple API calls.

```ruby
require "ruby-spacy"

lp = Spacy::Language.new("en_core_web_sm")

# Single document analysis
doc = nlp.read("Apple Inc. was founded by Steve Jobs in California.")

lp.with_openai(model: "gpt-5-mini") do |ai|
  result = ai.chat(
    system: "You are a linguistic analyst. Analyze the given linguistic data.",
    user: doc.linguistic_summary
  )
  puts result
end

# Batch processing with pipe
texts = ["The bank approved the loan.", "I sat on the river bank."]

lp.with_openai(model: "gpt-5-mini") do |ai|
  nlp.pipe(texts).each do |doc|
    result = ai.chat(
      system: "Identify the meaning of 'bank' in one word based on context.",
      user: doc.linguistic_summary
    )
    puts "#{doc.text} => #{result}"
  end
end

# Generate embeddings
lp.with_openai do |ai|
  vector = ai.embeddings("Hello world")
  puts "Embedding dimensions: #{vector.length}"
end
```

--------------------------------

### Initialize spaCy Language Models

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Demonstrates how to load different spaCy language models (small, large, or multi-language) using the Spacy::Language class, including optional timeout settings.

```ruby
require "ruby-spacy"

# Load English small model (fast, no word vectors)
nlp = Spacy::Language.new("en_core_web_sm")

# Load English large model (includes word vectors for similarity)
nlp_lg = Spacy::Language.new("en_core_web_lg")

# Load Japanese model
nlp_ja = Spacy::Language.new("ja_core_news_lg")

# With custom timeout for slow model loading
nlp = Spacy::Language.new("en_core_web_sm", timeout: 120)

# Check pipeline components
puts nlp.pipe_names
```

--------------------------------

### Perform Tokenization with ruby-spacy

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Demonstrates how to initialize a spaCy language model in Ruby and tokenize a text string.

```ruby
require "ruby-spacy"
require "terminal-table"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple is looking at buying U.K. startup for $1 billion")

row = []
doc.each { |token| row << token.text }
puts Terminal::Table.new(rows: [row])
```

--------------------------------

### Calculate Word Vector Analogies in Japanese

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Demonstrates how to perform vector arithmetic (e.g., Tokyo - Japan + France = Paris) using Japanese language models in ruby-spacy.

```ruby
require "ruby-spacy"
require "terminal-table"

nlp = Spacy::Language.new("ja_core_news_lg")

tokyo = nlp.get_lexeme("東京")
japan = nlp.get_lexeme("日本")
france = nlp.get_lexeme("フランス")

query = tokyo.vector - japan.vector + france.vector

headings = ["rank", "text", "score"]
rows = []

results = nlp.most_similar(query, 10)
results.each_with_index do |lexeme, i|
  index = (i + 1).to_s
  rows << [index, lexeme.text, lexeme.score]
end

table = Terminal::Table.new rows: rows, headings: headings
puts table
```

--------------------------------

### Integrate OpenAI Queries

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Shows how to use Doc#openai_query to send text to OpenAI models, optionally leveraging linguistic context for tasks like translation or POS analysis.

```ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("The Beatles released 12 studio albums")

# Translation query
result = doc.openai_query(
  access_token: api_key,
  prompt: "Translate the text to Japanese.",
  model: "gpt-5-mini",
  max_completion_tokens: 1000
)
puts result

# Query with linguistic analysis
doc2 = nlp.read("The quick brown fox jumps over the lazy dog")
result2 = doc2.openai_query(
  access_token: api_key,
  prompt: "List each word with its part of speech and dependency role in JSON format."
)
puts result2
```

--------------------------------

### Perform Dependency Parsing and Tree Navigation

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Demonstrates how to iterate over tokens to inspect dependency relations and navigate the syntactic tree using properties like head, children, ancestors, and subtree.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("The quick brown fox jumps over the lazy dog")

doc.each do |token|
  puts "#{token.text} --#{token.dep}--> #{token.head.text}"
end

root = doc.tokens.find { |t| t.dep == "ROOT" }
puts "Subtree of '#{root.text}':"
root.subtree.each { |t| puts "  #{t.text}" }

puts "Children of '#{root.text}': #{root.children.map(&:text).join(', ')}"

leaf = doc.tokens.last
puts "Ancestors of '#{leaf.text}': #{leaf.ancestors.map(&:text).join(' -> ')}"
```

--------------------------------

### Integrate OpenAI GPT Models

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Demonstrates how to use the doc.openai_query method to process text with GPT models, including translation and entity extraction.

```ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("The Beatles released 12 studio albums")

# Translation example
res1 = doc.openai_query(
  access_token: api_key,
  prompt: "Translate the text to Japanese."
)

# Elaboration example
doc2 = nlp.read("The Beatles were an English rock band formed in Liverpool in 1960.")
res2 = doc2.openai_query(
  access_token: api_key,
  prompt: "Extract the topic of the document and list 10 entities."
)
```

--------------------------------

### Perform Text Completion with OpenAI

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Demonstrates the use of Doc#openai_completion to generate text completions based on existing document content.

```ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Vladimir Nabokov was a")

result = doc.openai_completion(
  access_token: api_key,
  model: "gpt-5-mini",
  max_completion_tokens: 200
)
puts result
```

--------------------------------

### Text Completion using OpenAI

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This snippet demonstrates how to perform text completion using OpenAI's capabilities via the ruby-spacy library. It takes a partial sentence as input and generates a coherent continuation. Requires an OpenAI API key.

```Ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Vladimir Nabokov was a")

# default parameter values
# max_completion_tokens: 1000
# model: "gpt-5-mini"
res = doc.openai_completion(access_token: api_key)
puts res
```

--------------------------------

### Integrate OpenAI with spaCy Linguistic Analysis

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Demonstrates how to use the with_openai block to send linguistic summaries generated by spaCy to an LLM for analysis. It includes basic usage, batch processing with pipe, and customization of the summary data.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple Inc. was founded by Steve Jobs in California.")

nlp.with_openai(model: "gpt-5-mini") do |ai|
  result = ai.chat(
    system: "You are a linguistic analyst. Analyze the given linguistic data.",
    user: doc.linguistic_summary
  )
  puts result
end

# Batch processing
nlp.with_openai(model: "gpt-5-mini") do |ai|
  nlp.pipe(["The bank approved the loan.", "I sat on the river bank."]).each do |doc|
    result = ai.chat(
      system: "Identify the meaning of 'bank' in one word based on the linguistic context.",
      user: doc.linguistic_summary
    )
    puts "#{doc.text} => #{result}"
  end
end

# Customizing summary
summary = doc.linguistic_summary(
  sections: [:text, :tokens, :entities, :sentences],
  token_attributes: [:text, :lemma, :pos, :dep, :head, :morphology]
)
```

--------------------------------

### Perform Phrase Matching with PhraseMatcher

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Demonstrates how to use the PhraseMatcher class for efficient terminology matching. It covers both case-sensitive and case-insensitive matching configurations.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")

# Case-sensitive matching
matcher = nlp.phrase_matcher
matcher.add("PRODUCT", ["iPhone", "MacBook Pro", "iPad"])

doc = nlp.read("I bought an iPhone and a MacBook Pro yesterday.")
matches = matcher.match(doc)

matches.each do |span|
  puts "#{span.text} => #{span.label}"
end

# Case-insensitive matching
matcher_lower = nlp.phrase_matcher(attr: "LOWER")
matcher_lower.add("COMPANY", ["apple", "google", "microsoft"])

doc2 = nlp.read("Apple and GOOGLE are competitors of Microsoft.")
matches2 = matcher_lower.match(doc2)

matches2.each do |span|
  puts "#{span.text} => #{span.label}"
end
```

--------------------------------

### Batch Process Text with Pipe

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Demonstrates efficient processing of multiple texts using the Language#pipe method, which is optimized for large-scale operations.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")

texts = [
  "Apple reported strong earnings.",
  "Google announced a new AI model.",
  "Microsoft acquired a gaming company."
]

# Batch process texts
docs = nlp.pipe(texts, batch_size: 50)

docs.each do |doc|
  entities = doc.ents.map { |e| "#{e.text}(#{e.label})" }.join(", ")
  puts "#{doc.text[0..30]}... -> Entities: #{entities}"
end
```

--------------------------------

### Language Model Initialization

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Initializes a spaCy language model for text processing.

```APIDOC
## POST /Spacy::Language.new

### Description
Loads a specific spaCy language model (e.g., en_core_web_sm, ja_core_news_lg) to enable linguistic analysis.

### Method
POST

### Endpoint
Spacy::Language.new(model_name, options = {})

### Parameters
#### Path Parameters
- **model_name** (string) - Required - The name of the installed spaCy model.

#### Request Body
- **timeout** (integer) - Optional - Custom timeout in seconds for loading the model.

### Response
#### Success Response (200)
- **nlp** (object) - An instance of the language model ready for processing.
```

--------------------------------

### Generate Dependency and Entity Visualizations

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Uses the displacy method to export dependency parse trees as SVG files and named entity recognition results as HTML files.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Autonomous cars shift insurance liability toward manufacturers")

dep_svg = doc.displacy(style: "dep", compact: false)
File.write("dependency_tree.svg", dep_svg)

dep_svg_compact = doc.displacy(style: "dep", compact: true)
File.write("dependency_tree_compact.svg", dep_svg_compact)

doc_ner = nlp.read("Apple Inc. was founded by Steve Jobs in California.")
ent_html = doc_ner.displacy(style: "ent")
File.write("entities.html", ent_html)
```

--------------------------------

### Serialize and Deserialize Documents

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Explains how to save processed documents to binary format and restore them later to avoid redundant NLP processing.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple Inc. was founded by Steve Jobs in California.")

# Serialize document to binary
bytes = doc.to_bytes
File.binwrite("doc_cache.bin", bytes)
puts "Serialized #{bytes.bytesize} bytes"

# Restore document from cache
cached_bytes = File.binread("doc_cache.bin")
restored_doc = Spacy::Doc.from_bytes(nlp, cached_bytes)

puts "Restored text: #{restored_doc.text}"
restored_doc.ents.each do |ent|
  puts "Entity: #{ent.text} (#{ent.label})"
end
```

--------------------------------

### Perform Efficient Phrase Matching

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Shows how to use PhraseMatcher for efficient terminology extraction, supporting case-insensitive matching and multiple category labeling.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")

# Basic usage
matcher = nlp.phrase_matcher
matcher.add("PRODUCT", ["iPhone", "MacBook Pro", "iPad"])

doc = nlp.read("I bought an iPhone and a MacBook Pro yesterday.")
matches = matcher.match(doc)

# Case-insensitive matching
matcher = nlp.phrase_matcher(attr: "LOWER")
matcher.add("COMPANY", ["apple", "google", "microsoft"])

# Multiple categories
matcher = nlp.phrase_matcher(attr: "LOWER")
matcher.add("TECH_COMPANY", ["apple", "google", "microsoft", "amazon"])
matcher.add("PRODUCT", ["iphone", "pixel", "surface", "kindle"])
```

--------------------------------

### Part-of-Speech and Dependency Tagging (Japanese)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This snippet demonstrates how to perform part-of-speech (POS) and dependency tagging on Japanese text using ruby-spacy. It utilizes the 'ja_core_news_lg' model and the 'terminal-table' gem for formatted output, showing token text, lemma, POS tag, detailed tag, and dependency relation.

```ruby
require "ruby-spacy"
require "terminal-table"

lp = Spacy::Language.new("ja_core_news_lg")
doc = nlp.read("任天堂は1983年にファミコンを14,800円で発売した。")

headings = ["text", "lemma", "pos", "tag", "dep"]
rows = []

doc.each do |token|
  rows << [token.text, token.lemma, token.pos, token.tag, token.dep]
end

table = Terminal::Table.new rows: rows, headings: headings
puts table
```

--------------------------------

### Extract Part-of-speech and Dependencies

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Shows how to extract linguistic annotations such as lemmas, POS tags, and dependency labels from a document using ruby-spacy.

```ruby
require "ruby-spacy"
require "terminal-table"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple is looking at buying U.K. startup for $1 billion")

rows = doc.map { |token| [token.text, token.lemma, token.pos, token.tag, token.dep] }
puts Terminal::Table.new(rows: rows, headings: ["text", "lemma", "pos", "tag", "dep"])
```

--------------------------------

### POST /openai/chat

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Sends a linguistic summary of a spaCy document to an OpenAI chat model for analysis.

```APIDOC
## POST /openai/chat

### Description
Sends a JSON-formatted linguistic summary of a processed spaCy document to an OpenAI model for analysis or inference.

### Method
POST

### Endpoint
nlp.with_openai(model: String) { |ai| ai.chat(...) }

### Parameters
#### Request Body
- **system** (string) - Required - The system instruction for the AI model.
- **user** (string) - Required - The linguistic summary generated by doc.linguistic_summary.

### Request Example
{
  "system": "You are a linguistic analyst.",
  "user": "{\"tokens\": [...], \"entities\": [...]}"
}

### Response
#### Success Response (200)
- **result** (string) - The text response from the OpenAI model.

#### Response Example
{
  "result": "The subject is Apple Inc."
}
```

--------------------------------

### OpenAI Query Integration

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Integrates spaCy document analysis with OpenAI's GPT API for translation, analysis, and text completion.

```APIDOC
## Doc#openai_query

### Description
Sends text to OpenAI's GPT API with optional linguistic context.

### Parameters
#### Request Body
- **access_token** (string) - Required - OpenAI API key.
- **prompt** (string) - Required - The query or instruction for the model.
- **model** (string) - Optional - The GPT model to use.

### Response
- **result** (string) - The generated response from the OpenAI API.
```

--------------------------------

### Process Text and Extract Token Attributes

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Shows how to read text into a Doc object and iterate through tokens to access linguistic attributes like text, lemma, POS tags, and dependency labels.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple is looking at buying U.K. startup for $1 billion")

# Iterate over tokens
doc.each do |token|
  puts "#{token.text} -> #{token.lemma} (#{token.pos})"
end

# Access token attributes
doc.tokens.each do |token|
  puts "Text: #{token.text}, Lemma: #{token.lemma}, POS: #{token.pos}, Tag: #{token.tag}, Dep: #{token.dep}"
end
```

--------------------------------

### Extract Token Data with Meaning using OpenAI Query

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This snippet demonstrates how to query OpenAI to extract detailed token data for each word in a sentence, including a 'meaning' property with a brief semantic definition. It requires an OpenAI API key and uses the 'gpt-5-mini' model.

```Ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("The Beatles released 12 studio albums")

# default parameter values
# max_completion_tokens: 1000
# model: "gpt-5-mini"
res = doc.openai_query(
  access_token: api_key,
  prompt: "List token data of each of the words used in the sentence. Add 'meaning' property and value (brief semantic definition) to each token data. Output as a JSON object."
)
puts res
```

--------------------------------

### Configure spaCy Language Timeout

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Configures the timeout duration for the spaCy language model initialization.

```ruby
nlp = Spacy::Language.new("en_core_web_sm", timeout: 120)
```

--------------------------------

### Document Serialization

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Methods for serializing and deserializing spaCy documents to binary format for efficient caching.

```APIDOC
## Doc#to_bytes

### Description
Serializes the document object into a binary format.

### Method
GET

### Response
- **bytes** (binary) - The serialized document data.

## Doc.from_bytes

### Description
Reconstructs a document object from binary data.

### Method
POST

### Parameters
#### Request Body
- **nlp** (object) - Required - The language model instance.
- **bytes** (binary) - Required - The binary data to deserialize.
```

--------------------------------

### POST /openai/embeddings

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Generates vector embeddings for a given text string using OpenAI's embedding models.

```APIDOC
## POST /openai/embeddings

### Description
Generates a numerical vector representation (embedding) for a provided text string.

### Method
POST

### Endpoint
nlp.with_openai { |ai| ai.embeddings(text) }

### Parameters
#### Request Body
- **text** (string) - Required - The input text to convert into an embedding.

### Request Example
{
  "text": "Hello world"
}

### Response
#### Success Response (200)
- **vector** (array) - An array of floats representing the embedding.

#### Response Example
{
  "vector": [0.012, -0.045, ...]
}
```

--------------------------------

### POST /doc/openai_embeddings

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Generates text embeddings for a document using OpenAI's embedding models.

```APIDOC
## POST /doc/openai_embeddings

### Description
Generates text embeddings using OpenAI's embedding models for semantic search and similarity tasks.

### Method
POST

### Endpoint
/doc/openai_embeddings

### Parameters
#### Request Body
- **access_token** (string) - Required - OpenAI API key
- **model** (string) - Required - The embedding model to use (e.g., text-embedding-3-small)

### Request Example
{
  "access_token": "sk-...",
  "model": "text-embedding-3-small"
}

### Response
#### Success Response (200)
- **embeddings** (array) - An array of floats representing the vector dimensions

#### Response Example
[-0.002084, -0.016452, 0.01110, 0.012802, 0.001218]
```

--------------------------------

### Phrase Matching with PhraseMatcher

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

The PhraseMatcher class provides an efficient way to match terminology lists against documents, supporting both case-sensitive and case-insensitive matching.

```APIDOC
## PhraseMatcher.add

### Description
Adds a new pattern to the phrase matcher.

### Method
POST (Internal Method)

### Parameters
#### Request Body
- **label** (string) - Required - The label to assign to the matched phrase.
- **phrases** (array) - Required - A list of strings to match against the document.

### Request Example
matcher.add("PRODUCT", ["iPhone", "MacBook Pro"])
```

--------------------------------

### Perform Word Vector Arithmetic

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Retrieves lexemes and performs vector arithmetic to solve analogies, finding the most similar words based on the resulting vector.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_lg")

tokyo = nlp.get_lexeme("Tokyo")
japan = nlp.get_lexeme("Japan")
france = nlp.get_lexeme("France")

query = tokyo.vector - japan.vector + france.vector

results = nlp.most_similar(query, 10)
results.each_with_index do |result, i|
  puts "#{i + 1}. #{result.text} (score: #{result.score.round(4)})"
end
```

--------------------------------

### Serialize and Restore Documents

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Provides methods to serialize processed spaCy documents into binary format for efficient storage and restoration, preserving all annotations.

```ruby
nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple Inc. was founded by Steve Jobs in California.")

# Save to file
File.binwrite("doc_cache.bin", doc.to_bytes)

# Restore from file
bytes = File.binread("doc_cache.bin")
restored_doc = Spacy::Doc.from_bytes(nlp, bytes)
puts restored_doc.text
```

--------------------------------

### Access Lexeme Attributes and Vocabulary

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Retrieves information about specific vocabulary items and calculates similarity between individual lexemes.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_lg")

lexeme = nlp.vocab("apple")
puts "Text: #{lexeme.text}"
puts "Is stop: #{lexeme.is_stop}"

dog = nlp.vocab("dog")
cat = nlp.vocab("cat")
puts "dog <-> cat: #{dog.similarity(cat)}"
```

--------------------------------

### Execute Pattern Matching with Matcher

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Defines rule-based patterns using token attributes to identify specific sequences within a document.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")

matcher = nlp.matcher
pattern = [[{ LOWER: "hello" }, { IS_PUNCT: true }, { LOWER: "world" }]]
matcher.add("HelloWorld", pattern)

doc = nlp.read("Hello, world! Hello world!")

matches = matcher.match(doc)
matches.each do |match|
  label = nlp.vocab_string_lookup(match[:match_id])
  span = doc.span(match[:start_index]..match[:end_index])
  puts "#{label}: '#{span.text}'"
end
```

--------------------------------

### Manipulate Document Spans

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Shows how to create and interact with Span objects. It includes techniques for slicing documents, accessing root tokens, and iterating over span contents.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple Inc. is looking at buying U.K. startup for $1 billion")

# Create spans using range
span = doc.span(0..2)
puts "Span text: #{span.text}"

# Create span with start position and size
span2 = doc.span(0, 3)
puts "Span2 text: #{span2.text}"

# Access span properties
puts "Root token: #{span.root.text}"
puts "Sentence: #{span.sent.text}"

# Iterate over span tokens
span.each do |token|
  puts "  #{token.text} (#{token.pos})"
end

# Get entities within span
doc.ents.each do |ent|
  puts "Entity span: #{ent.text} [#{ent.label}]"
end
```

--------------------------------

### Similarity Calculation (Japanese)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Calculates and prints the similarity score between two Japanese text documents.

```APIDOC
## Similarity Calculation (Japanese)

### Description
Calculates and prints the similarity score between two Japanese text documents.

### Method
Not applicable (Ruby code)

### Endpoint
Not applicable (Ruby code)

### Parameters
None

### Request Example
```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("ja_core_news_lg")
ja_doc1 = nlp.read("今日は雨ばっかり降って、嫌な天気ですね。")
puts "doc1: #{ja_doc1.text}"
ja_doc2 = nlp.read("あいにくの悪天候で残念です。")
puts "doc2: #{ja_doc2.text}"
puts "Similarity: #{ja_doc1.similarity(ja_doc2)}"
```

### Response
#### Success Response (Text Output)
```text
doc1: 今日は雨ばっかり降って、嫌な天気ですね。
doc2: あいにくの悪天候で残念です。
Similarity: 0.8684192637149641
```
```

--------------------------------

### Access Token Morphology in Ruby spaCy

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

This code snippet shows how to retrieve detailed morphological features for a token using the `Token#morphology` method in ruby-spacy. The features can be returned as a hash (default) or a string representation.

```ruby
require "ruby-spacy"

lp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple is looking at buying U.K. startup")

doc.each do |token|
  morph = token.morphology  # Returns hash by default
  morph_str = token.morphology(hash: false)  # Returns string
  puts "#{token.text}: #{morph}" unless morph.empty?
end
```

--------------------------------

### Segment Text into Sentences

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Splits a document into individual sentence spans using the Doc#sents method based on the model's boundary detection.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("This is a sentence. This is another sentence.")

puts "Has sentence boundaries: #{doc.has_annotation("SENT_START")}"

doc.sents.each_with_index do |sent, i|
  puts "Sentence #{i + 1}: #{sent.text}"
end
```

--------------------------------

### Generate OpenAI Embeddings with Ruby spaCy

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

This snippet demonstrates how to generate text embeddings using OpenAI's models via the `Doc#openai_embeddings` method in ruby-spacy. It requires an OpenAI API key and specifies the model to use. The output includes the dimensions of the embedding vector and its first few values.

```ruby
require "ruby-spacy"

api_key = ENV["OPENAI_API_KEY"]
lp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Vladimir Nabokov was a Russian-American novelist.")

# Generate embeddings
embeddings = doc.openai_embeddings(
  access_token: api_key,
  model: "text-embedding-3-small"
)

puts "Vector dimensions: #{embeddings.length}"
puts "First 5 values: #{embeddings.first(5).map { |v| v.round(6) }}"
```

--------------------------------

### Retokenization Operations in Ruby spaCy

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Demonstrates the `Doc#retokenize` and `Doc#retokenize_split` methods in ruby-spacy for manipulating tokens within a document. `retokenize` merges multiple tokens into one, while `retokenize_split` performs the inverse operation.

```ruby
require "ruby-spacy"

lp = Spacy::Language.new("en_core_web_sm")

# Merge tokens
doc = nlp.read("I live in New York City")
puts "Before merge: #{doc.tokens.map(&:text).join(' | ')}"

# Merge "New York City" into single token (positions 3, 4, 5)
doc.retokenize(3, 5, { LEMMA: "New York City" })
puts "After merge: #{doc.tokens.map(&:text).join(' | ')}"
```

--------------------------------

### Check Word Vector Availability

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Analyzes tokens in a document to determine if they possess word vectors, their vector norms, and whether they are out-of-vocabulary (OOV).

```ruby
require "ruby-spacy"
require "terminal-table"

nlp = Spacy::Language.new("en_core_web_lg")
doc = nlp.read("dog cat banana afskfsd")

rows = []

doc.each do |token|
  rows << [token.text, token.has_vector, token.vector_norm, token.is_oov]
end

headings = ["text", "has_vector", "vector_norm", "is_oov"]
table = Terminal::Table.new rows: rows, headings: headings
puts table
```

--------------------------------

### Morphology Analysis (English)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

This code snippet shows how to extract morphological features from English text using ruby-spacy. It employs the 'en_core_web_sm' model and 'terminal-table' for display. The output includes the token's text, shape, whether it's alphabetic or a stop word, and its detailed morphology.

```ruby
require "ruby-spacy"
require "terminal-table"

lp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple is looking at buying U.K. startup for $1 billion")

headings = ["text", "shape", "is_alpha", "is_stop", "morphology"]
rows = []

doc.each do |token|
  morph = token.morphology.map do |k, v|
    "#{k} = #{v}"
  end.join("\n")
  rows << [token.text, token.shape, token.is_alpha, token.is_stop, morph]
end

table = Terminal::Table.new rows: rows, headings: headings
puts table
```

--------------------------------

### Similarity Calculation (English)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Calculates and prints the similarity score between two English text documents.

```APIDOC
## Similarity Calculation (English)

### Description
Calculates and prints the similarity score between two English text documents.

### Method
Not applicable (Ruby code)

### Endpoint
Not applicable (Ruby code)

### Parameters
None

### Request Example
```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_lg")
doc1 = nlp.read("I like salty fries and hamburgers.")
doc2 = nlp.read("Fast food tastes very good.")

puts "Doc 1: " + doc1.text
puts "Doc 2: " + doc2.text
puts "Similarity: #{doc1.similarity(doc2)}"
```

### Response
#### Success Response (Text Output)
```text
Doc 1: I like salty fries and hamburgers.
Doc 2: Fast food tastes very good.
Similarity: 0.7687607012190486
```
```

--------------------------------

### Generate Linguistic Summaries for LLMs in Ruby spaCy

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

The `Doc#linguistic_summary` method in ruby-spacy creates a JSON representation of a document's linguistic features, suitable for providing context to Large Language Models (LLMs). It supports default sections and customizable sections and token attributes.

```ruby
require "ruby-spacy"

lp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple Inc. acquired a startup in California.")

# Default summary (text, tokens, entities, noun_chunks)
summary = doc.linguistic_summary
puts summary

# Customized summary
custom_summary = doc.linguistic_summary(
  sections: [:text, :tokens, :entities, :sentences],
  token_attributes: [:text, :lemma, :pos, :dep, :head, :morphology]
)
puts custom_summary
```

--------------------------------

### POST /doc/retokenize

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Modifies the token structure of a document by merging or splitting tokens.

```APIDOC
## POST /doc/retokenize

### Description
Merges multiple tokens into a single token or splits existing tokens.

### Method
POST

### Endpoint
/doc/retokenize

### Parameters
#### Request Body
- **start_index** (integer) - Required - Starting token index
- **end_index** (integer) - Required - Ending token index
- **attributes** (object) - Optional - Attributes for the new token (e.g., LEMMA)

### Response
#### Success Response (200)
- **status** (string) - Confirmation of retokenization
```

--------------------------------

### Named Entity Recognition (Japanese)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Performs Named Entity Recognition on Japanese text and displays the results in a table.

```APIDOC
## Named Entity Recognition (Japanese)

### Description
Performs Named Entity Recognition on Japanese text and displays the results in a table.

### Method
Not applicable (Ruby code)

### Endpoint
Not applicable (Ruby code)

### Parameters
None

### Request Example
```ruby
require( "ruby-spacy")
require "terminal-table"

nlp = Spacy::Language.new("ja_core_news_lg")

sentence = "任天堂は1983年にファミコンを14,800円で発売した。"
doc = nlp.read(sentence)

rows = []

doc.ents.each do |ent|
  rows << [ent.text, ent.start_char, ent.end_char, ent.label]
end

headings = ["text", "start", "end", "label"]
table = Terminal::Table.new rows: rows, headings: headings
print table
```

### Response
#### Success Response (Table Output)
```text
| text       | start | end | label   |
|:-----------|------:|----:|:--------|
| 任天堂     | 0     | 3   | ORG     |
| 1983年     | 4     | 9   | DATE    |
| ファミコン | 10    | 15  | PRODUCT |
| 14,800円   | 16    | 23  | MONEY   |
```
```

--------------------------------

### Text Processing and Tokenization

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Processes raw text into a Doc object containing tokens and linguistic annotations.

```APIDOC
## POST /Language#read

### Description
Processes a string of text and returns a Doc object containing tokens with attributes like lemma, POS, and dependency labels.

### Method
POST

### Endpoint
nlp.read(text)

### Parameters
#### Request Body
- **text** (string) - Required - The raw text to be processed.

### Response
#### Success Response (200)
- **doc** (object) - A Doc object containing tokens and their linguistic annotations.
```

--------------------------------

### Named Entity Recognition (English)

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Performs Named Entity Recognition on English text and displays the results in a table.

```APIDOC
## Named Entity Recognition (English)

### Description
Performs Named Entity Recognition on English text and displays the results in a table.

### Method
Not applicable (Ruby code)

### Endpoint
Not applicable (Ruby code)

### Parameters
None

### Request Example
```ruby
require "ruby-spacy"
require "terminal-table"

nlp = Spacy::Language.new("en_core_web_sm")
doc = nlp.read("Apple is looking at buying U.K. startup for $1 billion")

rows = []

doc.ents.each do |ent|
  rows << [ent.text, ent.start_char, ent.end_char, ent.label]
end

headings = ["text", "start_char", "end_char", "label"]
table = Terminal::Table.new rows: rows, headings: headings
puts table
```

### Response
#### Success Response (Table Output)
```text
| text       | start_char | end_char | label |
|:-----------|-----------:|---------:|:------|
| Apple      | 0          | 5        | ORG   |
| U.K.       | 27         | 31       | GPE   |
| $1 billion | 44         | 54       | MONEY |
```
```

--------------------------------

### Calculate Document Similarity

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Computes the semantic similarity score between two documents using spaCy's vector-based similarity method.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_lg")
doc1 = nlp.read("I like salty fries and hamburgers.")
doc2 = nlp.read("Fast food tastes very good.")

puts "Doc 1: " + doc1.text
puts "Doc 2: " + doc2.text
puts "Similarity: #{doc1.similarity(doc2)}"
```

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("ja_core_news_lg")
ja_doc1 = nlp.read("今日は雨ばっかり降って、嫌な天気ですね。")
puts "doc1: #{ja_doc1.text}"
ja_doc2 = nlp.read("あいにくの悪天候で残念です。")
puts "doc2: #{ja_doc2.text}"
puts "Similarity: #{ja_doc1.similarity(ja_doc2)}"
```

--------------------------------

### Perform Named Entity Recognition

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Extracts named entities from text using spaCy language models. It outputs the entity text, character offsets, and labels in a formatted table.

```ruby
require "ruby-spacy"
require "terminal-table"

nlp = Spacy::Language.new("en_core_web_sm")
doc =nlp.read("Apple is looking at buying U.K. startup for $1 billion")

rows = []

doc.ents.each do |ent|
  rows << [ent.text, ent.start_char, ent.end_char, ent.label]
end

headings = ["text", "start_char", "end_char", "label"]
table = Terminal::Table.new rows: rows, headings: headings
puts table
```

```ruby
require( "ruby-spacy")
require "terminal-table"

nlp = Spacy::Language.new("ja_core_news_lg")

sentence = "任天堂は1983年にファミコンを14,800円で発売した。"
doc = nlp.read(sentence)

rows = []

doc.ents.each do |ent|
  rows << [ent.text, ent.start_char, ent.end_char, ent.label]
end

headings = ["text", "start", "end", "label"]
table = Terminal::Table.new rows: rows, headings: headings
print table
```

--------------------------------

### Compute Document Similarity

Source: https://context7.com/yohasebe/ruby-spacy/llms.txt

Calculates semantic similarity between documents using word vectors. Requires a model with vector support like en_core_web_lg.

```ruby
require "ruby-spacy"

nlp = Spacy::Language.new("en_core_web_lg")

doc1 = nlp.read("I like salty fries and hamburgers.")
doc2 = nlp.read("Fast food tastes very good.")
doc3 = nlp.read("The capital of France is Paris.")

puts "Similarity: #{doc1.similarity(doc2)}"
puts "Similarity: #{doc1.similarity(doc3)}"
```

--------------------------------

### Checking Availability of Word Vectors

Source: https://github.com/yohasebe/ruby-spacy/blob/main/README.md

Checks if words in a given text have associated word vectors and if they are out-of-vocabulary (OOV).

```APIDOC
## Checking Availability of Word Vectors

### Description
Checks if words in a given text have associated word vectors and if they are out-of-vocabulary (OOV).

### Method
Not applicable (Ruby code)

### Endpoint
Not applicable (Ruby code)

### Parameters
None

### Request Example
```ruby
require "ruby-spacy"
require "terminal-table"

nlp = Spacy::Language.new("en_core_web_lg")
doc = nlp.read("dog cat banana afskfsd")

rows = []

doc.each do |token|
  rows << [token.text, token.has_vector, token.vector_norm, token.is_oov]
end

headings = ["text", "has_vector", "vector_norm", "is_oov"]
table = Terminal::Table.new rows: rows, headings: headings
puts table
```

### Response
#### Success Response (Table Output)
```text
| text    | has_vector | vector_norm | is_oov |
|:--------|:-----------|:------------|:-------|
| dog     | true       | 7.0336733   | false  |
| cat     | true       | 6.6808186   | false  |
| banana  | true       | 6.700014    | false  |
| afskfsd | false      | 0.0         | true   |
```
```
Tokens:
Raw