spaCy

https://spacy.io
Admin
spaCy is an open-source library for industrial-strength Natural Language Processing (NLP) in Python,...
Tokens:956,345
Snippets:6,963
Trust Score:9.7
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Quickstart: Install spaCy and download en_core_web_sm

Source: https://spacy.io/usage

Quickstart installation commands for spaCy using pip, including upgrading pip/setuptools/wheel, installing spaCy, and downloading the en_core_web_sm pipeline.

```bash
pip install -U pip setuptools wheel
pip install -U spacy
python -m spacy download en_core_web_sm
```

--------------------------------

### Split Example into sentences with Example.split_sents

Source: https://spacy.io/api/example

Demonstrates splitting an Example into one Example per sentence using Example.split_sents. The example creates an Example with sentence start flags and asserts the text of the resulting split examples.

```python
doc = nlp("I went yesterday had lots of fun")
tokens_ref = ["I", "went", "yesterday", "had", "lots", "of", "fun"]
sents_ref = [True, False, False, True, False, False, False]
example = Example.from_dict(doc, {"words": tokens_ref, "sent_starts": sents_ref})
split_examples = example.split_sents()
assert split_examples[0].text == "I went yesterday "
assert split_examples[1].text == "had lots of fun"
```

--------------------------------

### Install spacy-huggingface-hub

Source: https://spacy.io/universe/project/spacy-huggingface-hub

Install the package using pip. This is the required setup before using the CLI command.

```bash
pip install spacy-huggingface-hub
```

--------------------------------

### Install DVC and initialize Git and DVC repos

Source: https://spacy.io/usage/examples

Run these commands to install DVC, initialize a Git repository, and initialize a DVC project. This is the required setup before using DVC with a spaCy project.

```bash
pip install dvc   # Install DVC
git init          # Initialize a Git repo
dvc init          # Initialize a DVC project
```

--------------------------------

### spacy package command examples

Source: https://spacy.io/api/cli

Generate an installable Python package from an existing pipeline data directory. The first example shows the full command with all options. The second shows a typical workflow: package the pipeline, change into the generated package directory, and install the built artifact with pip.

```bash
python -m spacy package input_dir output_dir --code --meta-path --create-meta --build --name --version --force
```

```bash
python -m spacy package /input /output
cd /output/en_pipeline-0.0.0
pip install dist/en_pipeline-0.0.0.tar.gz
```

--------------------------------

### tmtoolkit example: corpus loading, normalization, DTM, tf-idf, and top tokens

Source: https://spacy.io/universe/project/tmtoolkit

Shows the full tmtoolkit workflow: load a built-in corpus, inspect as a table, normalize text, build a document-term matrix, apply tf-idf, and print top tokens per document. Requires setup steps: pip install tmtoolkit[recommended] and python -m tmtoolkit setup en. Uses 4 worker processes and prints a pandas DataFrame of tokens.

```python
# Note: This requires these setup steps:
#   pip install tmtoolkit[recommended]
#   python -m tmtoolkit setup en
from tmtoolkit.corpus import Corpus, tokens_table, lemmatize, to_lowercase, dtm
from tmtoolkit.bow.bow_stats import tfidf, sorted_terms_table
# load built-in sample dataset and use 4 worker processes
corp = Corpus.from_builtin_corpus('en-News100', max_workers=4)
# investigate corpus as dataframe
toktbl = tokens_table(corp)
print(toktbl)
# apply some text normalization
lemmatize(corp)
to_lowercase(corp)
# build sparse document-token matrix (DTM)
# document labels identify rows, vocabulary tokens identify columns
mat, doc_labels, vocab = dtm(corp, return_doc_labels=True, return_vocab=True)
# apply tf-idf transformation to DTM
# operation is applied on sparse matrix and uses few memory
tfidf_mat = tfidf(mat)
# show top 5 tokens per document ranked by tf-idf
top_tokens = sorted_terms_table(tfidf_mat, vocab, doc_labels, top_n=5)
print(top_tokens)
```

--------------------------------

### Install and use fr_solipcysme pipeline

Source: https://spacy.io/universe/project/solipcysme

Install the fr_solipcysme pipeline from Hugging Face and load it with spaCy. The example iterates over a French sentence and prints token attributes including custom extensions tokentype, vv_pos, and vv_morph.

```bash
pip install https://huggingface.co/thjbdvlt/fr_solipcysme/resolve/main/fr_solipcysme-any-py3-none-any.whl
```

```python
import spacy

nlp = spacy.load('fr_solipcysme')
for i in nlp(
'la MACHINE à (b)rouiller le temps s'est peut-être déraillée..?'
):
    print(
        i, 
        i.norm_, 
        i.pos_, 
        i.morph, 
        i.lemma_, 
        i.dep_, 
        i._.tokentype,
        i._.vv_pos,
        i._.vv_morph
    )
```

--------------------------------

### Initialize llm component with get_examples() callback

Source: https://spacy.io/usage/large-language-models

Configuration for initializing the llm component with a get_examples callback. Set n_prompt_examples to a positive number to fetch examples, or -1 to use all examples.

```ini
[initialize.components.llm]
n_prompt_examples = 3
```

--------------------------------

### Get a Sub-Span with Span.__getitem__ Slice

Source: https://spacy.io/api/span

Retrieves a Span object by slicing a Span with a tuple of start and end indices. The example creates a span from tokens 1 to 4 and asserts that the slice from 1 to 3 has the text "back!".

```python
doc = nlp("Give it back! He pleaded.")
span = doc[1:4]
assert span[1:3].text == "back!"
```

--------------------------------

### Install Coreferee with pip

Source: https://spacy.io/universe/project/coreferee

Install the Coreferee package using pip. Models must be installed separately before running the example.

```bash
pip install coreferee
```

--------------------------------

### initialize(get_examples, *, nlp=None, patterns=None)

Source: https://spacy.io/api/spanruler

The initialize method prepares the component for training, optionally adding patterns.

```APIDOC
## initialize(get_examples, *, nlp=None, patterns=None)

### Description
Initialize the pipe for training.

### Parameters
- **get_examples** (Callable[[], Iterable[Example]]) - Required - Function that returns a representative sample of gold-standard Example objects.
- **nlp** (Language) - Optional - The current nlp object the component is part of.
- **patterns** (Optional[Iterable[PatternType]]) - Optional - The list of patterns.

### Documentation
https://spacy.io/api/spanruler#initialize
```

--------------------------------

### Fill config with defaults and train with spacy train

Source: https://spacy.io/usage/v3

After exporting a starter config from the quickstart widget, use `init fill-config` to fill it with all default values, then train with the config file. The diff shows the v2 command replaced by the v3 command.

```bash
- python -m spacy train en ./output ./train.json ./dev.json
--pipeline tagger,parser --cnn-window 1 --bilstm-depth 0
+ python -m spacy train ./config.cfg --output ./output
```

--------------------------------

### Install spacy-streamlit

Source: https://spacy.io/usage/examples

Install the spacy-streamlit package with the pre-release flag to get the latest version.

```bash
pip install spacy-streamlit --pre
```

--------------------------------

### Install spacy-experimental

Source: https://spacy.io/api/coref

Install the spacy-experimental package, which provides the CoreferenceResolver component starting in version 0.6.0.

```bash
pip install -U spacy-experimental
```

--------------------------------

### Create an Example with Example.from_dict

Source: https://spacy.io/api/annotation

Shows the basic call to create an Example object from a reference Doc and a dictionary of gold-standard annotations.

```python
example = Example.from_dict(doc, gold_dict)
```

--------------------------------

### Example.__init__

Source: https://spacy.io/api/example

Constructs an Example object from predicted and reference documents, with optional alignment.

```APIDOC
## Example.__init__

### Description
Construct an `Example` object from the `predicted` document and the `reference` document. If `alignment` is `None`, it will be initialized from the words in both documents.

### Parameters
- **predicted** (Doc) - Required - The document containing (partial) predictions. Cannot be `None`.
- **reference** (Doc) - Required - The document containing gold-standard annotations. Cannot be `None`.
- **alignment** (Optional[Alignment]) - Optional - An object holding the alignment between the tokens of the `predicted` and `reference` documents. Keyword-only.

### Example
```
from spacy.tokens import Doc
from spacy.training import Example
pred_words = ["Apply", "some", "sunscreen"]
pred_spaces = [True, True, False]
gold_words = ["Apply", "some", "sun", "screen"]
gold_spaces = [True, True, False, False]
gold_tags = ["VERB", "DET", "NOUN", "NOUN"]
predicted = Doc(nlp.vocab, words=pred_words, spaces=pred_spaces)
reference = Doc(nlp.vocab, words=gold_words, spaces=gold_spaces, tags=gold_tags)
example = Example(predicted, reference)
```
```

--------------------------------

### Create Example with gold-standard POS tags using Example.from_dict

Source: https://spacy.io/usage/training

Alternative way to create a reference Doc with gold-standard annotations using Example.from_dict, which takes a dictionary with keyword arguments like tags or entities. This is less verbose than manually building the reference Doc.

```python
words = ["I", "like", "stuff"]
tags = ["NOUN", "VERB", "NOUN"]
predicted = Doc(nlp.vocab, words=words)
example = Example.from_dict(predicted, {"tags": tags})
```

--------------------------------

### Install adept-augmentations with pip

Source: https://spacy.io/universe/project/adeptaugmentations

Install the library using pip. This is the required setup before using the augmenter.

```bash
pip install adept-augmentations
```

--------------------------------

### Dependency Visualizer options example

Source: https://spacy.io/api/top-level

Shows how to pass options to `displacy.serve` for the dependency visualizer (`style="dep"`). The example sets `compact` to `True` and `color` to `"blue"`.

```python
options = {"compact": True, "color": "blue"}
displacy.serve(doc, style="dep", options=options)
```

--------------------------------

### Install benepar with pip

Source: https://spacy.io/universe/project/self-attentive-parser

Install the benepar package using pip. This is the required setup before using the parser.

```bash
pip install benepar
```

--------------------------------

### Initialize Tok2Vec with examples

Source: https://spacy.io/api/tok2vec

Shows how to initialize the tok2vec component for training. The get_examples function must return at least one Example object, and the nlp object is passed as a keyword argument.

```python
tok2vec = nlp.add_pipe("tok2vec")
tok2vec.initialize(lambda: examples, nlp=nlp)
```

--------------------------------

### Tagger.get_loss example

Source: https://spacy.io/api/tagger

Computes the loss and gradient for a batch of documents and their predicted scores. The example shows how to get scores from the tagger and then call get_loss with the examples and scores.

```python
tagger = nlp.add_pipe("tagger")
scores = tagger.predict([eg.predicted for eg in examples])
loss, d_loss = tagger.get_loss(examples, scores)
```

--------------------------------

### Get package path with util.get_package_path

Source: https://spacy.io/api/top-level

Gets the path to an installed package. Currently imports the package to find its path.

```python
util.get_package_path("en_core_web_sm")
# /usr/lib/python3.6/site-packages/en_core_web_sm
```

--------------------------------

### Install the package to make entry points available

Source: https://spacy.io/usage/saving-loading

Command to install the package in the current environment so spaCy can discover the entry points.

```bash
python -m pip install .
```

--------------------------------

### Vocab.__init__ example

Source: https://spacy.io/api/vocab

Creates a Vocab instance with a list of strings. The strings are used to initialize the StringStore, mapping them to hash values.

```python
from spacy.vocab import Vocab
vocab = Vocab(strings=["hello", "world"])
```

--------------------------------

### Install spacy-cleaner with pip

Source: https://spacy.io/universe/project/spacy-cleaner

Install the spacy-cleaner package using pip. This is the required setup step before using the library.

```bash
pip install spacy-cleaner
```

--------------------------------

### Initialize DependencyParser for training with initialize

Source: https://spacy.io/api/dependencyparser

Shows how to initialize the parser component for training. The get_examples callback must return at least one Example. The nlp object is passed as a keyword argument. The config.cfg example shows how to provide labels via the [initialize.components.parser] block.

```python
parser = nlp.add_pipe("parser")
parser.initialize(lambda: examples, nlp=nlp)
```

```ini
[initialize.components.parser]

[initialize.components.parser.labels]
@readers = "spacy.read_labels.v1"
path = "corpus/labels/parser.json
```

--------------------------------

### Install scrubadub_spacy with pip

Source: https://spacy.io/universe/project/scrubadub_spacy

Install the scrubadub_spacy package using pip. This is the required setup step before using the library.

```bash
pip install scrubadub-spacy
```

--------------------------------

### Example: initialize DVC and generate config

Source: https://spacy.io/api/cli

Example workflow for the `spacy project dvc` command: initializes a git repository, initializes DVC, and generates the DVC config for the `all` workflow.

```bash
git init
dvc init
python -m spacy project dvc all
```

--------------------------------

### Install lemminflect with pip

Source: https://spacy.io/universe/project/lemminflect

Install the lemminflect package using pip. This is the required setup step before using the library.

```bash
pip install lemminflect
```

--------------------------------

### Install Holmes with pip

Source: https://spacy.io/universe/project/holmes

Install the holmes-extractor package using pip. This is the required setup step before using the library.

```bash
pip install holmes-extractor
```

--------------------------------

### PlainTextCorpus.__init__ example

Source: https://spacy.io/api/corpus

Initializes a PlainTextCorpus with a path to a text file.

```python
from spacy.training import PlainTextCorpus

corpus = PlainTextCorpus("./data/docs.txt")
```

--------------------------------

### Install HuSpaCy with pip

Source: https://spacy.io/universe/project/HuSpaCy

Install the huspacy package using pip. This is the required setup step before using the model.

```bash
pip install huspacy
```

--------------------------------

### Expose entry point in setup.py

Source: https://spacy.io/usage/saving-loading

Setup file that exposes the 'snek' component via the 'spacy_factories' entry point group, mapping the name to the function snek_component in the snek module.

```python
#### setup.py
from setuptools import setup

setup(
    name="snek",
    entry_points={
        "spacy_factories": ["snek = snek:snek_component"]
    }
)
```

--------------------------------

### util.get_package_path

Source: https://spacy.io/api/top-level

Get path to an installed package.

```APIDOC
## util.get_package_path

### Description
Get path to an installed package. Mainly used to resolve the location of pipeline packages. Currently imports the package to find its path.

### Method
function

### Parameters
- `package_name` (str) - Required - Name of installed package.

### Returns
Path to pipeline package directory. Path

### Example
```
util.get_package_path("en_core_web_sm")
# /usr/lib/python3.6/site-packages/en_core_web_sm
```
```

--------------------------------

### InMemoryLookupKB.to_disk example

Source: https://spacy.io/api/inmemorylookupkb

Saves the current state of the knowledge base to a directory. The directory will be created if it doesn't exist.

```python
kb.to_disk(path)
```

--------------------------------

### Install PyTextRank with pip

Source: https://spacy.io/universe/project/spacy-pytextrank

Install the pytextrank package using pip. This is the required setup before using the library in a spaCy pipeline.

```bash
pip install pytextrank
```

--------------------------------

### Tagger.initialize example and config

Source: https://spacy.io/api/tagger

Shows how to initialize the tagger component with a lambda function providing examples and the nlp object. The config.cfg block sets up the labels reader and path.

```python
tagger = nlp.add_pipe("tagger")
tagger.initialize(lambda: examples, nlp=nlp)
```

```ini
[initialize.components.tagger]

[initialize.components.tagger.labels]
@readers = "spacy.read_labels.v1"
path = "corpus/labels/tagger.json"
```

--------------------------------

### Holmes example: register search phrase and start chatbot

Source: https://spacy.io/universe/project/holmes

Example usage of Holmes: import the library, create a Manager with a spaCy model, register a search phrase, and start the chatbot mode console. This demonstrates the core workflow for interactive information extraction.

```python
import holmes_extractor as holmes
holmes_manager = holmes.Manager(model='en_core_web_lg')
holmes_manager.register_search_phrase('A big dog chases a cat')
holmes_manager.start_chatbot_mode_console()
```

--------------------------------

### spacy project document example

Source: https://spacy.io/api/cli

Example of generating project documentation to README.md. If the file already exists and contains auto-generated docs, only the auto-generated section is replaced.

```bash
python -m spacy project document --output README.md
```

--------------------------------

### Get aligned parse with Example.get_aligned_parse

Source: https://spacy.io/api/example

Creates an Example from a parsed doc and reference heads, then gets the aligned projective parse heads and labels.

```python
doc = nlp("He pretty quickly walks away")
example = Example.from_dict(doc, {"heads": [3, 2, 3, 0, 2]})
proj_heads, proj_labels = example.get_aligned_parse(projectivize=True)
assert proj_heads == [3, 2, 3, 0, 3]
```

--------------------------------

### InMemoryLookupKB.from_disk example

Source: https://spacy.io/api/inmemorylookupkb

Restores the state of the knowledge base from a directory. The Vocab must be the same as the one used to create the KB.

```python
from spacy.vocab import Vocab
vocab = Vocab().from_disk("/path/to/vocab")
kb = InMemoryLookupKB(vocab=vocab, entity_vector_length=64)
kb.from_disk("/path/to/kb")
```

--------------------------------

### PhraseMatcher.__len__ example

Source: https://spacy.io/api/phrasematcher

Gets the number of rules added to the matcher. The example shows that initially the count is 0, and after adding a rule it becomes 1.

```python
  matcher = PhraseMatcher(nlp.vocab)
  assert len(matcher) == 0
  matcher.add("OBAMA", [nlp("Barack Obama")])
  assert len(matcher) == 1
```

--------------------------------

### Token.__len__ example

Source: https://spacy.io/api/token

Gets the number of unicode characters in a token using len(token). The example asserts that the first token 'Give' has length 4.

```python
doc = nlp("Give it back! He pleaded.")
token = doc[0]
assert len(token) == 4
```

--------------------------------

### Migrate simple training style to Example

Source: https://spacy.io/usage/v3

Convert the simple training style from v2 to v3 by creating a doc and an Example object from the annotations.

```python
text = "Mark Zuckerberg is the CEO of Facebook"
annotations = {"entities": [(0, 15, "PERSON"), (30, 38, "ORG")]}
+ doc = nlp.make_doc(text)
+ example = Example.from_dict(doc, annotations)
```

--------------------------------

### Span.rights example

Source: https://spacy.io/api/span

Shows how to get the right children of a span. The example creates a doc, extracts a span for 'like New', and asserts that the rights are ['in'].

```python
doc = nlp("I like New York in Autumn.")
rights = [t.text for t in doc[2:4].rights]
assert rights == ["in"]
```
Tokens:
Raw