### Install CJK Support for Wordfreq

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Installs additional external dependencies required for correct tokenization of Chinese, Japanese, and Korean languages. This can be included in your project's dependency list.

```python
pip install wordfreq[cjk]
```

```toml
[tool.poetry.dependencies]
wordfreq = {extras = ["cjk"], version = "*"}
```

--------------------------------

### GET /get_language_info

Source: https://context7.com/rspeer/wordfreq/llms.txt

Retrieves configuration details for how a specific language is processed.

```APIDOC
## GET /get_language_info

### Description
Returns a dictionary with information about how text in a specific language is processed, including script, tokenizer type, and normalization rules.

### Method
GET

### Endpoint
/get_language_info

### Parameters
#### Query Parameters
- **lang** (string) - Required - The ISO language code.

### Request Example
GET /get_language_info?lang=zh

### Response
#### Success Response (200)
- **info** (object) - Metadata regarding script, tokenizer, and normalization.

#### Response Example
{
  "script": "Hans",
  "tokenizer": "jieba",
  "normal_form": "NFC"
}
```

--------------------------------

### GET /available_languages

Source: https://context7.com/rspeer/wordfreq/llms.txt

Retrieves a dictionary of supported language codes and their available wordlist configurations.

```APIDOC
## GET /available_languages

### Description
Returns a dictionary of available language codes and their corresponding data files. Useful for discovering which languages are supported and which wordlist sizes are available.

### Method
GET

### Endpoint
/available_languages

### Parameters
#### Query Parameters
- **wordlist** (string) - Optional - The size of the wordlist to check (e.g., 'small', 'large').

### Request Example
GET /available_languages?wordlist=large

### Response
#### Success Response (200)
- **languages** (object) - A dictionary where keys are language codes and values are metadata about the available data files.

#### Response Example
{
  "en": "...",
  "es": "..."
}
```

--------------------------------

### Get Full Frequency Dictionary

Source: https://context7.com/rspeer/wordfreq/llms.txt

Returns a complete mapping of words to their frequency values for a specified language. This is highly efficient for performing batch lookups of word frequencies.

```python
from wordfreq import get_frequency_dict

freq_dict = get_frequency_dict('en')
words_to_check = ['hello', 'world', 'python', 'frequency']
for word in words_to_check:
    freq = freq_dict.get(word, 0)
    print(f"{word}: {freq}")
```

--------------------------------

### Get Word Frequency (Python)

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Retrieves the frequency of a given word in a specified language. Handles numerical tokens by applying aggregation and distribution rules. Dependencies include the wordfreq library.

```python
from wordfreq import word_frequency

print(word_frequency("2022", "en"))
print(word_frequency("1922", "en"))
print(word_frequency("1022", "en"))
print(word_frequency("90210", "en"))
print(word_frequency("92222", "en"))
print(word_frequency("802.11n", "en"))
print(word_frequency("899.19n", "en"))
print(word_frequency("٥٤", "ar"))
print(word_frequency("54", "ar"))
print(word_frequency("٥٤", "en"))
```

--------------------------------

### GET /get_frequency_dict

Source: https://context7.com/rspeer/wordfreq/llms.txt

Retrieves the full frequency dictionary for a specified language.

```APIDOC
## GET /get_frequency_dict

### Description
Returns all word frequencies for a language as a dictionary mapping words to their frequency values.

### Method
GET

### Endpoint
/get_frequency_dict

### Parameters
#### Query Parameters
- **lang** (string) - Required - The ISO language code.
- **wordlist** (string) - Optional - The size of the wordlist to use.

### Request Example
GET /get_frequency_dict?lang=en&wordlist=large

### Response
#### Success Response (200)
- **frequency_map** (object) - A mapping of words to their frequency values.

#### Response Example
{
  "hello": 0.0001,
  "world": 0.00005
}
```

--------------------------------

### GET /random_words

Source: https://context7.com/rspeer/wordfreq/llms.txt

Generates a sequence of random words based on frequency data.

```APIDOC
## GET /random_words

### Description
Returns a string of random, space-separated words from a language's wordlist, useful for generating passwords or test data.

### Method
GET

### Endpoint
/random_words

### Parameters
#### Query Parameters
- **lang** (string) - Required - The ISO language code.
- **nwords** (integer) - Required - Number of words to generate.
- **bits_per_word** (integer) - Required - Entropy level per word.

### Request Example
GET /random_words?lang=en&nwords=5&bits_per_word=12

### Response
#### Success Response (200)
- **words** (string) - A space-separated string of random words.

#### Response Example
{
  "words": "house water about through before"
}
```

--------------------------------

### GET /zipf_frequency

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Retrieves the frequency of a word on a human-friendly logarithmic Zipf scale.

```APIDOC
## GET /zipf_frequency

### Description
Returns the word frequency on the Zipf scale, which is the base-10 logarithm of the number of times it appears per billion words.

### Method
GET

### Endpoint
zipf_frequency(word, lang, wordlist='best', minimum=0.0)

### Parameters
#### Path Parameters
- **word** (string) - Required - The Unicode string containing the word to look up.
- **lang** (string) - Required - The BCP 47 or ISO 639 code of the language.

#### Query Parameters
- **wordlist** (string) - Optional - The set of word frequencies to use ('small', 'large', 'best').
- **minimum** (float) - Optional - The minimum value to return if the word is not found.

### Request Example
zipf_frequency('the', 'en')

### Response
#### Success Response (200)
- **zipf_value** (float) - The Zipf frequency value (typically between 0 and 8).

#### Response Example
7.73
```

--------------------------------

### GET /word_frequency

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Retrieves the frequency of a specific word in a given language as a decimal value.

```APIDOC
## GET /word_frequency

### Description
Looks up a word's frequency in the given language, returning its frequency as a decimal between 0 and 1.

### Method
GET

### Endpoint
word_frequency(word, lang, wordlist='best', minimum=0.0)

### Parameters
#### Path Parameters
- **word** (string) - Required - The Unicode string containing the word to look up.
- **lang** (string) - Required - The BCP 47 or ISO 639 code of the language (e.g., 'en').

#### Query Parameters
- **wordlist** (string) - Optional - The set of word frequencies to use ('small', 'large', 'best'). Default is 'best'.
- **minimum** (float) - Optional - If the word is not in the list or has a frequency lower than this, return this value instead.

### Request Example
word_frequency('cafe', 'en')

### Response
#### Success Response (200)
- **frequency** (float) - The decimal frequency of the word.

#### Response Example
1.23e-05
```

--------------------------------

### Language-Specific Information Retrieval

Source: https://context7.com/rspeer/wordfreq/llms.txt

Demonstrates how to retrieve language-specific settings such as dotless i handling for Turkish or transliteration rules for Serbian.

```python
from wordfreq import get_language_info

# Turkish has special i handling
info = get_language_info('tr')
print(f"Turkish dotless i: {info['dotless_i']}")
print(f"Turkish diacritics: {info['diacritics_under']}")

# Serbian uses transliteration (Cyrillic to Latin)
info = get_language_info('sr')
print(f"Serbian transliteration: {info['transliteration']}")
```

--------------------------------

### simple_tokenize

Source: https://context7.com/rspeer/wordfreq/llms.txt

Performs basic Unicode-aware tokenization without language-specific rules.

```APIDOC
## POST /simple_tokenize

### Description
Performs straightforward Unicode-aware tokenization. Useful for consistent tokenization across all languages.

### Method
POST

### Endpoint
/simple_tokenize

### Parameters
#### Request Body
- **text** (string) - Required - The text to tokenize.
- **include_punctuation** (boolean) - Optional - Whether to include punctuation in the output.

### Request Example
{
  "text": "Hello, world!",
  "include_punctuation": true
}

### Response
#### Success Response (200)
- **tokens** (array) - List of tokens.

#### Response Example
{
  "tokens": ["hello", ",", "world", "!"]
}
```

--------------------------------

### List Supported Languages

Source: https://context7.com/rspeer/wordfreq/llms.txt

Retrieves a dictionary of available language codes and their corresponding data files. This allows users to verify support for specific languages and wordlist sizes.

```python
from wordfreq import available_languages

languages = available_languages()
print(f"Total languages: {len(languages)}")

large_languages = available_languages('large')
print(f"Languages with large lists: {len(large_languages)}")

if 'de' in available_languages():
    print("German is supported!")
```

--------------------------------

### Lookup Word Frequency as Decimal in Python

Source: https://context7.com/rspeer/wordfreq/llms.txt

Retrieves the probability of a word appearing in a given language. Supports specifying wordlist size and minimum frequency thresholds for unknown words.

```python
from wordfreq import word_frequency

# Basic word frequency lookup
freq = word_frequency('cafe', 'en')
print(freq)

# Use minimum parameter to avoid zero for unknown words
freq = word_frequency('esquivalience', 'en', minimum=1e-8)
print(freq)

# Specify wordlist size
freq_large = word_frequency('infrequency', 'en', wordlist='large')
print(freq_large)
```

--------------------------------

### Text Preprocessing for Consistency

Source: https://context7.com/rspeer/wordfreq/llms.txt

Applies language-specific normalization, case folding, and transliteration to clean text before further analysis.

```python
from wordfreq.preprocess import preprocess_text

# German ß handling
text = preprocess_text('groß', 'de')

# Turkish case folding with dotted i
text = preprocess_text('HAKKINDA İSTANBUL', 'tr')

# Serbian Cyrillic to Latin transliteration
text = preprocess_text('схваташ', 'sr')
```

--------------------------------

### Frequency Scale Conversions

Source: https://context7.com/rspeer/wordfreq/llms.txt

Utilities to convert between raw frequencies, centibels, and the Zipf scale.

```APIDOC
## POST /convert_frequency

### Description
Converts between different frequency representations: raw frequencies, centibels (cB), and Zipf scale.

### Method
POST

### Endpoint
/convert_frequency

### Parameters
#### Request Body
- **value** (float) - Required - The numeric value to convert.
- **from_type** (string) - Required - The source scale ('cB', 'zipf', 'freq').
- **to_type** (string) - Required - The target scale ('cB', 'zipf', 'freq').

### Request Example
{
  "value": 6,
  "from_type": "zipf",
  "to_type": "freq"
}

### Response
#### Success Response (200)
- **result** (float) - The converted frequency value.

#### Response Example
{
  "result": 0.001
}
```

--------------------------------

### Cite wordfreq in BibTeX

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Standard BibTeX entry for citing the wordfreq library in academic research. This ensures proper attribution to the author and the Zenodo DOI.

```BibTeX
@software{robyn_speer_2022_7199437,
  author       = {Robyn Speer},
  title        = {rspeer/wordfreq: v3.0},
  month        = sep,
  year         = 2022,
  publisher    = {Zenodo},
  version      = {v3.0.2},
  doi          = {10.5281/zenodo.7199437},
  url          = {https://doi.org/10.5281/zenodo.7199437}
}
```

--------------------------------

### Tokenize Text with Wordfreq

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Splits text into tokens using language-specific rules consistent with how wordfreq data was originally counted. It handles special cases like gender-neutral symbols and multi-token queries.

```python
from wordfreq import tokenize, zipf_frequency
print(tokenize('l@s niñ@s', 'es'))
print(zipf_frequency('l@s', 'es'))
print(zipf_frequency('New York', 'en'))
print(zipf_frequency('北京地铁', 'zh'))
```

--------------------------------

### Retrieve Language Processing Info

Source: https://context7.com/rspeer/wordfreq/llms.txt

Fetches metadata about how a specific language is processed, including tokenizer types, script definitions, and normalization rules.

```python
from wordfreq.language_info import get_language_info

info = get_language_info('en')
print(info)

info_zh = get_language_info('zh')
print(f"Chinese tokenizer: {info_zh['tokenizer']}")
```

--------------------------------

### Retrieve Top Frequency Words

Source: https://context7.com/rspeer/wordfreq/llms.txt

Demonstrates how to fetch the most frequent words for a given language using the top_n_list function. Supports filtering by ASCII characters and selecting specific wordlist sizes.

```python
from wordfreq import top_n_list

top_words = top_n_list('es', 10)
print(top_words)

ascii_words = top_n_list('en', 100, ascii_only=True)
print(ascii_words[:10])

top_words_large = top_n_list('en', 1000, wordlist='large')
print(len(top_words_large))
```

--------------------------------

### Frequency Scale Conversions

Source: https://context7.com/rspeer/wordfreq/llms.txt

Converts between raw frequencies, centibels (internal format), and the Zipf scale for standardized frequency representation.

```python
from wordfreq import cB_to_freq, zipf_to_freq, freq_to_zipf

# Centibels to frequency
freq = cB_to_freq(-600)

# Zipf scale to frequency
freq = zipf_to_freq(3)

# Frequency to Zipf scale
zipf = freq_to_zipf(0.001)
```

--------------------------------

### Tokenize Text with Language-Specific Rules in Python

Source: https://context7.com/rspeer/wordfreq/llms.txt

Splits text into tokens while handling language-specific requirements like case-folding, punctuation, and script-specific segmentation (e.g., CJK).

```python
from wordfreq import tokenize

# Basic English tokenization
tokens = tokenize("I don't split at apostrophes, you see.", 'en')

# Case folding for German
tokens = tokenize("WEISS", 'de')

# Chinese tokenization
tokens = tokenize("中国文字", 'zh')
```

--------------------------------

### Iterate Through Wordlists

Source: https://context7.com/rspeer/wordfreq/llms.txt

Provides an iterator to process words in descending order of frequency. This approach is memory-efficient as it does not require loading the entire wordlist into memory.

```python
from wordfreq import iter_wordlist

for word in iter_wordlist('en'):
    print(word)
    break

total = sum(1 for _ in iter_wordlist('de'))
print(f"Total German words: {total}")
```

--------------------------------

### Retrieve Top N Most Common Words in Python

Source: https://context7.com/rspeer/wordfreq/llms.txt

Fetches a list of the N most frequent words for a specified language, sorted by frequency.

```python
from wordfreq import top_n_list

# Get top 10 English words
top_words = top_n_list('en', 10)
print(top_words)
```

--------------------------------

### Simple Unicode Tokenization

Source: https://context7.com/rspeer/wordfreq/llms.txt

Performs basic Unicode-aware tokenization without language-specific logic. It is useful for general-purpose segmentation including punctuation and emoji handling.

```python
from wordfreq.tokens import simple_tokenize

# Basic tokenization using Unicode word boundaries
tokens = simple_tokenize("Hello, world! How are you?")

# Include punctuation
tokens = simple_tokenize("Hello, world!", include_punctuation=True)

# Handles emoji correctly
tokens = simple_tokenize("Test 😀 emoji 🎉")
```

--------------------------------

### Calculate Word Frequency on Zipf Scale in Python

Source: https://context7.com/rspeer/wordfreq/llms.txt

Returns a logarithmic Zipf scale value for words, which is more human-readable than decimal probabilities. Values typically range from 0 to 8.

```python
from wordfreq import zipf_frequency

# Common words have high Zipf values
print(zipf_frequency('the', 'en'))

# Rare words have low Zipf values
print(zipf_frequency('zipf', 'en'))

# Set minimum Zipf value for unknown words
print(zipf_frequency('nonexistentword', 'en', minimum=1.0))
```

--------------------------------

### Lossy Tokenization with Normalization

Source: https://context7.com/rspeer/wordfreq/llms.txt

Tokenizes text with aggressive normalization, including Chinese simplification and quote standardization, which is ideal for consistent frequency lookups.

```python
from wordfreq import tokenize, lossy_tokenize, word_frequency

# Standard tokenize preserves original characters
tokens = tokenize("let's go", 'en')

# lossy_tokenize converts curly quotes to straight quotes
tokens = lossy_tokenize("let's go", 'en')

# For Chinese, lossy_tokenize simplifies Traditional to Simplified
traditional = "臺灣"
tokens_lossy = lossy_tokenize(traditional, 'zh')

# Use lossy tokenization for consistent frequency lookups
freq1 = word_frequency("let's", 'en')
freq2 = word_frequency("let's", 'en')
print(f"Same frequency: {freq1 == freq2}")
```

--------------------------------

### preprocess_text

Source: https://context7.com/rspeer/wordfreq/llms.txt

Applies language-specific text normalization steps such as case folding, transliteration, and diacritic handling.

```APIDOC
## POST /preprocess_text

### Description
Applies language-specific preprocessing including Unicode normalization, case folding, and transliteration.

### Method
POST

### Endpoint
/preprocess_text

### Parameters
#### Request Body
- **text** (string) - Required - The text to process.
- **lang** (string) - Required - The language code.

### Request Example
{
  "text": "HAKKINDA İSTANBUL",
  "lang": "tr"
}

### Response
#### Success Response (200)
- **text** (string) - The processed text.

#### Response Example
{
  "text": "hakkında istanbul"
}
```

--------------------------------

### Lookup Zipf frequency in Python

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Uses the zipf_frequency function to return word frequency on a logarithmic Zipf scale. This is useful for human-readable frequency comparisons where values typically range from 0 to 8.

```python
from wordfreq import zipf_frequency
# Get Zipf frequency for 'the' in English
zipf = zipf_frequency('the', 'en')
print(zipf)
# Get Zipf frequency using a specific wordlist
zipf_small = zipf_frequency('zipf', 'en', wordlist='small')
print(zipf_small)
```

--------------------------------

### Generate Random Words

Source: https://context7.com/rspeer/wordfreq/llms.txt

Generates random sequences of words based on frequency data. Useful for creating memorable passwords or text samples with configurable entropy levels.

```python
from wordfreq import random_words, random_ascii_words

password = random_words(lang='en', nwords=5, bits_per_word=12)
print(password)

ascii_password = random_ascii_words(lang='en', nwords=5, bits_per_word=12)
print(f"ASCII password: {ascii_password}")
```

--------------------------------

### Lookup word frequency in Python

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Uses the word_frequency function to retrieve the decimal frequency of a word within a specified language. It supports different wordlist sizes such as 'small', 'large', or 'best'.

```python
from wordfreq import word_frequency
# Look up frequency of 'cafe' in English
freq = word_frequency('cafe', 'en')
print(freq)
# Look up frequency of 'café' in French
freq_fr = word_frequency('café', 'fr')
print(freq_fr)
```

--------------------------------

### lossy_tokenize

Source: https://context7.com/rspeer/wordfreq/llms.txt

Tokenizes text with aggressive normalization, including quote standardization and character simplification, optimized for frequency lookups.

```APIDOC
## POST /lossy_tokenize

### Description
Tokenizes input text with lossy normalization. It handles curly-to-straight quote conversion and simplifies Traditional Chinese to Simplified Chinese for consistent lookup.

### Method
POST

### Endpoint
/lossy_tokenize

### Parameters
#### Request Body
- **text** (string) - Required - The text to tokenize.
- **lang** (string) - Required - The language code (e.g., 'en', 'zh').

### Request Example
{
  "text": "let's go",
  "lang": "en"
}

### Response
#### Success Response (200)
- **tokens** (array) - List of normalized tokens.

#### Response Example
{
  "tokens": ["let's", "go"]
}
```

--------------------------------

### Retrieve Top N Words in Wordfreq

Source: https://github.com/rspeer/wordfreq/blob/master/README.md

Retrieves the most common n words for a specified language in descending frequency order. This function helps identify the most frequent vocabulary for a given language code.

```python
from wordfreq import top_n_list
print(top_n_list('en', 10))
print(top_n_list('es', 10))
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.