WeTextProcessing

https://github.com/wenet-e2e/wetextprocessing
Admin
WeTextProcessing is a production-ready text processing toolkit that performs text normalization and...
Tokens:10,742
Snippets:97
Trust Score:7.7
License:Apache-2.0
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install WeTextProcessing Package

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/README.md

Install the WeTextProcessing library using pip. This is the first step before using the toolkit.

```bash
pip install WeTextProcessing
```

--------------------------------

### Use Command Line Interface

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Examples of using the command-line tools for quick text normalization and inverse normalization across different languages.

```bash
# Text Normalization (TN)
# Chinese
wetn --text "2.5平方电线"
# Output: 二点五平方电线

# English
wetn --text "2010-03-21" --language en
# Output: the twenty first of march twenty ten

# Japanese
wetn --text "123円" --language ja

# Inverse Text Normalization (ITN)
```

--------------------------------

### Command-Line Usage for TN and ITN

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/README.md

Use the command-line interface to perform text normalization (wetn) and inverse text normalization (weitn) with example inputs.

```bash
wetn --text "2.5平方电线"
```

```bash
weitn --text "二点五平方电线"
```

--------------------------------

### Command-line Text Normalization (Chinese)

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Use the `weitn` command-line tool for Chinese text normalization. This example normalizes a Chinese string.

```bash
weitn --text "二点五平方电线"
```

--------------------------------

### CMake project configuration

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/patch/openfst/src/CMakeLists.txt

CMake directives for managing include directories, installation paths, and conditional subdirectory builds.

```cmake
include_directories(./include/)
install(DIRECTORY include/ DESTINATION include/
        FILES_MATCHING PATTERN "*.h")

add_subdirectory(lib)

if(HAVE_SCRIPT)
  add_subdirectory(script)
endif(HAVE_SCRIPT)

if(HAVE_BIN)
  add_subdirectory(bin)
endif(HAVE_BIN)

add_subdirectory(extensions)

if(BUILD_TESTING)
  enable_testing()
  add_subdirectory(test)
endif(BUILD_TESTING)
```

--------------------------------

### Configure Chinese Inverse Normalizer

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Examples of initializing and configuring the InverseNormalizer for Chinese text, including options for cache management and number conversion rules.

```python
inverse_normalizer = InverseNormalizer(
    cache_dir=None,                   # Custom cache directory
    overwrite_cache=False,            # Set True to rebuild FST graphs
    remove_interjections=True,        # Remove interjections
    enable_standalone_number=True,    # Convert standalone numbers (一百 => 100)
    enable_0_to_9=False,              # Convert single digits 0-9
    enable_million=False              # Convert large numbers to pure digits
)
```

```python
itn_default = InverseNormalizer(enable_standalone_number=True)
text = "幸运一百"
result = itn_default.normalize(text)
# Output: "幸运100"
```

```python
itn_no_standalone = InverseNormalizer(
    enable_standalone_number=False,
    overwrite_cache=True
)
text = "幸运一百"
result = itn_no_standalone.normalize(text)
# Output: "幸运一百"
```

```python
itn_with_digits = InverseNormalizer(
    enable_0_to_9=True,
    overwrite_cache=True
)
text = "零和九"
result = itn_with_digits.normalize(text)
# Output: "0和9"
```

```python
itn_default = InverseNormalizer(enable_0_to_9=False)
text = "零和九"
result = itn_default.normalize(text)
# Output: "零和九"
```

--------------------------------

### Chinese Text Normalization Examples

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Demonstrates various Chinese text normalization tasks including numbers, dates, currencies, measurements, times, phone numbers, full-width to half-width conversion, percentages, and fractions.

```python
from tn.chinese.normalizer import Normalizer

# Initialize the normalizer with default settings
normalizer = Normalizer()

# Basic number normalization
text = "共465篇，约315万字"
result = normalizer.normalize(text)
# Output: "共四百六十五篇,约三百一十五万字"

# Date normalization
text = "2002-01-28"
result = normalizer.normalize(text)
# Output: "二零零二年一月二十八日"

# Currency normalization
text = "价格是￥13.5"
result = normalizer.normalize(text)
# Output: "价格是十三点五元"

# Measurement normalization
text = "重达25kg"
result = normalizer.normalize(text)
# Output: "重达二十五千克"

# Time normalization
text = "8:00 a.m.准时开会"
result = normalizer.normalize(text)
# Output: "上午八点准时开会"

# Phone numbers
text = "可以拨打12306来咨询"
result = normalizer.normalize(text)
# Output: "可以拨打幺二三零六来咨询"

# Full-width to half-width conversion
text = "苹果宣布发布新ＩＰＨＯＮＥ"
result = normalizer.normalize(text)
# Output: "苹果宣布发布新IPHONE"

# Percentage normalization
text = "同比增长6.3%"
result = normalizer.normalize(text)
# Output: "同比增长百分之六点三"

# Fraction normalization
text = "总量的1/5以上"
result = normalizer.normalize(text)
# Output: "总量的五分之一以上"
```

--------------------------------

### Chinese Inverse Text Normalization Examples

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Demonstrates Chinese inverse text normalization for numbers, dates, currencies, measurements, times, percentages, fractions, and math expressions.

```python
from itn.chinese.inverse_normalizer import InverseNormalizer

# Initialize the inverse normalizer
inverse_normalizer = InverseNormalizer()

# Number denormalization
text = "共四百六十五篇,约三百一十五万字"
result = inverse_normalizer.normalize(text)
# Output: "共465篇,约315万字"

# Date denormalization
text = "二零零二年一月二十八日"
result = inverse_normalizer.normalize(text)
# Output: "2002/01/28"

# Currency denormalization
text = "价格是十三点五元"
result = inverse_normalizer.normalize(text)
# Output: "价格是¥13.5"

# Measurement denormalization
text = "重达二十五千克"
result = inverse_normalizer.normalize(text)
# Output: "重达25kg"

# Time denormalization
text = "我是五点零二分开始的"
result = inverse_normalizer.normalize(text)
# Output: "我是5:02开始的"

# Percentage denormalization
text = "同比增长百分之六点三"
result = inverse_normalizer.normalize(text)
# Output: "同比增长6.3%"

# Fraction denormalization
text = "总量的五分之一以上"
result = inverse_normalizer.normalize(text)
# Output: "总量的1/5以上"

# Math expression denormalization
text = "计算负二的绝对值是二"
result = inverse_normalizer.normalize(text)
# Output: "计算-2的绝对值是2"
```

--------------------------------

### Enable Testing and Link Libraries

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/test/CMakeLists.txt

Enables testing infrastructure and links Google Test and Google Mock libraries. This is a common setup for C++ projects using Google Test.

```cmake
enable_testing()
link_libraries(gtest_main gmock)
include(GoogleTest)
```

--------------------------------

### Define FST Module Helper Function

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/patch/openfst/src/extensions/special/CMakeLists.txt

A helper function to simplify the creation and installation of FST modules.

```cmake
function (add_module _name)
  add_library(${ARGV})
  if (TARGET ${_name})
    target_link_libraries(${_name} fst)
    set_target_properties(${_name}
      PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS true
      FOLDER special/modules
    )
  endif()

  install(TARGETS ${_name} LIBRARY DESTINATION lib/fst)
endfunction()

add_module(phi-fst MODULE phi-fst.cc)
add_module(rho-fst MODULE rho-fst.cc)
add_module(sigma-fst MODULE sigma-fst.cc)
```

--------------------------------

### Python Module Text Normalization

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Run Text Normalization (TN) as a Python module. This example shows normalizing Chinese, English, and Japanese text using the `tn` module.

```bash
python -m tn --text "2.5平方电线" --language zh
```

```bash
python -m tn --text "2024-05-06" --language en
```

```bash
python -m tn --text "123円" --language ja
```

--------------------------------

### C++ API Text Normalization

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Integrate Text Normalization (TN) into native C++ applications using the WeTextProcessor API. This example shows creating a processor, tagging text, and verbalizing the result.

```cpp
#include "processor/wetext_processor.h"

int main() {
    // Create processor with tagger and verbalizer FST paths
    wetext::Processor processor("zh_tn_tagger.fst", "zh_tn_verbalizer.fst");

    // Tag input text
    std::string input = "2.5平方电线";
    std::string tagged = processor.Tag(input);

    // Verbalize tagged text
    std::string result = processor.Verbalize(tagged);

    // Or use combined Normalize
    std::string normalized = processor.Normalize(input);

    return 0;
}
```

--------------------------------

### Python Module Inverse Text Normalization

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Run Inverse Text Normalization (ITN) as a Python module. This example demonstrates normalizing Chinese and Japanese text using the `itn` module.

```bash
python -m itn --text "二点五平方电线" --language zh
```

```bash
python -m itn --text "百二十三" --language ja
```

--------------------------------

### Python TN Pipeline: Tagging and Verbalization

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Utilize the Python module interface for Text Normalization (TN) to perform tagging and verbalization separately or combined. This example uses the Chinese normalizer.

```python
from tn.chinese.normalizer import Normalizer

normalizer = Normalizer()

# Step 1: Tag the input text
text = "价格是￥13.5"
tagged = normalizer.tag(text)
# Output: 'zh_normalizer { money { integer: "13" dot: "点" decimal: "5" currency: "元" } } zh_normalizer { char { value: "" } }'

# Step 2: Verbalize the tagged text
result = normalizer.verbalize(tagged)
# Output: "价格是十三点五元"

# Combined normalize (tag + verbalize)
result = normalizer.normalize(text)
# Output: "价格是十三点五元"
```

--------------------------------

### Python ITN Pipeline: Tagging and Verbalization

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Use the Python module interface for Inverse Text Normalization (ITN) to tag and verbalize text. This example demonstrates the process for Chinese input.

```python
from itn.chinese.inverse_normalizer import InverseNormalizer

inverse_normalizer = InverseNormalizer()

text = "价格是十三点五元"
tagged = inverse_normalizer.tag(text)
result = inverse_normalizer.verbalize(tagged)
# Output: "价格是¥13.5"
```

--------------------------------

### Create and Use WetextProcessor in C

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Demonstrates how to create a WetextProcessor handle, normalize text, and clean up resources using the C API. Ensure the FST files are accessible.

```c
#include "processor/wetext_processor_c_api.h"

int main() {
    // Create processor
    WetextProcessorHandle handle = wetext_create_processor(
        "zh_tn_tagger.fst",
        "zh_tn_verbalizer.fst"
    );

    if (handle == NULL) {
        return 1;
    }

    // Normalize text
    const char* input = "2.5平方电线";
    const char* result = wetext_normalize(handle, input);

    // Use result...
    printf("%s\n", result);

    // Free the result string
    wetext_free_string(result);

    // Destroy processor
    wetext_destroy_processor(handle);

    return 0;
}
```

--------------------------------

### Command-line All TN Options

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Demonstrates the use of all available options for Text Normalization (TN) via the command line. This includes language, cache directory, cache overwriting, and various rule enablers/disablers.

```bash
weitn --text "text" \
    --language zh \
    --cache_dir /path \
    --overwrite_cache \
    --remove_interjections True \
    --remove_erhua True \
    --traditional_to_simple True \
    --remove_puncts False \
    --full_to_half True \
    --tag_oov False
```

--------------------------------

### Command-line All ITN Options

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Demonstrates the use of all available options for Inverse Text Normalization (ITN) via the command line. This includes language, cache directory, cache overwriting, and specific number-related rule enablers/disablers.

```bash
weitn --text "text" \
    --language zh \
    --cache_dir /path \
    --overwrite_cache \
    --enable_standalone_number True \
    --enable_0_to_9 False \
    --enable_million False
```

--------------------------------

### C++ Runtime Deployment with Custom Rules

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/README.md

Deploy custom rules using the C++ runtime by building the processor and specifying the `cache_dir` for tagger and verbalizer FST files.

```bash
cmake -B build -S runtime -DCMAKE_BUILD_TYPE=Release
cmake --build build
# tn usage
cache_dir=PATH_TO_GIT_CLONED_WETEXTPROCESSING/tn
./build/processor_main --tagger $cache_dir/zh_tn_tagger.fst --verbalizer $cache_dir/zh_tn_verbalizer.fst --text "2.5平方电线"
# itn usage
cache_dir=PATH_TO_GIT_CLONED_WETEXTPROCESSING/itn
./build/processor_main --tagger $cache_dir/zh_itn_tagger.fst --verbalizer $cache_dir/zh_itn_verbalizer.fst --text "二点五平方电线"
```

--------------------------------

### Python Usage: TN and ITN with Default and Custom Parameters

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/README.md

Demonstrates Python usage for Chinese and English text normalization (TN) and inverse text normalization (ITN). Shows how to initialize models with different parameters and whether to overwrite the cache for rule recompilation.

```python
from itn.chinese.inverse_normalizer import InverseNormalizer
from tn.chinese.normalizer import Normalizer as ZhNormalizer
from tn.english.normalizer import Normalizer as EnNormalizer

# NOTE(xcsong): 和默认参数不一致时，必须重新构图，要重新构图请务必指定 `overwrite_cache=True`
#               When the parameters differ from the defaults, it is mandatory to re-compose. To re-compose, please ensure you specify `overwrite_cache=True`.

zh_tn_text = "你好 WeTextProcessing 1.0，船新版本儿，船新体验儿，简直666，9和10"
zh_itn_text = "你好 WeTextProcessing 一点零，船新版本儿，船新体验儿，简直六六六，九和六"
en_tn_text = "Hello WeTextProcessing 1.0, life is short, just use wetext, 666, 9 and 10"
zh_tn_model = ZhNormalizer(remove_erhua=True, overwrite_cache=True)
zh_itn_model = InverseNormalizer(enable_0_to_9=False, overwrite_cache=True)
en_tn_model = EnNormalizer(overwrite_cache=True)
print("中文 TN (去除儿化音，重新在线构图):\n\t{} => {}".format(zh_tn_text, zh_tn_model.normalize(zh_tn_text)))
print("中文ITN (小于10的单独数字不转换，重新在线构图):\n\t{} => {}".format(zh_itn_text, zh_itn_model.normalize(zh_itn_text)))
print("英文 TN (暂时还没有可控的选项，后面会加...):\n\t{} => \n".format(en_tn_text, en_tn_model.normalize(en_tn_text)))

zh_tn_model = ZhNormalizer(overwrite_cache=False)
zh_itn_model = InverseNormalizer(overwrite_cache=False)
en_tn_model = EnNormalizer(overwrite_cache=False)
print("中文 TN (复用之前编译好的图):\n\t{} => {}".format(zh_tn_text, zh_tn_model.normalize(zh_tn_text)))
print("中文ITN (复用之前编译好的图):\n\t{} => {}".format(zh_itn_text, zh_itn_model.normalize(zh_itn_text)))
print("英文 TN (复用之前编译好的图):\n\t{} => \n".format(en_tn_text, en_tn_model.normalize(en_tn_text)))

zh_tn_model = ZhNormalizer(remove_erhua=False, overwrite_cache=True)
zh_itn_model = InverseNormalizer(enable_0_to_9=True, overwrite_cache=True)
print("中文 TN (不去除儿化音，重新在线构图):\n\t{} => {}".format(zh_tn_text, zh_tn_model.normalize(zh_tn_text)))
print("中文ITN (小于10的单独数字也进行转换，重新在线构图):\n\t{} => \n".format(zh_itn_text, zh_itn_model.normalize(zh_itn_text)))
```

--------------------------------

### Python Deployment with Custom Rules

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/README.md

Load custom normalized rules in Python by specifying the `cache_dir` pointing to your cloned WeTextProcessing repository.

```python
# tn usage
>>> from tn.chinese.normalizer import Normalizer
>>> normalizer = Normalizer(cache_dir="PATH_TO_GIT_CLONED_WETEXTPROCESSING/tn")
>>> normalizer.normalize("2.5平方电线")
# itn usage
>>> from itn.chinese.inverse_normalizer import InverseNormalizer
>>> invnormalizer = InverseNormalizer(cache_dir="PATH_TO_GIT_CLONED_WETEXTPROCESSING/itn")
>>> invnormalizer.normalize("二点五平方电线")
```

--------------------------------

### CMake Project Configuration

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/CMakeLists.txt

Initializes the project, sets C++14 standard, and configures platform-specific compiler flags for Linux and Windows.

```cmake
cmake_minimum_required(VERSION 3.14 FATAL_ERROR)

project(wetextprocessing VERSION 0.1)
set(CMAKE_CXX_STANDARD 14)

set(CMAKE_VERBOSE_MAKEFILE OFF)
option(BUILD_TESTING "whether to build unit test" OFF)

include(FetchContent)
set(FETCHCONTENT_QUIET OFF)
get_filename_component(fc_base "fc_base-${CMAKE_CXX_COMPILER_ID}" REALPATH BASE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
set(FETCHCONTENT_BASE_DIR ${fc_base})
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake)

if(NOT MSVC)
  # Keep the same with openfst, -fPIC or -fpic
  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14 -pthread -fPIC")
else()
  # For windows, please use unicode(3 bytes per chinese char) instead of gbk(2 bytes per chinese char).
  # https://github.com/wenet-e2e/wenet/issues/882#issuecomment-1101246299
  set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)
  add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/utf-8>")
endif()

if(${CMAKE_SYSTEM_NAME} STREQUAL "Darwin")
  set(CMAKE_MACOSX_RPATH 1)
endif()

include(openfst)
include_directories(${PROJECT_SOURCE_DIR})

add_subdirectory(utils)
add_subdirectory(processor)
add_subdirectory(bin)

if(BUILD_TESTING)
  include(gtest)
  add_subdirectory(test)
endif()
```

--------------------------------

### C++ Runtime Download FST Files

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Download pre-built FST (Finite State Transducer) files required for the C++ runtime. These files contain the models for text normalization and inverse text normalization.

```bash
wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_tn_tagger.fst
wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_tn_verbalizer.fst
wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_itn_tagger.fst
wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_itn_verbalizer.fst
```

--------------------------------

### Command-line Processing from File

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Process text normalization for Chinese language from an input file using the `wetn` command-line tool. Ensure the input file contains the text to be processed.

```bash
wetn --file input.txt --language zh
```

```bash
weitn --file input.txt --language zh
```

--------------------------------

### Advanced Usage: Rebuilding Rules and Deploying

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/README.md

Clone the repository to modify rules and rebuild them using `overwrite_cache=True`. This allows for custom rule deployment via Python or C++ runtime.

```bash
git clone https://github.com/wenet-e2e/WeTextProcessing.git
cd WeTextProcessing
pip install -r requirements.txt
pre-commit install # for clean and tidy code
# `overwrite_cache` will rebuild all rules according to
#   your modifications on tn/chinese/rules/xx.py (itn/chinese/rules/xx.py).
#   After rebuild, you can find new far files at `$PWD/tn` and `$PWD/itn`.
python -m tn --text "2.5平方电线" --overwrite_cache
python -m itn --text "二点五平方电线" --overwrite_cache
```

--------------------------------

### Build WeTextProcessing Runtime (Linux/macOS)

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/README.md

Use these CMake commands to build the WeTextProcessing project in Release mode on Linux or macOS.

```bash
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
```

--------------------------------

### Command-line Rebuild Cache with Options

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Rebuild the cache for text normalization with custom options. Use `--overwrite_cache` to force cache rebuilding and `--remove_erhua True` or `--enable_0_to_9 True` to apply specific normalization rules.

```bash
wetn --text "2.5平方电线" --overwrite_cache --remove_erhua True
```

```bash
weitn --text "二点五平方电线" --overwrite_cache --enable_0_to_9 True
```

--------------------------------

### Use WeTextProcessing for Text Normalization (TN)

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/README.md

Download the necessary FST files and run the processor_main executable with the TN tagger and verbalizer for processing text containing numbers like '2.5平方电线'.

```bash
# tn usage
$ wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_tn_tagger.fst
$ wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_tn_verbalizer.fst
$ ./build/processor_main --tagger zh_tn_tagger.fst --verbalizer zh_tn_verbalizer.fst --text "2.5平方电线"
```

--------------------------------

### Command-line Text Normalization (Japanese)

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Use the `weitn` command-line tool for Japanese text normalization. Specify the language code 'ja' for Japanese input.

```bash
weitn --text "百二十三" --language ja
```

--------------------------------

### Configure wetext_processor static library

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/processor/CMakeLists.txt

Defines the static library target and manages platform-specific dependencies for linking.

```cmake
add_library(wetext_processor STATIC
  wetext_processor.cc
  wetext_token_parser.cc
)
if(ANDROID)
  target_link_libraries(wetext_processor PUBLIC fst wetext_utils)
else()
  if(MSVC)
    target_link_libraries(wetext_processor PUBLIC fst wetext_utils)
  else()
    target_link_libraries(wetext_processor PUBLIC dl fst wetext_utils)
  endif()
endif()
```

--------------------------------

### Build WeTextProcessing Runtime (Windows)

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/README.md

Use these CMake commands with the Visual Studio generator to build the WeTextProcessing project on Windows. Note the flags for debugging information and disabling shared libraries.

```bash
cmake -DCMAKE_BUILD_TYPE=Release -B build -G "Visual Studio 17 2022" -DBUILD_SHARED_LIBS=0 -DCMAKE_CXX_FLAGS="/ZI"
cmake --build build
```

--------------------------------

### Define Executable and Link Libraries

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/test/CMakeLists.txt

Defines a C++ executable target named 'string_test' and links it with the 'wetext_utils' library. It then discovers tests for this executable.

```cmake
add_executable(string_test string_test.cc)
target_link_libraries(string_test PUBLIC wetext_utils)
gtest_discover_tests(string_test)
```

--------------------------------

### C++ Runtime Process from File

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Process text normalization from an input file using the C++ runtime. This command requires the tagger, verbalizer FST files, and the path to the input file.

```bash
./build/processor_main \
    --tagger zh_tn_tagger.fst \
    --verbalizer zh_tn_verbalizer.fst \
    --file input.txt
```

--------------------------------

### CMake Build Configuration for wetextprocessing

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/android/app/src/main/cpp/CMakeLists.txt

Defines project settings, includes necessary modules, and adds subdirectories for the build process.

```cmake
cmake_minimum_required(VERSION 3.4.1)
set(TARGET wetextprocessing)
project(${TARGET} CXX)
set(CMAKE_CXX_STANDARD 14)
include(ExternalProject)

set(CMAKE_VERBOSE_MAKEFILE on)
set(build_DIR ${CMAKE_SOURCE_DIR}/../../../build)
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake)
string(REPLACE "-Wl,--exclude-libs,libgcc_real.a" "" CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS}")

include(openfst)

include_directories(
  ${CMAKE_SOURCE_DIR}
)

add_subdirectory(utils)
add_subdirectory(processor)

link_libraries(wetext_processor android)
add_library(${TARGET} SHARED wetextprocessing.cc)
```

--------------------------------

### Compilation command for symbol-table.cc

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/patch/openfst/src/CMakeLists.txt

Standard compiler flags and dependency tracking for building the symbol table object file.

```bash
-DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT symbol-table.lo -MD -MP -MF .deps/symbol-table.Tpo -c symbol-table.cc  -fno-common -DPIC -o .libs/symbol-table.o
```

--------------------------------

### Configure wetext_processor_c shared library

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/processor/CMakeLists.txt

Defines the shared library target for the C API and links it against the core wetext_processor library.

```cmake
# ----------------------------------------------------------------------------
# C API shared library (wetext_processor_c)
# ----------------------------------------------------------------------------
add_library(wetext_processor_c SHARED
  wetext_processor_c_api.cc
)

target_link_libraries(wetext_processor_c PUBLIC wetext_processor)
```

--------------------------------

### Chinese ITN Configuration Options

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Configure Chinese inverse text normalization behavior with various options.

```python
from itn.chinese.inverse_normalizer import InverseNormalizer

```

--------------------------------

### Use WeTextProcessing for Inverse Text Normalization (ITN)

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/README.md

Download the necessary FST files and run the processor_main executable with the ITN tagger and verbalizer for processing text containing numbers written out like '二点五平方电线'.

```bash
# itn usage
$ wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_itn_tagger.fst
$ wget https://github.com/wenet-e2e/WeTextProcessing/releases/download/WeTextProcessing/zh_itn_verbalizer.fst
$ ./build/processor_main --tagger zh_itn_tagger.fst --verbalizer zh_itn_verbalizer.fst --text "二点五平方电线"
```

--------------------------------

### Command-line Custom Cache Directory

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Specify a custom directory for the cache using the `--cache_dir` option. This is useful for managing cache files in specific locations.

```bash
weitn --text "example" --cache_dir /path/to/cache
```

```bash
weitn --text "example" --cache_dir /path/to/cache
```

--------------------------------

### Apply Whitelist Replacements

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Applies predefined replacements from a whitelist to specific Chinese phrases.

```text
三心二意
```

--------------------------------

### Define Executable for Token Parser Test (Non-MSVC)

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/test/CMakeLists.txt

Defines a C++ executable target for 'token_parser_test' and links it with 'wetext_processor'. This is conditionally compiled only when not using MSVC. It then discovers tests.

```cmake
if(NOT MSVC)
  # token_parser_test uses the macro to access the private members
  add_executable(token_parser_test token_parser_test.cc)
  target_link_libraries(token_parser_test PUBLIC wetext_processor)
  gtest_discover_tests(token_parser_test)
endif()
```

--------------------------------

### Chinese Normalizer Configuration Options

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Configure the Chinese normalizer with options like cache directory, overwriting cache, removing interjections, removing erhua, traditional to simplified conversion, punctuation removal, full-width to half-width conversion, and OOV tagging.

```python
from tn.chinese.normalizer import Normalizer

# Full configuration example
normalizer = Normalizer(
    cache_dir=None,              # Custom cache directory for FST files
    overwrite_cache=False,       # Set True to rebuild FST graphs
    remove_interjections=True,   # Remove interjections like "啊", "呃"
    remove_erhua=True,           # Remove erhua "儿" suffix
    traditional_to_simple=True,  # Convert traditional to simplified Chinese
    remove_puncts=False,         # Remove punctuation marks
    full_to_half=True,           # Convert full-width to half-width characters
    tag_oov=False                # Tag out-of-vocabulary characters
)

# Interjection removal
text = "呃这个呃啊我不知道"
result = normalizer.normalize(text)
# Output: "这个我不知道"

# Erhua removal
text = "我儿子喜欢这地儿"
result = normalizer.normalize(text)
# Output: "我儿子喜欢这地"

# OOV tagging (when tag_oov=True)
normalizer_oov = Normalizer(tag_oov=True)
text = "我们안녕"
result = normalizer_oov.normalize(text)
# Output: "我们<oov>안</oov><oov>녕</oov>"

# Rebuild graphs with new options
normalizer_custom = Normalizer(
    remove_erhua=False,
    overwrite_cache=True  # Must set True when changing options
)
```

--------------------------------

### C++ Runtime Inverse Text Normalization

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Run Inverse Text Normalization (ITN) using the compiled C++ runtime executable. Provide the tagger and verbalizer FST files and the input text for ITN.

```bash
./build/processor_main \
    --tagger zh_itn_tagger.fst \
    --verbalizer zh_itn_verbalizer.fst \
    --text "二点五平方电线"
```

--------------------------------

### C++ Runtime Build

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Build the C++ runtime for WeTextProcessing using CMake. This is a prerequisite for using the C++ executable for text normalization tasks.

```bash
cmake -B build -S runtime -DCMAKE_BUILD_TYPE=Release
cmake --build build
```

--------------------------------

### Run Inverse Text Normalization Script

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Execute the ITN script with the text to be denormalized. Ensure the script is available in your Python environment.

```bash
$ python inverse_normalize.py --text "text to be denormalized"
```

--------------------------------

### Define Executable for Processor Test

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/test/CMakeLists.txt

Defines a C++ executable target named 'processor_test' and links it with the 'wetext_processor' library. It then discovers tests for this executable.

```cmake
add_executable(processor_test processor_test.cc)
target_link_libraries(processor_test PUBLIC wetext_processor)
gtest_discover_tests(processor_test)
```

--------------------------------

### Python Module Rebuild Cache

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Rebuild the cache for both Text Normalization (TN) and Inverse Text Normalization (ITN) using Python modules. The `--overwrite_cache` flag is used to force cache regeneration.

```bash
python -m tn --text "2.5平方电线" --overwrite_cache
```

```bash
python -m itn --text "二点五平方电线" --overwrite_cache
```

--------------------------------

### Python Custom Cache Directory

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Configure the Python modules for Text Normalization (TN) and Inverse Text Normalization (ITN) to use a custom cache directory. This is useful for deployment or when modifying rules.

```python
from tn.chinese.normalizer import Normalizer
from itn.chinese.inverse_normalizer import InverseNormalizer

# Use custom cache directory
normalizer = Normalizer(cache_dir="/path/to/custom/cache")
inverse_normalizer = InverseNormalizer(cache_dir="/path/to/custom/cache")

# Clone repository and modify rules, then use custom cache
# git clone https://github.com/wenet-e2e/WeTextProcessing.git
# Modify rules in tn/chinese/rules/*.py or itn/chinese/rules/*.py
# Then rebuild with overwrite_cache=True

from tn.chinese.normalizer import Normalizer

normalizer = Normalizer(
    cache_dir="/path/to/WeTextProcessing/tn",
    overwrite_cache=True
)

result = normalizer.normalize("2.5平方电线")
```

--------------------------------

### Configure FST Special Build Targets

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/runtime/patch/openfst/src/extensions/special/CMakeLists.txt

Defines the executable and library targets for FST special extensions, including linking dependencies and setting target properties.

```cmake
file(GLOB HEADER_FILES ../../include/fst/extensions/special/*.h)
message(STATUS "${HEADER_FILES}")

if(HAVE_BIN)
  add_executable(fstspecial-bin
    ../../bin/fstconvert.cc
    ../../bin/fstconvert-main.cc
    phi-fst.cc
    rho-fst.cc
    sigma-fst.cc
  )

  set_target_properties(fstspecial-bin PROPERTIES
    FOLDER special/bin
    OUTPUT_NAME fstspecial
  )

  target_link_libraries(fstspecial-bin
    fstscript
    fst
    ${CMAKE_DL_LIBS}
  )
endif(HAVE_BIN)


add_library(fstspecial
  phi-fst.cc
  rho-fst.cc
  sigma-fst.cc
  ${HEADER_FILES}
)

set_target_properties(fstspecial PROPERTIES
  SOVERSION "${SOVERSION}"
  FOLDER special
)
target_link_libraries(fstspecial
  fst
)

set(FST_SPECIAL_INSTALL_TARGETS fstspecial)
if(HAVE_BIN)
  list(APPEND FST_SPECIAL_INSTALL_TARGETS fstspecial-bin)
endif()

install(TARGETS ${FST_SPECIAL_INSTALL_TARGETS}
  LIBRARY DESTINATION lib
  RUNTIME DESTINATION bin
  ARCHIVE DESTINATION lib
)
```

--------------------------------

### C++ Runtime Text Normalization

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Run Text Normalization (TN) using the compiled C++ runtime executable. Specify the tagger and verbalizer FST files and the input text.

```bash
./build/processor_main \
    --tagger zh_tn_tagger.fst \
    --verbalizer zh_tn_verbalizer.fst \
    --text "2.5平方电线"
```

--------------------------------

### Normalize Japanese Text

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Usage of the Japanese Normalizer to convert written Japanese text into spoken form, including configuration for character width conversion.

```python
from tn.japanese.normalizer import Normalizer

# Initialize the Japanese normalizer
normalizer = Normalizer()

# Basic normalization
text = "2024年5月6日"
result = normalizer.normalize(text)

# Configuration options
normalizer = Normalizer(
    cache_dir=None,
    overwrite_cache=False,
    transliterate=False,           # Enable transliteration
    remove_interjections=False,    # Remove interjections
    remove_puncts=False,           # Remove punctuation
    full_to_half=True,             # Convert full-width to half-width
    tag_oov=False                  # Tag out-of-vocabulary characters
)

# With full-width to half-width conversion
text = "１２３"
result = normalizer.normalize(text)
```

--------------------------------

### Denormalize Percentages

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts percentage expressions in Chinese text to a numerical format with a '%' sign.

```text
同比增长百分之六点三
```

--------------------------------

### Inverse Normalize Japanese Text

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Usage of the Japanese Inverse Normalizer to convert spoken Japanese text back to written form.

```python
from itn.japanese.inverse_normalizer import InverseNormalizer

# Initialize the Japanese inverse normalizer
inverse_normalizer = InverseNormalizer()

# Basic denormalization
text = "百二十三"
result = inverse_normalizer.normalize(text)

# Configuration options
inverse_normalizer = InverseNormalizer(
    cache_dir=None,
    overwrite_cache=False,
    full_to_half=False,               # Convert full-width to half-width
    enable_standalone_number=True,    # Convert standalone numbers
    enable_0_to_9=False,              # Convert single digits 0-9
    enable_million=False              # Convert large numbers to pure digits
)
```

--------------------------------

### Denormalize Fractions

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts Chinese fraction expressions into a more standard numerical format.

```text
总量的五分之一以上
相当于头发丝的十六分之一
```

--------------------------------

### Denormalize Money

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts Chinese currency expressions into a format with currency symbols.

```text
价格是十三点五元
价格是十三点五美元
```

--------------------------------

### Denormalize Numbers (Cardinal)

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts Chinese number words to Arabic numerals. The `enable_standalone_number` option controls whether standalone numbers are converted.

```text
共四百六十五篇约三百一十五万字
共计六点四二万人
幸运一百
```

```text
幸运一百
```

--------------------------------

### Denormalize Measures

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts measurement expressions (weight, temperature, speed) into a standard numerical format with units.

```text
重达二十五千克
最高气温三十八摄氏度
速度是每小时十公里
```

--------------------------------

### Denormalize Dates

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts Chinese date expressions into a YYYY/MM/DD format.

```text
二零零二年一月二十八日
二零零二年一月
```

--------------------------------

### Normalize English Text

Source: https://context7.com/wenet-e2e/wetextprocessing/llms.txt

Usage of the English Normalizer to convert written text into spoken form, covering various formats like dates, currencies, and measurements.

```python
from tn.english.normalizer import Normalizer

# Initialize the English normalizer
normalizer = Normalizer()

# Cardinal number normalization
text = "number 256"
result = normalizer.normalize(text)
# Output: "number two hundred and fifty six"

# Ordinal number normalization
text = "this is 12th game"
result = normalizer.normalize(text)
# Output: "this is twelfth game"

# Date normalization
text = "2024-05-06"
result = normalizer.normalize(text)
# Output: "the sixth of may twenty twenty four"

# Time normalization
text = "let's eat at 03:43 p.m."
result = normalizer.normalize(text)
# Output: "let's eat at three forty three PM"

# Currency normalization
text = "give me $12.345 please"
result = normalizer.normalize(text)
# Output: "give me twelve point three four five dollars please"

# Measurement normalization
text = "run 10 km"
result = normalizer.normalize(text)
# Output: "run ten kilometers"

# Telephone number normalization
text = "call 123-123-5678-1"
result = normalizer.normalize(text)
# Output: "call one two three, one two three, five six seven eight, one"

# Fraction normalization
text = "3/4 people"
result = normalizer.normalize(text)
# Output: "three quarters people"

# Range normalization
text = "children of 3-4 years"
result = normalizer.normalize(text)
# Output: "children of three to four years"

# Electronic/URL normalization
text = "visit example.com"
result = normalizer.normalize(text)
# Output: "visit example dot com"

# Configure with cache options
normalizer = Normalizer(
    cache_dir=None,          # Custom cache directory
    overwrite_cache=False    # Set True to rebuild FST graphs
)
```

--------------------------------

### Denormalize Math Expressions

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts mathematical expressions, including scores and absolute values, into a standard numerical format. The `enable_standalone_number` option affects the conversion of negative numbers.

```text
比分定格在七十八比九十六
计算负二的绝对值是二
```

```text
计算负二的绝对值是二
```

```text
正负二的平方都是四
```

```text
正负二的平方都是四
```

--------------------------------

### Apply Blacklist Removals

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Removes specific Chinese words or phrases identified in a blacklist.

```text
呃这个呃啊我不知道
```

--------------------------------

### Denormalize Measures (excluding 'one')

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts measurement expressions, with an option to exclude the word 'one' from conversion.

```text
一年后
```

```text
一年后
```

--------------------------------

### Denormalize Times

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts Chinese time expressions into a standard HH:MM or HH:MM:SS format.

```text
八月十六号十二点之前
我是五点零二分开始的
于五点三十五分三十六秒发射
早上八点半准时开会
```

--------------------------------

### Denormalize Number Series

Source: https://github.com/wenet-e2e/wetextprocessing/blob/master/itn/README.md

Converts sequences of Chinese number words, often used for phone numbers or codes, into Arabic numerals.

```text
可以拨打幺二三零六来咨询
```
Tokens:
Raw