UAX29

https://pkg.go.dev/github.com/clipperhouse/uax29/v2
Admin
The uax29 package provides Unicode text segmentation (UAX #29) for tokenizing words, sentences, and...
Tokens:10,068
Snippets:149
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Quick start with uax29/words

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Basic usage example for tokenizing a string into words.

```go
import "github.com/clipperhouse/uax29/v2/words"

text := "Hello, 世界. Nice dog! 👍🐶"

tokens := words.FromString(text)

for tokens.Next() {                     // Next() returns true until end of data or error
	fmt.Println(tokens.Value())         // Do something with the current token
}
```

--------------------------------

### Install the package

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes

Use the go get command to add the dependency to your project.

```bash
go get github.com/clipperhouse/uax29/v2/graphemes
```

--------------------------------

### Install the phrases package

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Use go get to add the package to your project.

```bash
go get "github.com/clipperhouse/uax29/v2/phrases"
```

--------------------------------

### Install uax29/sentences package

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Use go get to add the package to your project.

```bash
go get "github.com/clipperhouse/uax29/v2/sentences"
```

--------------------------------

### Install uax29 package

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0

Use the go get command to add the words package to your project.

```bash
go get "github.com/clipperhouse/uax29/v2/words"
```

--------------------------------

### Get sentence start position

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Returns the byte position of the current sentence.

```go
func (iter *Iterator[T]) Start() int
```

--------------------------------

### Get first sentence

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Returns the first sentence without advancing the iterator.

```go
func (iter *Iterator[T]) First() T
```

--------------------------------

### Retrieve token positions

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

Methods to get the start and end byte positions of the current token within the original data.

```go
func (iter *Iterator[T]) End() int
```

```go
func (iter *Iterator[T]) Start() int
```

--------------------------------

### Get sentence end position

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Returns the byte position after the current sentence.

```go
func (iter *Iterator[T]) End() int
```

--------------------------------

### Reset iterator

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Resets the iterator to the start of the data.

```go
func (iter *Iterator[T]) Reset()
```

--------------------------------

### Tokenize text with words package

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0

Demonstrates how to import the words package, convert a string into tokens, and iterate through them.

```go
import "github.com/clipperhouse/uax29/v2/words"

text := "Hello, 世界. Nice dog! 👍🐶"

tokens := words.FromString(text)

for tokens.Next() {                     // Next() returns true until end of data
	fmt.Println(tokens.Value())         // Do something with the current token
}
```

--------------------------------

### Create Scanner from reader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Initializes a scanner from an io.Reader.

```go
func FromReader(r io.Reader) *Scanner
```

--------------------------------

### Benchmark results

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes

Performance comparison of the uax29 package against other implementations.

```text
goos: darwin
goarch: arm64
pkg: github.com/clipperhouse/uax29/graphemes/comparative
cpu: Apple M2

BenchmarkGraphemesMixed/clipperhouse/uax29-8  	    142635 ns/op	 245.12 MB/s    0 B/op	   0 allocs/op
BenchmarkGraphemesMixed/rivo/uniseg-8         	   2018284 ns/op	  17.32 MB/s    0 B/op	   0 allocs/op

BenchmarkGraphemesASCII/clipperhouse/uax29-8  	      8846 ns/op	 508.73 MB/s    0 B/op	   0 allocs/op
BenchmarkGraphemesASCII/rivo/uniseg-8         	    366760 ns/op	  12.27 MB/s    0 B/op	   0 allocs/op
```

--------------------------------

### Create Iterator from string

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Initializes an iterator for a string.

```go
func FromString(s string) *Iterator[string]
```

--------------------------------

### Iterator navigation and state methods

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Methods to control the iterator state and retrieve word information.

```go
func (iter *Iterator[T]) End() int
```

```go
func (iter *Iterator[T]) First() T
```

```go
func (iter *Iterator[T]) Joiners(j *Joiners[T])
```

```go
func (iter *Iterator[T]) Next() bool
```

```go
func (iter *Iterator[T]) Reset()
```

```go
func (iter *Iterator[T]) SetText(data T)
```

```go
func (iter *Iterator[T]) Split(split func(T, bool) (int, T, error))
```

```go
func (iter *Iterator[T]) Start() int
```

```go
func (iter *Iterator[T]) Value() T
```

--------------------------------

### Configure Word Joiners

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Sets specific runes to be treated as word characters, preventing them from acting as word boundaries.

```go
func (sc *Scanner) Joiners(j *Joiners[[]byte])
```

--------------------------------

### Configure custom joiners

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Use Joiners to prevent specific characters from breaking words.

```go
text := "Hello, 世界. Tell me about your super-cool .com. I'm .01% interested and 3/4 of a mile away. Email me at foo@example.biz. #winning"
joiners := &words.Joiners{
	Middle:  []rune("@-/"), // appearing in the middle of a word
	Leading: []rune("#."),  // appearing at the front of a word
}

tokens := words.FromString(text)
tokens.Joiners(joiners)

for tokens.Next() {
	fmt.Println(tokens.Value())
}
```

--------------------------------

### Initialize Iterator

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

Creates a new Iterator instance with the provided split function and input data.

```go
func New[T ~string | ~[]byte](split SplitFunc[T], data T) *Iterator[T]
```

--------------------------------

### Iterator Configuration Methods

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/internal/iterators%40v2.7.0

Methods to update the iterator's data or split function.

```go
func (iter *Iterator[T]) SetText(data T)
```

```go
func (iter *Iterator[T]) Split(split SplitFunc[T])
```

--------------------------------

### Tokenize a string

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Tokenize a standard Go string using FromString.

```go
text := "Hello, 世界. Nice dog! 👍🐶"

tokens := words.FromString(text)

for tokens.Next() {                          // Next() returns true until end of data
	fmt.Println(tokens.Value())        // Do something with the current word
}
```

--------------------------------

### phrases.FromBytes

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Creates a tokenizer for a byte slice input.

```APIDOC
## phrases.FromBytes(b []byte)

### Description
Creates a tokenizer from a byte slice. Returns an object with Next() and Value() methods to iterate through phrases.

### Parameters
- **b** ([]byte) - Required - The input byte slice to tokenize.
```

--------------------------------

### Tokenize a byte slice

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Tokenize a byte slice using FromBytes.

```go
b := []byte("Hello, 世界. Nice dog! 👍🐶")

tokens := words.FromBytes(b)

for tokens.Next() {                     // Next() returns true until end of data
	fmt.Println(tokens.Value())         // Do something with the current word
}
```

--------------------------------

### Process phrases from a byte slice

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Tokenize a byte slice directly into phrases.

```go
b := []byte("Hello, 世界. Nice dog! 👍🐶")

tokens := phrases.FromBytes(b)

for tokens.Next() {                     // Next() returns true until end of data
	fmt.Println(tokens.Value())         // Do something with the current phrase
}
```

--------------------------------

### Segment text into phrases

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Basic usage for tokenizing a string into phrases.

```go
import "github.com/clipperhouse/uax29/v2/phrases"

text := "Hello, 世界. Nice — and totally adorable — dog; perhaps the "best one"! 🏆 🐶"

tokens := phrases.FromString(text)

for tokens.Next() {                    // Next() returns true until end of data
	fmt.Println(tokens.Value())        // Do something with the current phrase
}
```

--------------------------------

### Create Iterator from bytes or string

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/sentences

Functions to initialize an iterator for byte slices or strings.

```go
func FromBytes(b []byte) *Iterator[[]byte]
```

```go
func FromString(s string) *Iterator[string]
```

--------------------------------

### Iterator Navigation Methods

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/internal/iterators%40v2.7.0

Methods for traversing and resetting the iterator state.

```go
func (iter *Iterator[T]) First() T
```

```go
func (iter *Iterator[T]) Next() bool
```

```go
func (iter *Iterator[T]) Reset()
```

--------------------------------

### phrases.FromBytes

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/phrases

Creates a token iterator from a byte slice input.

```APIDOC
## phrases.FromBytes(b []byte)

### Description
Parses a byte slice into a sequence of phrases.

### Usage
```go
tokens := phrases.FromBytes(b)
for tokens.Next() {
	fmt.Println(tokens.Value())
}
```
```

--------------------------------

### Segment sentences from a byte slice

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Use FromBytes to process input provided as a byte slice.

```go
b := []byte("Hello, 世界. Nice dog! 👍🐶")

tokens := sentences.FromBytes(b)

for tokens.Next() {                     // Next() returns true until end of data
	fmt.Println(tokens.Value())         // Do something with the current sentence
}
```

--------------------------------

### words.FromReader(r io.Reader)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Creates a token iterator from an io.Reader.

```APIDOC
## words.FromReader(r io.Reader)

### Description
Creates a new token iterator from an io.Reader. This implementation embeds a bufio.Scanner.

### Parameters
- **r** (io.Reader) - Required - The input reader to be segmented.

### Usage
```go
tokens := words.FromReader(r)
for tokens.Scan() {
	fmt.Println(tokens.Text())
}
```
```

--------------------------------

### Joiners Configuration Struct

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Defines the structure for specifying characters that should join tokens instead of splitting them.

```go
	// Leading specifies which characters (runes) should
	// join words (tokens) where they would otherwise be split,
	// at the beginning of a word.
	//
	// For example, specifying "#" will join #hashtags.
	// Specifying "." will preserve leading decimals like .01.
	Leading []rune
}
```

--------------------------------

### Process phrases from a string

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Tokenize a standard Go string into phrases.

```go
text := "Hello, 世界. Nice dog! 👍🐶"

tokens := phrases.FromString(text)

for tokens.Next() {                    // Next() returns true until end of data
	fmt.Println(tokens.Value())        // Do something with the current phrase
}
```

--------------------------------

### Create Iterator from bytes

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Initializes an iterator for a byte slice.

```go
func FromBytes(b []byte) *Iterator[[]byte]
```

--------------------------------

### words.FromBytes(b []byte)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Creates a token iterator from a byte slice input.

```APIDOC
## words.FromBytes(b []byte)

### Description
Creates a new token iterator from the provided byte slice.

### Parameters
- **b** ([]byte) - Required - The input byte slice to be segmented.

### Usage
```go
tokens := words.FromBytes(b)
for tokens.Next() {
	fmt.Println(tokens.Value())
}
```
```

--------------------------------

### Process phrases from an io.Reader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Tokenize input from an io.Reader using FromReader, which utilizes bufio.Scanner.

```go
r := getYourReader()                      // from a file or network maybe
tokens := phrases.FromReader(r)

for tokens.Scan() {                       // Scan() returns true until error or EOF
	fmt.Println(tokens.Text())            // Do something with the current phrase
}

if tokens.Err() != nil {                  // Check the error
	log.Fatal(tokens.Err())
}
```

--------------------------------

### graphemes.FromReader(r io.Reader)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/graphemes

Creates a scanner for an io.Reader input to iterate over grapheme clusters.

```APIDOC
## graphemes.FromReader(r io.Reader)

### Description
Creates a scanner that iterates over grapheme clusters from an io.Reader. This method embeds a bufio.Scanner.

### Usage
```go
r := getYourReader()
g := graphemes.FromReader(r)

for g.Scan() {
	fmt.Println(g.Text())
}

if g.Err() != nil {
	log.Fatal(g.Err())
}
```
```

--------------------------------

### Segment graphemes from a byte slice

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes

Use FromBytes to process raw byte slices.

```go
b := []byte("Hello, 世界. Nice dog! 👍🐶")

g := graphemes.FromBytes(b)

for g.Next() {                     // Next() returns true until end of data
	fmt.Println(g.Value())         // Do something with the current grapheme
}
```

--------------------------------

### phrases.FromString

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Creates a tokenizer for a string input.

```APIDOC
## phrases.FromString(text string)

### Description
Creates a tokenizer from a string. Returns an object with Next() and Value() methods to iterate through phrases.

### Parameters
- **text** (string) - Required - The input string to tokenize.
```

--------------------------------

### phrases.FromReader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Creates a tokenizer for an io.Reader input.

```APIDOC
## phrases.FromReader(r io.Reader)

### Description
Creates a tokenizer from an io.Reader. Returns an object with Scan(), Text(), and Err() methods to iterate through phrases.

### Parameters
- **r** (io.Reader) - Required - The input reader to tokenize.
```

--------------------------------

### Handle ANSI escape sequences

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes

Enable AnsiEscapeSequences to treat ANSI codes as single clusters.

```go
text := "Hello, \x1b[31mworld\x1b[0m!"
g := graphemes.FromString(text)
g.AnsiEscapeSequences = true

for g.Next() {
	fmt.Println(g.Value())
}
```

```go
g.AnsiEscapeSequences = true     // 7-bit forms (ESC ...)
g.AnsiEscapeSequences8Bit = true // 8-bit C1 forms (0x80-0x9F), not valid UTF-8
```

--------------------------------

### Control iteration flow

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

Methods to advance the iterator, reset it, or update its configuration.

```go
func (iter *Iterator[T]) Next() bool
```

```go
func (iter *Iterator[T]) Reset()
```

```go
func (iter *Iterator[T]) SetText(data T)
```

```go
func (iter *Iterator[T]) Split(split SplitFunc[T])
```

--------------------------------

### phrases.FromReader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/phrases

Creates a token iterator from an io.Reader input.

```APIDOC
## phrases.FromReader(r io.Reader)

### Description
Parses an io.Reader into a sequence of phrases using a bufio.Scanner.

### Usage
```go
tokens := phrases.FromReader(r)
for tokens.Scan() {
	fmt.Println(tokens.Text())
}
if tokens.Err() != nil {
	log.Fatal(tokens.Err())
}
```
```

--------------------------------

### Advance Iterator

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/internal/iterators

Advances the iterator to the next available token, returning false if no tokens remain.

```go
func (iter *Iterator[T]) Next() bool
```

--------------------------------

### words.FromString(text string)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Creates a token iterator from a string input.

```APIDOC
## words.FromString(text string)

### Description
Creates a new token iterator from the provided string. The iterator returns all tokens, including whitespace and punctuation.

### Parameters
- **text** (string) - Required - The input string to be segmented.

### Usage
```go
tokens := words.FromString(text)
for tokens.Next() {
	fmt.Println(tokens.Value())
}
```
```

--------------------------------

### Segment sentences from an io.Reader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Use FromReader to process input from an io.Reader, which embeds a bufio.Scanner.

```go
r := getYourReader()                       // from a file or network maybe
tokens := sentences.FromReader(r)

for tokens.Scan() {                        // Scan() returns true until error or EOF
	fmt.Println(tokens.Text())             // Do something with the current sentence
}

if tokens.Err() != nil {                   // Check the error
	log.Fatal(tokens.Err())
}
```

--------------------------------

### Segment graphemes from an io.Reader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes

Use FromReader to process streams, which embeds a bufio.Scanner.

```go
r := getYourReader()                    // from a file or network maybe
g := graphemes.FromReader(r)

for g.Scan() {                         // Scan() returns true until error or EOF
	fmt.Println(g.Text())              // Do something with the current grapheme
}

if g.Err() != nil {                    // Check the error
	log.Fatal(g.Err())
}
```

--------------------------------

### graphemes.FromBytes(b []byte)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/graphemes

Creates a scanner for a byte slice input to iterate over grapheme clusters.

```APIDOC
## graphemes.FromBytes(b []byte)

### Description
Creates a scanner that iterates over grapheme clusters in a provided byte slice.

### Usage
```go
b := []byte("Hello, 世界. Nice dog! 👍🐶")
g := graphemes.FromBytes(b)

for g.Next() {
	fmt.Println(g.Value())
}
```
```

--------------------------------

### Access token data

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

Methods to retrieve the current token or the first token in the sequence.

```go
func (iter *Iterator[T]) First() T
```

```go
func (iter *Iterator[T]) Value() T
```

--------------------------------

### Define Joiners

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/words

Specifies characters that should join tokens instead of breaking them, such as hashtags or decimals.

```go
// Leading specifies which characters (runes) should
	// join words (tokens) where they would otherwise be split,
	// at the beginning of a word.
	//
	// For example, specifying "#" will join #hashtags.
	// Specifying "." will preserve leading decimals like .01.
	Leading []rune
}
```

--------------------------------

### Configure Iterator behavior

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/words

Methods to customize how the iterator splits text or handles specific characters as joiners.

```go
func (iter *Iterator[T]) Joiners(j *Joiners[T])
```

```go
func (iter *Iterator[T]) Split(split func(T, bool) (int, T, error))
```

--------------------------------

### New[T ~string | ~[]byte](split SplitFunc[T], data T) *Iterator[T]

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

Creates a new Iterator instance for the specified data using the provided SplitFunc.

```APIDOC
### func New

Creates a new Iterator for the given data and SplitFunc.

#### Signature
`func New[T ~string | ~[]byte](split SplitFunc[T], data T) *Iterator[T]`
```

--------------------------------

### Define Joiners type

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Configuration for runes that should be treated as word characters to prevent splitting.

```go
type Joiners[T ~string | ~[]byte] struct {
	// Middle specifies which characters (runes) should
	// join words (tokens) where they would otherwise be split,
	// in the middle of a word.
	//
	// For example, specifying "-" will join hypenated-words.
	// Specifying "@" will preserve email addresses.
	//
	// Note that . (as in "example.com") and ' (as in "it's") are already mid-joiners,
	// specifying them will be redundant and hurt performance.
	Middle []rune
```

--------------------------------

### func FromBytes

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Creates an iterator for phrases in a byte slice.

```APIDOC
## func FromBytes(b []byte) *Iterator[[]byte]

### Description
Returns an iterator for the phrases in the input bytes. Iterate while Next() is true, and access the phrase via Value().
```

--------------------------------

### phrases.FromString

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/phrases

Creates a token iterator from a string input.

```APIDOC
## phrases.FromString(text string)

### Description
Parses a string into a sequence of phrases.

### Usage
```go
tokens := phrases.FromString(text)
for tokens.Next() {
	fmt.Println(tokens.Value())
}
```
```

--------------------------------

### Define Scanner struct

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Scanner structure embedding bufio.Scanner for sentence-aware reading.

```go
type Scanner struct {
	*bufio.Scanner
}
```

--------------------------------

### func FromReader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Initializes a new Scanner from an io.Reader to split words based on Unicode TR29 word boundaries.

```APIDOC
## func FromReader

### Description
Returns a Scanner to split words per https://unicode.org/reports/tr29/#Word_Boundaries. It embeds a bufio.Scanner, allowing the use of standard bufio.Scanner methods.

### Signature
`func FromReader(r io.Reader) *Scanner`

### Parameters
- **r** (io.Reader) - Required - The reader source for the text to be scanned.

### Returns
- ***Scanner** - A pointer to a Scanner instance.
```

--------------------------------

### func FromReader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Creates a scanner for reading sentences from an io.Reader.

```APIDOC
## func FromReader(r io.Reader) *Scanner

### Description
Returns a Scanner to split sentences per Unicode TR29 standards. It embeds a bufio.Scanner, allowing standard Scan() and Err() usage.

### Parameters
- **r** (io.Reader) - Required - The reader source for text.
```

--------------------------------

### sentences.FromBytes

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Segments a byte slice into sentences.

```APIDOC
## func FromBytes(b []byte) *Iterator

### Description
Segments a byte slice into sentences and returns an iterator.

### Parameters
- **b** ([]byte) - The input byte slice.

### Usage
```go
b := []byte("Hello, 世界. Nice dog! 👍🐶")
tokens := sentences.FromBytes(b)
for tokens.Next() {
	fmt.Println(tokens.Value())
}
```
```

--------------------------------

### sentences.FromString

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Segments a string into sentences.

```APIDOC
## func FromString(text string) *Iterator

### Description
Segments a string into sentences and returns an iterator.

### Parameters
- **text** (string) - The input text to segment.

### Usage
```go
text := "Hello, 世界. Nice dog! 👍🐶"
tokens := sentences.FromString(text)
for tokens.Next() {
	fmt.Println(tokens.Value())
}
```
```

--------------------------------

### func (*Scanner) Joiners

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Configures custom characters that should join words instead of acting as word boundaries.

```APIDOC
## func (*Scanner) Joiners

### Description
Sets runes that should be treated like word characters, preventing them from splitting words where they would otherwise occur.

### Signature
`func (sc *Scanner) Joiners(j *Joiners[[]byte])`

### Parameters
- **j** (*Joiners) - Required - A pointer to a Joiners struct containing the runes to be treated as joiners.
```

--------------------------------

### func FromBytes

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Creates an iterator for sentences within a byte slice.

```APIDOC
## func FromBytes(b []byte) *Iterator[[]byte]

### Description
Returns an iterator for the sentences in the input bytes. Iterate while Next() is true, and access the sentence via Value().

### Parameters
- **b** ([]byte) - Required - The input byte slice to segment.
```

--------------------------------

### Segment sentences from a string

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Use FromString to iterate over sentences in a string.

```go
import "github.com/clipperhouse/uax29/v2/sentences"

text := "Hello, 世界. Nice dog! 👍🐶"

tokens := sentences.FromString(text)

for tokens.Next() {                         // Next() returns true until end of data
	fmt.Println(tokens.Value())        // Do something with the current sentence
}
```

```go
text := "Hello, 世界. Nice dog! 👍🐶"

tokens := sentences.FromString(text)

for tokens.Next() {                      // Next() returns true until end of data
	fmt.Println(tokens.Value())          // Do something with the current sentence
}
```

--------------------------------

### sentences.FromReader

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Segments text from an io.Reader into sentences.

```APIDOC
## func FromReader(r io.Reader) *Scanner

### Description
Segments text from an io.Reader into sentences using a bufio.Scanner-like interface.

### Parameters
- **r** (io.Reader) - The input source.

### Usage
```go
r := getYourReader()
tokens := sentences.FromReader(r)
for tokens.Scan() {
	fmt.Println(tokens.Text())
}
if tokens.Err() != nil {
	log.Fatal(tokens.Err())
}
```
```

--------------------------------

### graphemes.FromString(text string)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/graphemes

Creates a scanner for a string input to iterate over grapheme clusters.

```APIDOC
## graphemes.FromString(text string)

### Description
Creates a scanner that iterates over grapheme clusters in a provided string.

### Usage
```go
text := "Hello, 世界. Nice dog! 👍🐶"
g := graphemes.FromString(text)

for g.Next() {
	fmt.Println(g.Value())
}
```
```

--------------------------------

### Manage Iterator State

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/internal/iterators

Methods to reset the iterator, update the input text, or change the split function.

```go
func (iter *Iterator[T]) Reset()
```

```go
func (iter *Iterator[T]) SetText(data T)
```

```go
func (iter *Iterator[T]) Split(split SplitFunc[T])
```

--------------------------------

### Define SplitFunc variable

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Provides a bufio.SplitFunc implementation for sentence segmentation.

```go
var SplitFunc bufio.SplitFunc = splitFunc[[]byte]
```

--------------------------------

### tokens.Joiners(joiners *words.Joiners)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Configures custom joiners to prevent specific characters from breaking words.

```APIDOC
## tokens.Joiners(joiners *words.Joiners)

### Description
Configures the iterator to treat specific characters as joiners, preventing them from acting as word boundaries.

### Parameters
- **joiners** (*words.Joiners) - Required - A struct containing Middle and Leading runes to be treated as joiners.

### Usage
```go
joiners := &words.Joiners{
	Middle:  []rune("@-/"),
	Leading: []rune("#."),
}
tokens.Joiners(joiners)
```
```

--------------------------------

### Iterator Methods

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

Methods available on the Iterator instance to control iteration and retrieve token information.

```APIDOC
### Iterator Methods

- **Next() bool**: Advances the iterator to the next token. Returns false when no tokens remain.
- **Value() T**: Returns the current token value.
- **Start() int**: Returns the byte position of the current token in the original data.
- **End() int**: Returns the byte position after the current token in the original data.
- **Reset()**: Resets the iterator to the beginning of the data.
- **SetText(data T)**: Sets new text for the iterator and resets its state.
- **Split(split SplitFunc[T])**: Updates the SplitFunc used by the iterator.
- **First() T**: Returns the first token.
```

--------------------------------

### func FromString

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/phrases

Creates an iterator for phrases in a string.

```APIDOC
## func FromString(s string) *Iterator[string]

### Description
Returns an iterator for the phrases in the input string. Iterate while Next() is true, and access the phrase via Value().
```

--------------------------------

### Define Func type

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/internal/iterators/filter

Defines the signature for filter functions that evaluate a byte slice.

```go
type Func func([]byte) bool
```

--------------------------------

### Set iterator data

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Updates the data source and resets the iterator state.

```go
func (iter *Iterator[T]) SetText(data T)
```

--------------------------------

### Check token types with Bleve segmenter definitions

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/words

These functions determine if a token is comprised of ideographs or numeric characters based on Bleve segmenter definitions.

```go
func BleveIdeographic(token []byte) bool
```

```go
func BleveNumeric(token []byte) bool
```

--------------------------------

### func FromString

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/sentences

Creates an iterator for sentences within a string.

```APIDOC
## func FromString(s string) *Iterator[string]

### Description
Returns an iterator for the sentences in the input string. Iterate while Next() is true, and access the sentence via Value().

### Parameters
- **s** (string) - Required - The input string to segment.
```

--------------------------------

### Scanner API

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/phrases

The Scanner type provides a way to split phrases from an io.Reader, embedding a bufio.Scanner.

```APIDOC
## func FromReader(r io.Reader) *Scanner

### Description
Returns a Scanner to split phrases from an io.Reader. Phrases are defined as a series of words separated only by spaces. It embeds a bufio.Scanner, allowing the use of standard bufio.Scanner methods.
```

--------------------------------

### Segment graphemes from a string

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes

Iterate over grapheme clusters in a string using FromString.

```go
import "github.com/clipperhouse/uax29/v2/graphemes"

text := "Hello, 世界. Nice dog! 👍🐶"
g := graphemes.FromString(text)

for g.Next() {                     // Next() returns true until end of data
	fmt.Println(g.Value())         // Do something with the current grapheme
}
```

```go
text := "Hello, 世界. Nice dog! 👍🐶"
g := graphemes.FromString(text)

for g.Next() {                     // Next() returns true until end of data
	fmt.Println(g.Value())         // Do something with the current grapheme
}
```

--------------------------------

### Iterator.Joiners(joiners *words.Joiners)

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/words

Configures custom joiners to prevent specific characters from breaking words.

```APIDOC
## Iterator.Joiners(joiners *words.Joiners)

### Description
Configures the iterator to treat specific characters as joiners, preventing them from triggering a word boundary split.

### Parameters
- **joiners** (*words.Joiners) - Required - A struct containing 'Middle' and 'Leading' rune slices.

### Usage
```go
joiners := &words.Joiners{
	Middle:  []rune("@-/"),
	Leading: []rune("#."),
}

tokens := words.FromString(text)
tokens.Joiners(joiners)
```
```

--------------------------------

### Scanner

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/graphemes

The Scanner type provides a way to split graphemes from an io.Reader.

```APIDOC
## func FromReader(r io.Reader) *Scanner

### Description
Returns a Scanner to split graphemes from an io.Reader. It embeds a bufio.Scanner, allowing the use of standard bufio.Scanner methods like Scan() and Err().
```

--------------------------------

### Iterator Value Accessor

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/internal/iterators%40v2.7.0

Retrieves the current token from the iterator.

```go
func (iter *Iterator[T]) Value() T
```

--------------------------------

### Check for numeric tokens

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Determines if a token is numeric based on the Bleve segmenter definition.

```go
func BleveNumeric(token []byte) bool
```

--------------------------------

### Define SplitFunc type

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

The function signature used for splitting data into tokens.

```go
type SplitFunc[T ~string | ~[]byte] func(T, bool) (int, T, error)
```

--------------------------------

### Check for ideographic tokens

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/words

Determines if a token consists of ideographs (Han, Katakana, Hiragana) using the Bleve segmenter definition.

```go
func BleveIdeographic(token []byte) bool
```

--------------------------------

### Implement AlphaNumeric filter

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/internal/iterators/filter

A filter implementation that returns true if a token contains at least one Unicode letter or number.

```go
var AlphaNumeric Func = func(token []byte) bool {
	pos := 0
	for pos < len(token) {
		r, w := utf8.DecodeRune(token[pos:])
		if unicode.IsLetter(r) || unicode.IsNumber(r) {

			return true
		}
		pos += w
	}

	return false
}
```

--------------------------------

### Iterator

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/graphemes

The Iterator type allows for traversing grapheme clusters in strings or byte slices. It supports configuration for handling ANSI escape sequences.

```APIDOC
## func FromString(s string) *Iterator[string]

### Description
Returns an iterator for the grapheme clusters in the input string.

## func FromBytes(b []byte) *Iterator[[]byte]

### Description
Returns an iterator for the grapheme clusters in the input bytes.

## func (iter *Iterator[T]) Next() bool

### Description
Advances the iterator to the next grapheme cluster. Returns false when there are no more grapheme clusters.

## func (iter *Iterator[T]) Value() T

### Description
Returns the current grapheme cluster.

## func (iter *Iterator[T]) Start() int

### Description
Returns the byte position of the current grapheme in the original data.

## func (iter *Iterator[T]) End() int

### Description
Returns the byte position after the current grapheme in the original data.
```

--------------------------------

### Iterator API

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/phrases

The Iterator type allows for traversing phrases within strings or byte slices. It provides methods to navigate through segments and retrieve their values and positions.

```APIDOC
## func FromBytes(b []byte) *Iterator[[]byte]

### Description
Returns an iterator for the phrases in the input byte slice.

## func FromString(s string) *Iterator[string]

### Description
Returns an iterator for the phrases in the input string.

## func (*Iterator[T]) Next() bool

### Description
Advances the iterator to the next phrase. Returns false when there are no more phrases.

## func (*Iterator[T]) Value() T

### Description
Returns the current phrase.

## func (*Iterator[T]) Start() int

### Description
Returns the byte position of the current phrase in the original data.

## func (*Iterator[T]) End() int

### Description
Returns the byte position after the current phrase in the original data.

## func (*Iterator[T]) Reset()

### Description
Resets the iterator to the beginning of the data.

## func (*Iterator[T]) SetText(data T)

### Description
Sets the data for the iterator to operate on and resets all state.
```

--------------------------------

### Define Iterator struct

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes

The Iterator struct supports optional handling of ANSI escape sequences as single grapheme clusters.

```go
type Iterator[T ~string | ~[]byte] struct {

	// AnsiEscapeSequences treats 7-bit ANSI escape sequences (ECMA-48) as
	// single grapheme clusters when true. The default is false.
	//
	// 8-bit controls are not enabled by this option. See [AnsiEscapeSequences8Bit].
	AnsiEscapeSequences bool
	// AnsiEscapeSequences8Bit treats 8-bit C1 ANSI escape sequences (ECMA-48) as single
	// grapheme clusters when true. The default is false.
	//
	// 8-bit control bytes are not UTF-8 encoded, i.e. not valid UTF-8. If you
	// choose this option, you are choosing to interpret non-UTF-8 data, caveat
	// emptor.
	AnsiEscapeSequences8Bit bool
	// contains filtered or unexported fields
}
```

--------------------------------

### Define Iterator struct

Source: https://pkg.go.dev/github.com/clipperhouse/uax29/v2%40v2.7.0/internal/iterators

The Iterator struct is a generic type that supports both string and []byte data types.

```go
type Iterator[T ~string | ~[]byte] struct {
	// contains filtered or unexported fields
}
```
Tokens:
Raw