WordSegment

https://grantjenks.com/docs/wordsegment/
Admin
WordSegment is an Apache2 licensed module for English word segmentation, written in pure-Python, and...
Tokens:1,041
Snippets:16
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install WordSegment

Source: https://grantjenks.com/docs/wordsegment

Install the WordSegment module using pip. This is the first step to using the library.

```bash
pip install wordsegment

```

--------------------------------

### Run WordSegment as a Server Process

Source: https://grantjenks.com/docs/wordsegment

Execute the wordsegment module with unbuffered output for use in a server-like process. This example demonstrates reading from stdin and writing to stdout using subprocess.

```python
import subprocess as sp
wordsegment = sp.Popen(
        ['python', '-um', 'wordsegment'],
        stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.STDOUT)
wordsegment.stdin.write('thisisatest\n')
wordsegment.stdout.readline()
wordsegment.stdin.write('workswithotherlanguages\n')
wordsegment.stdout.readline()
wordsegment.stdin.close()
wordsegment.wait()  # Process exit code.

```

--------------------------------

### Explore Bigrams Starting with <s>

Source: https://grantjenks.com/docs/wordsegment

Examine bigram counts that indicate the start of a phrase. The '<s>' prefix denotes the beginning of a bigram sequence.

```python
ws.BIGRAMS['<s> where']
ws.BIGRAMS['<s> what']

```

--------------------------------

### Explore Bigram Counts

Source: https://grantjenks.com/docs/wordsegment

Explore bigram counts, representing the frequency of word pairs. The BIGRAMS dictionary maps phrases (words joined by a space) to their counts. This example shows the top 10 most frequent bigrams.

```python
import heapq
from pprint import pprint
from operator import itemgetter
pprint(heapq.nlargest(10, ws.BIGRAMS.items(), itemgetter(1)))

```

--------------------------------

### Access Documentation

Source: https://grantjenks.com/docs/wordsegment

Access the built-in documentation for the wordsegment module within a Python interpreter.

```python
import wordsegment
help(wordsegment)

```

--------------------------------

### Command-Line Segmentation

Source: https://grantjenks.com/docs/wordsegment

Use the wordsegment module from the command line to segment text piped from stdin to stdout. This is useful for batch processing.

```bash
echo thisisatest | python -m wordsegment

```

--------------------------------

### Segment a Phrase

Source: https://grantjenks.com/docs/wordsegment

Load the word segmentation data and segment a given phrase into a list of words. The load function should be called once before segmenting.

```python
from wordsegment import load, segment
load()
segment('thisisatest')

```

--------------------------------

### Explore Unigram Counts

Source: https://grantjenks.com/docs/wordsegment

Access and explore the unigram counts, which represent the frequency of individual words. The UNIGRAMS dictionary maps words to their counts.

```python
import wordsegment as ws
ws.load()
ws.UNIGRAMS['the']
ws.UNIGRAMS['gray']
ws.UNIGRAMS['grey']

```

--------------------------------

### Clean and Segment Text

Source: https://grantjenks.com/docs/wordsegment

Clean input text by removing punctuation and lowercasing, then segment the cleaned text. The clean function transforms input to a canonical form.

```python
from wordsegment import clean
clean('She said, "Python rocks!"')
segment('She said, "Python rocks!"')

```
Tokens:
Raw