### Install Dependencies with pnpm Source: https://github.com/jo3-l/obscenity/blob/main/CONTRIBUTING.md Run this command after forking and cloning the repository to install all necessary project dependencies. ```bash pnpm install ``` -------------------------------- ### Install Obscenity using npm, yarn, or pnpm Source: https://context7.com/jo3-l/obscenity/llms.txt Install the Obscenity library using your preferred package manager. ```bash npm install obscenity # or yarn add obscenity # or pnpm add obscenity ``` -------------------------------- ### Install Obscenity with npm, yarn, or pnpm Source: https://github.com/jo3-l/obscenity/blob/main/README.md Use your preferred package manager to install the Obscenity library. ```shell $ npm install obscenity $ yarn add obscenity $ pnpm add obscenity ``` -------------------------------- ### Initialize RegExpMatcher with English Dataset Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/variables/englishDataset.md Initialize a RegExpMatcher using the built-in englishDataset and recommended transformers. This is a common setup for profanity detection. ```typescript const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers, }); ``` -------------------------------- ### Build RegExpMatcher options from DataSet Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/DataSet.md Call build() to get a configuration object suitable for initializing a RegExpMatcher. This includes blacklisted terms and whitelisted terms derived from the DataSet. ```typescript // With the RegExpMatcher: const matcher = new RegExpMatcher({ ...dataset.build(), // additional options here }); ``` -------------------------------- ### Create and use asteriskCensorStrategy Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/asteriskCensorStrategy.md Instantiate the asteriskCensorStrategy to get a censoring strategy. Then, set this strategy on a TextCensor instance to censor text. ```typescript const strategy = asteriskCensorStrategy(); const censor = new TextCensor().setStrategy(strategy); // Before: 'fuck you' // After: '**** you' ``` -------------------------------- ### Extend English Dataset Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/variables/englishDataset.md Create a custom dataset by extending the englishDataset. This example adds a new word and removes an existing one, demonstrating dataset manipulation. ```typescript // Extending the data-set by adding a new word and removing an existing one. const myDataset = new DataSet() .addAll(englishDataset) .removePhrasesIf((phrase) => phrase.metadata.originalWord === 'vagina') .addPhrase((phrase) => phrase.addPattern(pattern`|balls|`)); ``` -------------------------------- ### Retrieving All Matches with getAllMatches Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/matchers.md Use `getAllMatches()` to get detailed information about all occurrences of blacklisted terms in the input text. ```typescript const payloads = matcher.getAllMatches(input); ``` -------------------------------- ### remapCharactersTransformer() with Multiple Replacements Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/remapCharactersTransformer.md This example demonstrates mapping multiple characters to a single replacement string. Ensure the transformer is added to the blacklistMatcherTransformers array. ```typescript const transformer = remapCharactersTransformer({ o: 'πŸ‡΄0' }); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` -------------------------------- ### Implement a Stateful Transformer - Obscenity Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/transformers.md Stateful transformers manage internal state. Implement `transform` and `reset` methods. This example collapses duplicate characters. ```typescript class CollapseDuplicates implements StatefulTransformer { private lastCharacter = -1; public transform(char: number) { if (char === this.lastCharacter) return undefined; this.lastCharacter = char; return char; } public reset() { this.lastCharacter = -1; } } ``` -------------------------------- ### keepStartCensorStrategy() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/keepStartCensorStrategy.md A text censoring strategy that extends another strategy, adding the first character matched at the start of the generated string. ```APIDOC ## Function: keepStartCensorStrategy() > **keepStartCensorStrategy**(`baseStrategy`): [`TextCensorStrategy`](../type-aliases/TextCensorStrategy.md) Defined in: [src/censor/BuiltinStrategies.ts:28](https://github.com/jo3-l/obscenity/blob/ae4d9794c82884d20a8b302b776b16d7a17f2d99/src/censor/BuiltinStrategies.ts#L28) A text censoring strategy that extends another strategy, adding the first character matched at the start of the generated string. ### Parameters #### baseStrategy - **baseStrategy** ([`TextCensorStrategy`](../type-aliases/TextCensorStrategy.md)) - Required - Strategy to extend. It will be used to produce the end of the generated string. ### Returns - [`TextCensorStrategy`](../type-aliases/TextCensorStrategy.md) - A [[TextCensorStrategy]] for use with the [[TextCensor]]. ### Examples ```typescript const strategy = keepStartCensorStrategy(grawlixCensorStrategy()); const censor = new TextCensor().setStrategy(strategy); // Before: 'fuck you' // After: 'f$@* you' ``` ```typescript // Since keepEndCensorStrategy() returns another text censoring strategy, you can use it // as the base strategy to pass to keepStartCensorStrategy(). const strategy = keepStartCensorStrategy(keepEndCensorStrategy(asteriskCensorStrategy())); const censor = new TextCensor().setStrategy(strategy); // Before: 'fuck you' // After: 'f**k you' ``` ``` -------------------------------- ### Retrieving Sorted Matches with getAllMatches Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/matchers.md Pass `true` as an argument to `getAllMatches()` to obtain a list of matches sorted by their start index. ```typescript const sortedPayloads = matcher.getAllMatches(input, true); ``` -------------------------------- ### Create and Use Fixed Character Censor Strategy Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/fixedCharCensorStrategy.md Instantiate the `fixedCharCensorStrategy` with a replacement character and then set it on a `TextCensor` instance. This example shows how to censor a string using the created strategy. ```typescript const strategy = fixedCharCensorStrategy('*'); const censor = new TextCensor().setStrategy(strategy); // Before: 'fuck you' // After: '**** you'. ``` -------------------------------- ### Get All Matches with matcher.getAllMatches() Source: https://context7.com/jo3-l/obscenity/llms.txt Retrieve all non-whitelisted matches from the input text using `matcher.getAllMatches()`. Returns an array of `MatchPayload` objects, optionally sorted by position. ```typescript import { RegExpMatcher, englishDataset, englishRecommendedTransformers } from 'obscenity'; const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers, }); const input = 'ΚƒπŸΚƒα½—Ζˆο½‹ α»ΉΠΎα»© π”Ÿβ±αΊ—π™˜Ι¦'; const matches = matcher.getAllMatches(input, true); // sorted by position for (const match of matches) { // Enrich with phrase metadata from the dataset const { phraseMetadata, startIndex, endIndex } = englishDataset.getPayloadWithPhraseMetadata(match); console.log(`"${phraseMetadata?.originalWord}" at [${startIndex}..${endIndex}]`); } // "fuck" at [0..6] // "bitch" at [12..18] ``` -------------------------------- ### Compose DataSet: addAll, removePhrasesIf, addPhrase Source: https://context7.com/jo3-l/obscenity/llms.txt Start from a preset dataset, remove specific phrases based on metadata, and add custom phrases with patterns and metadata. Ensure the matcher is rebuilt after dataset modifications. ```typescript import { DataSet, englishDataset, englishRecommendedTransformers, RegExpMatcher, pattern } from 'obscenity'; // Start from the English preset, remove two words, add a custom one const myDataset = new DataSet<{ originalWord: string }>() .addAll(englishDataset) .removePhrasesIf((phrase) => phrase.metadata?.originalWord === 'bitch' || phrase.metadata?.originalWord === 'ass', ) .addPhrase((phrase) => phrase .setMetadata({ originalWord: 'darnit' }) .addPattern(pattern`darnit`) .addPattern(pattern`d[a]rnit`), ); const matcher = new RegExpMatcher({ ...myDataset.build(), ...englishRecommendedTransformers, }); console.log(matcher.hasMatch('bitch')); // false β€” removed console.log(matcher.hasMatch('darnit')); // true β€” custom word added console.log(matcher.hasMatch('fuck')); // true β€” still present from English preset ``` -------------------------------- ### Built-in Censor Strategy: Keep Start Utility Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/censoring-text.md Extends another censor strategy by preserving the first character of the matched term. Useful for maintaining some readability. ```typescript import { TextCensor, RegExpMatcher, englishDataset, asteriskCensorStrategy, keepStartCensorStrategy } from 'obscenity'; const matcher = new RegExpMatcher({ ...englishDataset.build() }); const censor = new TextCensor().setStrategy(keepStartCensorStrategy(asteriskCensorStrategy())); const text = 'f u c k you!'; const matches = matcher.getAllMatches(text); console.log(censor.applyTo(text, matches)); //> "f*** you!" ``` -------------------------------- ### Get All Profanity Match Positions and Words Source: https://github.com/jo3-l/obscenity/blob/main/README.md Retrieve detailed information about all profanity matches, including their positions and original words. Pass `true` to sort matches by their position. ```javascript // Pass "true" as the "sorted" parameter so the matches are sorted by their position. const matches = matcher.getAllMatches('ΚƒπŸΚƒα½—Ζˆο½‹ α»ΉΠΎα»© π”Ÿβ±αΊ—π™˜Ι¦', true); for (const match of matches) { const { phraseMetadata, startIndex, endIndex } = englishDataset.getPayloadWithPhraseMetadata(match); console.log(`Match for word ${phraseMetadata.originalWord} found between ${startIndex} and ${endIndex}.`); } ``` -------------------------------- ### Creating and Populating a DataSet Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/datasets.md Demonstrates how to create a DataSet and add phrases with associated metadata and patterns. This approach is more organized and maintainable for managing terms. ```typescript import { DataSet, pattern } from 'obscenity'; const dataset = new DataSet<{ originalWord: string }>() // addPhrase() adds a new phrase to the dataset. .addPhrase((phrase) => phrase // setMetadata() sets the metadata of the phrase. .setMetadata({ originalWord: 'fuck' }) // addPattern() associates a pattern with the phrase. .addPattern(pattern`fck`) .addPattern(pattern`fuck`), ) .addPhrase((phrase) => phrase .setMetadata({ originalWord: 'bitch' }) .addPattern(pattern`bish`) .addPattern(pattern`bitch`) // addWhitelistedTerm() associates a whitelisted term with the phrase. .addWhitelistedTerm('abish'), ); ``` -------------------------------- ### Constructor Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/DataSet.md Initializes a new instance of the DataSet class. ```APIDOC ## new DataSet() ### Description Initializes a new instance of the DataSet class. ### Returns `DataSet` ``` -------------------------------- ### Constructor Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/RegExpMatcher.md Creates a new RegExpMatcher with the specified options. Options can include blacklisted terms, whitelisted terms, and transformers for advanced matching. ```APIDOC ## Constructor ### Description Creates a new [[RegExpMatcher]] with the options given. ### Parameters #### options - **options** (`RegExpMatcherOptions`) - Required - Options to use. ### Returns - `RegExpMatcher` ### Examples ```typescript // Use the options provided by the English preset. const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers, }); ``` ```typescript // Simple matcher that only has blacklisted patterns. const matcher = new RegExpMatcher({ blacklistedTerms: assignIncrementingIds([ pattern`fuck`, pattern`f?uck`, // wildcards (?) pattern`bitch`, pattern`b[i]tch` // optionals ([i] matches either "i" or "") ]), }); // Check whether some string matches any of the patterns. const doesMatch = matcher.hasMatch('fuck you bitch'); ``` ```typescript // A more advanced example, with transformers and whitelisted terms. const matcher = new RegExpMatcher({ blacklistedTerms: [ { id: 1, pattern: pattern`penis` }, { id: 2, pattern: pattern`fuck` }, ], whitelistedTerms: ['pen is'], blacklistMatcherTransformers: [ resolveConfusablesTransformer(), // 'πŸ…°' => 'a' resolveLeetSpeakTransformer(), // '$' => 's' foldAsciiCharCaseTransformer(), // case insensitive matching skipNonAlphabeticTransformer(), // 'f.u...c.k' => 'fuck' collapseDuplicatesTransformer(), // 'aaaa' => 'a' ], }); // Output all matches. console.log(matcher.getAllMatches('fu.....uuuuCK the pen is mightier than the sword!')); ``` ``` -------------------------------- ### build() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/DataSet.md Returns the dataset in a format suitable for RegExpMatcher. ```APIDOC ## build(): Pick ### Description Returns the dataset in a format suitable for usage with the [[RegExpMatcher]]. ### Returns `Pick` ### Example ```typescript // With the RegExpMatcher: const matcher = new RegExpMatcher({ ...dataset.build(), // additional options here }); ``` ``` -------------------------------- ### Create a Transformer to Skip Characters - Obscenity Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/transformers.md To ignore a character, return `undefined` from the transformer function. This example skips spaces. ```typescript import { createSimpleTransformer } from 'obscenity'; const space = ' '.charCodeAt(0); const skipSpaces = createSimpleTransformer((c) => (c === space ? undefined : c)); ``` -------------------------------- ### build() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/PhraseBuilder.md Builds the phrase and returns a PhraseContainer. ```APIDOC ## build(): PhraseContainer ### Description Builds the phrase, returning a [[PhraseContainer]] for use with the [[DataSet]]. ### Returns [`PhraseContainer`](../interfaces/PhraseContainer.md)<`MetadataType`> - The constructed PhraseContainer. ``` -------------------------------- ### Create a RegExpMatcher with English Preset Source: https://github.com/jo3-l/obscenity/blob/main/README.md Initialize a RegExpMatcher instance using the English dataset and recommended transformers for profanity detection. ```javascript const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers, }); ``` -------------------------------- ### `assignIncrementingIds()` Source: https://context7.com/jo3-l/obscenity/llms.txt Wraps an array of `ParsedPattern` objects into `BlacklistedTerm` objects with auto-incrementing numeric IDs, starting from 0. Use when custom IDs are not needed. ```APIDOC ## `assignIncrementingIds()` β€” Auto ID Assignment Wraps an array of `ParsedPattern` objects into `BlacklistedTerm` objects with auto-incrementing numeric IDs, starting from `0`. Use this when you don't need to track which specific pattern matched by a custom ID. ```typescript import { assignIncrementingIds, pattern, RegExpMatcher } from 'obscenity'; const terms = assignIncrementingIds([ pattern`f?uck`, pattern`|shit|`, pattern`bitch`, ]); // [ // { id: 0, pattern: … }, // { id: 1, pattern: … }, // { id: 2, pattern: … }, // ] const matcher = new RegExpMatcher({ blacklistedTerms: terms }); const matches = matcher.getAllMatches('fuck that shit'); console.log(matches.map((m) => m.termId)); // [0, 1] ``` ``` -------------------------------- ### Using English Preset with RegExpMatcher Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/matchers.md Initialize RegExpMatcher using the English preset for blacklisted terms, whitelisted terms, and recommended transformers. ```typescript import { RegExpMatcher, englishDataset, englishRecommendedTransformers } from 'obscenity'; const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers, }); ``` -------------------------------- ### resolveConfusablesTransformer() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/resolveConfusablesTransformer.md Creates a transformer that maps confusable Unicode characters to their normalized equivalent. It is recommended to apply this transformer near the start of the transformer chain. ```APIDOC ## Function: resolveConfusablesTransformer() ### Description Creates a transformer that maps confusable Unicode characters to their normalized equivalent. For example, `β“΅`, `➊`, and `β‘΄` become `1` when using this transformer. It is recommended that this transformer be applied near the start of the transformer chain. ### Returns [`SimpleTransformerContainer`](../interfaces/SimpleTransformerContainer.md) A container holding the transformer, which can then be passed to the [[RegExpMatcher]]. ### Example ```typescript const transformer = resolveConfusablesTransformer(); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` ``` -------------------------------- ### Building a Matcher from a DataSet Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/datasets.md Shows how to build a RegExpMatcher using the terms and configurations from a DataSet. This includes options for directly passing the built object or using spread notation. ```typescript const built = dataset.build(); const matcher = new RegExpMatcher({ blacklistedTerms: built.blacklistedTerms, whitelistedTerms: built.whitelistedTerms, // Other options go here. }); // Or, using spread notation: const matcher = new RegExpMatcher({ ...built, // Other options go here. }); ``` -------------------------------- ### Import Obscenity Classes and Presets (JavaScript) Source: https://github.com/jo3-l/obscenity/blob/main/README.md Import necessary classes and datasets for using Obscenity in a JavaScript project. ```javascript const { RegExpMatcher, TextCensor, englishDataset, englishRecommendedTransformers } = require('obscenity'); ``` -------------------------------- ### Get phrase metadata for a match payload Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/DataSet.md Use getPayloadWithPhraseMetadata to attach phrase-specific metadata to a match payload. This is helpful for understanding the context of a detected match. ```typescript const matches = matcher.getAllMatches(input); const matchesWithPhraseMetadata = matches.map((match) => dataset.getPayloadWithPhraseMetadata(match)); // Now we can access the 'phraseMetadata' property: const phraseMetadata = matchesWithPhraseMetadata[0].phraseMetadata; ``` -------------------------------- ### Basic Pattern Matching (Pre-Dataset) Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/datasets.md Illustrates a less maintainable approach using if-else chains for pattern matching before datasets were introduced. This method becomes unmanageable with a large number of patterns. ```typescript const patterns = [ { id: 0, pattern: pattern`fck` }, { id: 1, pattern: pattern`fuck` }, { id: 2, pattern: pattern`bish` }, { id: 3, pattern: pattern`bitch` }, // ... ]; const matcher = new RegExpMatcher({ ... }); const payloads = matcher.getAllMatches(text); for (const payload of payloads) { if (payload.termId === 0 || payload.termId === 1) console.log('Original word: fuck'); else if (payload.termId === 2 || payload.termId === 3) console.log('Original word: bitch'); // ... } ``` -------------------------------- ### Run Tests Source: https://github.com/jo3-l/obscenity/blob/main/CONTRIBUTING.md After making changes, run this command to verify that all tests are passing and no regressions have been introduced. ```bash pnpm test ``` -------------------------------- ### assignIncrementingIds() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/assignIncrementingIds.md Assigns incrementing IDs to patterns, starting from 0. This is useful when you need to match against a list of patterns but do not require specific identification of which pattern matched. ```APIDOC ## Function: assignIncrementingIds() > **assignIncrementingIds**(`patterns`): [`BlacklistedTerm`](../interfaces/BlacklistedTerm.md)[] Defined in: [src/matcher/BlacklistedTerm.ts:37](https://github.com/jo3-l/obscenity/blob/ae4d9794c82884d20a8b302b776b16d7a17f2d99/src/matcher/BlacklistedTerm.ts#L37) Assigns incrementing IDs to the patterns provided, starting with 0. It is useful if you have a list of patterns to match against but don't care about identifying which pattern matched. ### Parameters #### patterns [`ParsedPattern`](../interfaces/ParsedPattern.md)[] List of parsed patterns. ### Returns [`BlacklistedTerm`](../interfaces/BlacklistedTerm.md)[] A list of blacklisted terms with valid IDs which can then be passed to the [[RegExpMatcher]]. ### Example ```typescript const matcher = new RegExpMatcher({ ..., blacklistedTerms: assignIncrementingIds([ pattern`f?uck`, pattern`|shit|`, ]), }); ``` ``` -------------------------------- ### TextCensor Constructor Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/TextCensor.md Initializes a new instance of the TextCensor class. ```APIDOC ## Constructor > **new TextCensor**(): `TextCensor` #### Returns `TextCensor` ``` -------------------------------- ### PhraseBuilder Constructor Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/PhraseBuilder.md Initializes a new instance of the PhraseBuilder class. ```APIDOC ## new PhraseBuilder() ### Description Initializes a new instance of the PhraseBuilder class. ### Returns `PhraseBuilder` - A new instance of PhraseBuilder. ``` -------------------------------- ### compareMatchByPositionAndId() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/compareMatchByPositionAndId.md Compares two match payloads. It first compares their start and end indices, and then their term IDs if the indices are equal. This is useful for sorting match results. ```APIDOC ## Function: compareMatchByPositionAndId() > **compareMatchByPositionAndId**(`a`, `b`): `-1` | `0` | `1` Compares two match payloads. If the first match payload's start index is less than the second's, -1 is returned; If the second match payload's start index is less than the first's, 1 is returned; If the first match payload's end index is less than the second's, -1 is returned; If the second match payload's end index is less than the first's, 1 is returned; If the first match payload's term ID is less than the second's, -1 is returned; If the first match payload's term ID is equal to the second's, 0 is returned; Otherwise, 1 is returned. ### Parameters #### a [`MatchPayload`](../interfaces/MatchPayload.md) First match payload. #### b [`MatchPayload`](../interfaces/MatchPayload.md) Second match payload. ### Returns -1 | 0 | 1 The result of the comparison: -1 if the first should sort lower than the second, 0 if they are the same, and 1 if the second should sort lower than the first. ``` -------------------------------- ### Create RegExpMatcher with Blacklisted Patterns Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/RegExpMatcher.md Shows how to initialize RegExpMatcher with a simple list of blacklisted terms, including patterns with wildcards and optional characters. Use this for basic profanity detection. ```typescript const matcher = new RegExpMatcher({ blacklistedTerms: assignIncrementingIds([ pattern`fuck`, pattern`f?uck`, // wildcards (?) pattern`bitch`, pattern`b[i]tch` // optionals ([i] matches either "i" or "") ]), }); // Check whether some string matches any of the patterns. const doesMatch = matcher.hasMatch('fuck you bitch'); ``` -------------------------------- ### Lint and Format Code Source: https://github.com/jo3-l/obscenity/blob/main/CONTRIBUTING.md Execute these commands to ensure your code adheres to the project's linting and formatting standards before committing. ```bash pnpm lint ``` ```bash pnpm style ``` -------------------------------- ### Censor Profanities in Text Source: https://github.com/jo3-l/obscenity/blob/main/README.md Utilize the `TextCensor` class to replace detected profanities within a given text. This example assumes `matcher` and `matches` are already defined. ```javascript const { TextCensor, ... } = require('obscenity'); // ... const censor = new TextCensor(); const input = 'fuck you little bitch'; const matches = matcher.getAllMatches(input); console.log(censor.applyTo(input, matches)); ``` -------------------------------- ### Use RegExpMatcher with English Preset Source: https://context7.com/jo3-l/obscenity/llms.txt Initialize RegExpMatcher with the built-in English dataset and recommended transformers for basic profanity detection. Handles leet-speak, duplicates, and punctuation normalization. ```typescript import { RegExpMatcher, TextCensor, englishDataset, englishRecommendedTransformers, assignIncrementingIds, pattern, resolveConfusablesTransformer, resolveLeetSpeakTransformer, toAsciiLowerCaseTransformer, collapseDuplicatesTransformer, } from 'obscenity'; // --- Example 1: Using the built-in English preset --- const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers, }); console.log(matcher.hasMatch('what the fu.....uuuuCK')); // true (leet-speak + duplicates + punctuation are all normalised) console.log(matcher.hasMatch('the pen is mightier than the sword')); // false ("penis" is whitelisted when preceded by "pen is") ``` -------------------------------- ### Built-in Censor Strategy: Fixed Character Utility Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/censoring-text.md Creates a censor strategy that repeats a specified character to match the length of the censored term. For example, '$' repeated. ```typescript import { TextCensor, RegExpMatcher, englishDataset, fixedCharCensorStrategy } from 'obscenity'; const matcher = new RegExpMatcher({ ...englishDataset.build() }); const censor = new TextCensor().setStrategy(fixedCharCensorStrategy('$')); const text = 'f u c k you!'; const matches = matcher.getAllMatches(text); console.log(censor.applyTo(text, matches)); //> "$$$$ you!" ``` -------------------------------- ### Extending Existing Datasets Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/datasets.md Illustrates how to extend an existing dataset, such as the English preset, by adding all its data and then removing specific phrases based on a condition. This is useful for customizing default datasets. ```typescript const myDataset = new DataSet<{ originalWord: string }>() // addAll() adds all the data from the dataset passed. .addAll(englishDataset) // removePhrasesIf() removes phrases from the current dataset if the function provided // returns true. .removePhrasesIf((phrase) => phrase.metadata.originalWord === 'bitch'); ``` ```typescript const matcher = new RegExpMatcher({ ...myDataset.build(), // Other options go here. }); ``` -------------------------------- ### Assigning IDs to Patterns Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/assignIncrementingIds.md Use assignIncrementingIds to prepare a list of patterns for the RegExpMatcher when individual pattern identification is not necessary. This function ensures each pattern gets a unique ID. ```typescript const matcher = new RegExpMatcher({ ..., blacklistedTerms: assignIncrementingIds([ pattern`f?uck`, pattern`|shit|`, ]), }); ``` -------------------------------- ### Import Obscenity Classes and Presets (TypeScript/ESM) Source: https://github.com/jo3-l/obscenity/blob/main/README.md Import necessary classes and datasets for using Obscenity in a TypeScript or ESM project. ```typescript import { RegExpMatcher, TextCensor, englishDataset, englishRecommendedTransformers } from 'obscenity'; ``` -------------------------------- ### Create a New Feature Branch Source: https://github.com/jo3-l/obscenity/blob/main/CONTRIBUTING.md Before making changes, create a new branch for your feature using this command. Replace 'my-feature' with a descriptive name for your changes. ```bash git checkout -b feat/my-feature ``` -------------------------------- ### remapCharactersTransformer() with Map Mapping Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/remapCharactersTransformer.md Use this when you need to map characters using a Map object, for example, to replace a character with an emoji. Ensure the transformer is added to the blacklistMatcherTransformers array. ```typescript const transformer = remapCharactersTransformer(new Map([['b', 'πŸ…±οΈ']])); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` -------------------------------- ### Get All Matches from Input String Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/RegExpMatcher.md Retrieves all occurrences of blacklisted terms within a given input string. For efficiency when only presence checking is needed, consider using the `hasMatch()` method. ```typescript const matches = matcher.getAllMatches('input string'); ``` -------------------------------- ### RegExpMatcherOptions Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/interfaces/RegExpMatcherOptions.md Options for configuring the RegExpMatcher, including blacklisted terms, whitelisted terms, and transformers for both. ```APIDOC ## Interface: RegExpMatcherOptions Options for the [[RegExpMatcher]]. ### Properties - **blacklistedTerms** (`BlacklistedTerm[]`) - A list of blacklisted terms. - **blacklistMatcherTransformers?** (`TransformerContainer[]`) - Optional. A set of transformers applied before blacklisted patterns are matched. Applied in order. - **whitelistedTerms?** (`string[]`) - Optional. A list of whitelisted terms. Matches within these terms are ignored. Default: `[]`. - **whitelistMatcherTransformers?** (`TransformerContainer[]`) - Optional. A set of transformers applied before whitelisted terms are matched. Applied in order. Default: `[]`. ``` -------------------------------- ### matcher.getAllMatches() Source: https://context7.com/jo3-l/obscenity/llms.txt Retrieves all non-whitelisted matches within a string, returning them as an array of MatchPayload objects. Each object details the matched term ID, start and end indices, and match length. ```APIDOC ## matcher.getAllMatches() ### Description Returns every non-whitelisted match in the input as an array of `MatchPayload` objects, each containing `termId`, `startIndex`, `endIndex`, and `matchLength`. Pass `true` as the second argument to sort results by position. ### Method `RegExpMatcher.prototype.getAllMatches(text: string, sorted?: boolean): MatchPayload[]` ### Parameters #### Path Parameters None #### Query Parameters - **sorted** (boolean) - Optional - If `true`, the results are sorted by their starting position in the text. #### Request Body None ### Request Example ```typescript import { RegExpMatcher, englishDataset, englishRecommendedTransformers } from 'obscenity'; const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers, }); const input = 'ΚƒπŸΚƒα½—Ζˆο½‹ α»ΉΠΎα»© π”Ÿβ±αΊ—π™˜Ι¦'; const matches = matcher.getAllMatches(input, true); // sorted by position for (const match of matches) { // Enrich with phrase metadata from the dataset const { phraseMetadata, startIndex, endIndex } = englishDataset.getPayloadWithPhraseMetadata(match); console.log(`"${phraseMetadata?.originalWord}" at [${startIndex}..${endIndex}]`); } // "fuck" at [0..6] // "bitch" at [12..18] ``` ### Response #### Success Response (MatchPayload[]) An array of `MatchPayload` objects. Each object has the following structure: - **termId** (number) - The unique identifier for the matched term. - **startIndex** (number) - The starting index of the match in the input string. - **endIndex** (number) - The ending index of the match in the input string. - **matchLength** (number) - The length of the matched substring. #### Response Example ```json [ { "termId": 1, "startIndex": 0, "endIndex": 6, "matchLength": 4 }, { "termId": 2, "startIndex": 12, "endIndex": 18, "matchLength": 5 } ] ``` ``` -------------------------------- ### Create a Simple Transformer - Obscenity Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/transformers.md Use `createSimpleTransformer` to adapt a character-mapping function for use with matchers. Remember to use character codes as input. ```typescript import { createSimpleTransformer } from 'obscenity'; const a = 'a'.charCodeAt(0); const b = 'b'.charCodeAt(0); const changeAToB = createSimpleTransformer((c) => (c === a ? b : c)); ``` -------------------------------- ### Create and Use Leet-Speak Transformer Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/resolveLeetSpeakTransformer.md Instantiate the leet-speak transformer and include it in the RegExpMatcher's blacklistMatcherTransformers. This transformer maps leet-speak characters to their normalized equivalents. ```typescript const transformer = resolveLeetSpeakTransformer(); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` -------------------------------- ### Auto Assign IDs with `assignIncrementingIds()` Source: https://context7.com/jo3-l/obscenity/llms.txt Wrap `ParsedPattern` objects with `assignIncrementingIds` to create `BlacklistedTerm` objects with auto-incrementing numeric IDs starting from 0. This is useful when you do not need custom IDs for tracking specific patterns. ```typescript import { assignIncrementingIds, pattern, RegExpMatcher } from 'obscenity'; const terms = assignIncrementingIds([ pattern`f?uck`, pattern`|shit|`, pattern`bitch`, ]); // [ // { id: 0, pattern: … }, // { id: 1, pattern: … }, // { id: 2, pattern: … }, // ] const matcher = new RegExpMatcher({ blacklistedTerms: terms }); const matches = matcher.getAllMatches('fuck that shit'); console.log(matches.map((m) => m.termId)); // [0, 1] ``` -------------------------------- ### addAll() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/DataSet.md Adds all the phrases from another dataset to this one. ```APIDOC ## addAll(other: DataSet): DataSet ### Description Adds all the phrases from the dataset provided to this one. ### Parameters #### other - **other** (`DataSet`) - Other dataset. ### Returns `DataSet` ### Example ```typescript const customDataset = new DataSet().addAll(englishDataset); ``` ``` -------------------------------- ### CensorContext Type Alias Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/type-aliases/CensorContext.md The CensorContext type alias is a union of MatchPayload and an object, used to pass context to text censoring strategies. It includes the original input string and boolean flags indicating overlaps at the start and end of a region. ```APIDOC ## Type Alias: CensorContext > **CensorContext** = [`MatchPayload`](../interfaces/MatchPayload.md) & `object` ### Description Context passed to [[TextCensorStrategy | text censoring strategies]]. ### Type declaration #### input > **input**: `string` > The entire input text, without any censoring applied to it. #### overlapsAtEnd > **overlapsAtEnd**: `boolean` > Whether the current region overlaps at the end with some other region. #### overlapsAtStart > **overlapsAtStart**: `boolean` > Whether the current region overlaps at the start with some other region. ``` -------------------------------- ### `DataSet` Source: https://context7.com/jo3-l/obscenity/llms.txt Manages phrases, grouping related patterns and whitelisted terms, and associates metadata with each group. The `build()` method prepares the dataset for `RegExpMatcher`. ```APIDOC ## `DataSet` β€” Phrase Management with Metadata Groups related patterns and whitelisted terms into named "phrases" and associates arbitrary typed metadata with each group. The `build()` method converts the dataset into the format expected by `RegExpMatcher`. ```typescript import { DataSet, RegExpMatcher, pattern, englishDataset, englishRecommendedTransformers } from 'obscenity'; // --- Building a custom dataset with metadata --- const dataset = new DataSet<{ originalWord: string; severity: 'low' | 'high' }>() .addPhrase((phrase) => phrase .setMetadata({ originalWord: 'fuck', severity: 'high' }) .addPattern(pattern`f[?]ck`) .addPattern(pattern`|fk`) .addWhitelistedTerm('kung-fu'), ) .addPhrase((phrase) => phrase .setMetadata({ originalWord: 'bitch', severity: 'high' }) .addPattern(pattern`bitch`) .addPattern(pattern`bich|`), ); const matcher = new RegExpMatcher({ ...dataset.build(), ...englishRecommendedTransformers, }); const matches = matcher.getAllMatches('fck you bitch', true); for (const match of matches) { const { phraseMetadata, startIndex, endIndex } = dataset.getPayloadWithPhraseMetadata(match); console.log(`[${phraseMetadata?.severity}] "${phraseMetadata?.originalWord}" at ${startIndex}-${endIndex}`); } // [high] "fuck" at 0-2 // [high] "bitch" at 7-11 ``` ``` -------------------------------- ### collapseDuplicatesTransformer() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/collapseDuplicatesTransformer.md Creates a transformer that collapses duplicate characters. This is useful for detecting variants of patterns in which a character is repeated to bypass detection. For example, the pattern `hi` does not match `hhiii` by default, as the frequency of the characters does not match. With this transformer, `hhiii` would become `hi`, and would therefore match the pattern. ```APIDOC ## Function: collapseDuplicatesTransformer() > **collapseDuplicatesTransformer**(`options`): [`StatefulTransformerContainer`](../interfaces/StatefulTransformerContainer.md) Defined in: [src/transformer/collapse-duplicates/index.ts:46](https://github.com/jo3-l/obscenity/blob/ae4d9794c82884d20a8b302b776b16d7a17f2d99/src/transformer/collapse-duplicates/index.ts#L46) Creates a transformer that collapses duplicate characters. This is useful for detecting variants of patterns in which a character is repeated to bypass detection. As an example, the pattern `hi` does not match `hhiii` by default, as the frequency of the characters does not match. With this transformer, `hhiii` would become `hi`, and would therefore match the pattern. **Application order** It is recommended that this transformer be applied after all other transformers. Using it before other transformers may have the effect of not catching duplicates of certain characters that were originally different but became the same after a series of transformations. **Warning** This transformer should be used with caution, as while it can make certain patterns match text that wouldn't have been matched before, it can also go the other way. For example, the pattern `hello` clearly matches `hello`, but with this transformer, by default, `hello` would become `helo` which does _not_ match. In this cases, the `customThresholds` option can be used to allow two `l`s in a row, making it leave `hello` unchanged. ### Parameters #### options [`CollapseDuplicatesTransformerOptions`](../interfaces/CollapseDuplicatesTransformerOptions.md) = `{}` Options for the transformer. ### Returns [`StatefulTransformerContainer`](../interfaces/StatefulTransformerContainer.md) A container holding the transformer, which can then be passed to the [[RegExpMatcher]]. ### Examples ```typescript // Collapse runs of the same character. const transformer = collapseDuplicatesTransformer(); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` ```typescript // Collapse runs of characters other than 'a'. const transformer = collapseDuplicatesTransformer({ customThresholds: new Map([['a', Infinity]]) }); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` ``` -------------------------------- ### Using keepStartCensorStrategy with grawlixCensorStrategy Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/keepStartCensorStrategy.md Demonstrates how to use keepStartCensorStrategy by extending the grawlixCensorStrategy. This strategy censors text by replacing characters with '$' while keeping the first character of the original word. ```typescript const strategy = keepStartCensorStrategy(grawlixCensorStrategy()); const censor = new TextCensor().setStrategy(strategy); // Before: 'fuck you' // After: 'f$@* you' ``` -------------------------------- ### Configuring RegExpMatcher with Custom Options Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/matchers.md Configure the RegExpMatcher with custom blacklisted terms, whitelisted terms, and transformers for blacklist and whitelist matching. ```typescript import { RegExpMatcher, pattern } from 'obscenity'; const matcher = new RegExpMatcher({ blacklistedTerms: [ { id: 0, pattern: pattern`hi` }, { id: 1, pattern: pattern`bye` }, ], whitelistedTerms: ['achingly'], blacklistMatcherTransformers: [skipSpaces], whitelistMatcherTransformers: [], }); // This will match `hi` and `bye` (ignoring spaces) unless the `hi` is part of `achingly` (not ignoring spaces). ``` -------------------------------- ### Advanced RegExpMatcher with Transformers and Whitelisting Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/RegExpMatcher.md Illustrates an advanced RegExpMatcher configuration with blacklisted and whitelisted terms, along with various transformers for sophisticated text processing like resolving confusables, leetspeak, case folding, skipping non-alphabetic characters, and collapsing duplicates. Use this for complex profanity detection scenarios. ```typescript const matcher = new RegExpMatcher({ blacklistedTerms: [ { id: 1, pattern: pattern`penis` }, { id: 2, pattern: pattern`fuck` }, ], whitelistedTerms: ['pen is'], blacklistMatcherTransformers: [ resolveConfusablesTransformer(), // 'πŸ…°' => 'a' resolveLeetSpeakTransformer(), // '$' => 's' foldAsciiCharCaseTransformer(), // case insensitive matching skipNonAlphabeticTransformer(), // 'f.u...c.k' => 'fuck' collapseDuplicatesTransformer(), // 'aaaa' => 'a' ], }); // Output all matches. console.log(matcher.getAllMatches('fu.....uuuuCK the pen is mightier than the sword!')); ``` -------------------------------- ### addPattern() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/classes/PhraseBuilder.md Associates a pattern with the phrase being built. ```APIDOC ## addPattern(pattern: ParsedPattern): PhraseBuilder ### Description Associates a pattern with this phrase. ### Parameters #### pattern - **pattern** (`ParsedPattern`) - Required - Pattern to add. ### Returns `PhraseBuilder` - The PhraseBuilder instance for chaining. ``` -------------------------------- ### Default RegExpMatcherOptions Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/interfaces/RegExpMatcherOptions.md Shows the default empty arrays for blacklistMatcherTransformers, whitelistedTerms, and whitelistMatcherTransformers. ```typescript [] ``` -------------------------------- ### Using keepEndCensorStrategy Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/keepEndCensorStrategy.md Demonstrates how to create and use the keepEndCensorStrategy by extending the asteriskCensorStrategy and applying it to a TextCensor instance. This strategy censors text while keeping the last character of the censored portion. ```typescript const strategy = keepEndCensorStrategy(asteriskCensorStrategy()); const censor = new TextCensor().setStrategy(strategy); // Before: 'fuck you' // After: '***k you' ``` -------------------------------- ### resolveLeetSpeakTransformer() Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/resolveLeetSpeakTransformer.md Creates a transformer that maps leet-speak characters to their normalized equivalent. It's recommended to apply this transformer early in the chain. ```APIDOC ## Function: resolveLeetSpeakTransformer() > **resolveLeetSpeakTransformer**(): [`SimpleTransformerContainer`](../interfaces/SimpleTransformerContainer.md) Defined in: [src/transformer/resolve-leetspeak/index.ts:23](https://github.com/jo3-l/obscenity/blob/ae4d9794c82884d20a8b302b776b16d7a17f2d99/src/transformer/resolve-leetspeak/index.ts#L23) Creates a transformer that maps leet-speak characters to their normalized equivalent. For example, `$` becomes `s` when using this transformer. **Application order** It is recommended that this transformer be applied near the start of the transformer chain, but after similar transformers that map characters to other characters, such as the [[resolveConfusablesTransformer | transformer that resolves confusable Unicode characters]]. ## Returns [`SimpleTransformerContainer`](../interfaces/SimpleTransformerContainer.md) A container holding the transformer, which can then be passed to the [[RegExpMatcher]]. ## Example ```typescript const transformer = resolveLeetSpeakTransformer(); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` ``` -------------------------------- ### Create and Apply skipNonAlphabeticTransformer Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/skipNonAlphabeticTransformer.md Instantiate the transformer and include it in the RegExpMatcher's configuration. This transformer is not part of the default set and should be used with caution due to known issues. ```typescript const transformer = skipNonAlphabeticTransformer(); const matcher = new RegExpMatcher({ ..., blacklistMatcherTransformers: [transformer] }); ``` -------------------------------- ### Basic Pattern Matching Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/functions/pattern.md Use this to match a literal string followed by any single character. ```typescript const parsed = pattern`hello?`; // match "hello", then any character ``` -------------------------------- ### Retrieving Phrase Metadata from Matches Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/datasets.md Demonstrates how to use `getPayloadWithPhraseMetadata` to retrieve phrase metadata associated with matches. This allows access to custom data like the original word. ```typescript const payloads = matcher.getAllMatches(input); const payloadsWithMetadata = payloads.map(dataset.getPayloadWithPhraseMetadata); ``` ```typescript const originalWord = payloadsWithMetadata[0].phraseMetadata!.originalWord; ``` -------------------------------- ### Manage Phrases and Metadata with `DataSet` Source: https://context7.com/jo3-l/obscenity/llms.txt Use `DataSet` to group related patterns and whitelisted terms into named phrases, associating typed metadata with each group. The `build()` method converts the dataset for `RegExpMatcher`. Retrieve phrase metadata using `getPayloadWithPhraseMetadata`. ```typescript import { DataSet, RegExpMatcher, pattern, englishDataset, englishRecommendedTransformers } from 'obscenity'; // --- Building a custom dataset with metadata --- const dataset = new DataSet<{ originalWord: string; severity: 'low' | 'high' }>() .addPhrase((phrase) => phrase .setMetadata({ originalWord: 'fuck', severity: 'high' }) .addPattern(pattern`f[?]ck`) .addPattern(pattern`|fk`) .addWhitelistedTerm('kung-fu'), ) .addPhrase((phrase) => phrase .setMetadata({ originalWord: 'bitch', severity: 'high' }) .addPattern(pattern`bitch`) .addPattern(pattern`bich|`), ); const matcher = new RegExpMatcher({ ...dataset.build(), ...englishRecommendedTransformers, }); const matches = matcher.getAllMatches('fck you bitch', true); for (const match of matches) { const { phraseMetadata, startIndex, endIndex } = dataset.getPayloadWithPhraseMetadata(match); console.log(`[${phraseMetadata?.severity}] "${phraseMetadata?.originalWord}" at ${startIndex}-${endIndex}`); } // [high] "fuck" at 0-2 // [high] "bitch" at 7-11 ``` -------------------------------- ### Basic Text Censoring Source: https://github.com/jo3-l/obscenity/blob/main/docs/guide/censoring-text.md Demonstrates the default censoring behavior using TextCensor and RegExpMatcher. The offending content is replaced with grawlix by default. ```typescript import { TextCensor, RegExpMatcher, englishDataset, englishRecommendedTransformers } from 'obscenity'; const matcher = new RegExpMatcher({ ...englishDataset.build(), ...englishRecommendedTransformers }); const censor = new TextCensor(); // (1) const text = 'f u c k you!'; const matches = matcher.getAllMatches(text); console.log(censor.applyTo(text, matches)); // (2) //> "@$** you!" ``` -------------------------------- ### StatefulTransformer Interface Source: https://github.com/jo3-l/obscenity/blob/main/docs/reference/interfaces/StatefulTransformer.md This interface outlines the methods and properties that stateful transformers must implement. It includes a `transform` method for character processing and a `reset` method for state management. ```APIDOC ## Interface: StatefulTransformer An interface that stateful transformers should implement. ### Properties #### transform > **transform**: [`TransformerFn`](../type-aliases/TransformerFn.md) Transforms input characters. ##### Param Input character. ##### Returns The transformed character. A return value of `undefined` indicates that the character should be ignored. ### Methods #### reset() > **reset**(): `void` Resets the state of the transformer. ##### Returns `void` ```