### Decode Tokens to Text in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This Java example illustrates how to convert token IDs back into human-readable text using jtokkit's `decode` method. It covers decoding a manually created list of tokens, decoding tokens obtained from an `encode` operation for a round-trip verification, and demonstrates proper handling of Unicode characters. The example also includes error handling for invalid token IDs, which will throw an `IllegalArgumentException`.

```java
import java.util.Arrays;

Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Decode token list
IntArrayList tokens = new IntArrayList();
tokens.add(15339);
tokens.add(1917);
String decoded = enc.decode(tokens);
System.out.println("Decoded text: " + decoded); // "hello world"

// Decode from encode result
IntArrayList encoded = enc.encode("This is a sample sentence.");
String original = enc.decode(encoded);
System.out.println("Round-trip text: " + original); // "This is a sample sentence."

// Decode handles unicode properly
IntArrayList unicodeTokens = enc.encode("Unicode: 你好🌍");
String unicodeDecoded = enc.decode(unicodeTokens);
System.out.println("Unicode decoded: " + unicodeDecoded); // "Unicode: 你好🌍"

// Invalid token IDs throw IllegalArgumentException
IntArrayList invalidTokens = new IntArrayList();
invalidTokens.add(15339);
invalidTokens.add(Integer.MAX_VALUE);
try {
    enc.decode(invalidTokens);
} catch (IllegalArgumentException e) {
    System.out.println("Error: Invalid token ID");
}

```

--------------------------------

### Get OpenAI Model Encoding in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This Java code shows how to automatically retrieve the correct JTokkit encoding for various OpenAI models using the EncodingRegistry. It includes examples for GPT-4, GPT-4o, GPT-3.5-Turbo, and text-embedding models, along with their context lengths and support for string names and version snapshots.

```java
import com.knuddels.jtokkit.api.ModelType;

// Get encoding for GPT-4
Encoding gpt4Encoding = registry.getEncodingForModel(ModelType.GPT_4);
System.out.println("GPT-4 uses: " + gpt4Encoding.getName()); // cl100k_base
System.out.println("Max context length: " + ModelType.GPT_4.getMaxContextLength()); // 8192

// Get encoding for GPT-4o
Encoding gpt4oEncoding = registry.getEncodingForModel(ModelType.GPT_4O);
System.out.println("GPT-4o uses: " + gpt4oEncoding.getName()); // o200k_base

// Get encoding for GPT-3.5-Turbo
Encoding gpt35Encoding = registry.getEncodingForModel(ModelType.GPT_3_5_TURBO);

// Get encoding for text-embedding models
Encoding adaEncoding = registry.getEncodingForModel(ModelType.TEXT_EMBEDDING_ADA_002);

// Get encoding for model by string name (supports version snapshots)
java.util.Optional<Encoding> gpt4Snapshot = registry.getEncodingForModel("gpt-4-0314");
java.util.Optional<Encoding> gpt35Snapshot = registry.getEncodingForModel("gpt-3.5-turbo-0301");
```

--------------------------------

### Efficient Token Storage with IntArrayList in JTokkit

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This example showcases the usage of JTokkit's custom `IntArrayList` for efficient storage and manipulation of token IDs. It covers creating, adding, accessing, modifying, and converting the `IntArrayList` to standard Java arrays or boxed lists. It also demonstrates pre-allocating capacity for performance.

```java
import com.knuddels.jtokkit.api.IntArrayList;

// Create new IntArrayList
IntArrayList tokens = new IntArrayList();
tokens.add(100);
tokens.add(200);
tokens.add(300);

System.out.println("Size: " + tokens.size()); // 3
System.out.println("First token: " + tokens.get(0)); // 100
System.out.println("Is empty: " + tokens.isEmpty()); // false

// Set value at index
int oldValue = tokens.set(1, 250);
System.out.println("Old value: " + oldValue); // 200
System.out.println("New value: " + tokens.get(1)); // 250

// Convert to standard Java array
int[] array = tokens.toArray();
System.out.println("Array: " + Arrays.toString(array)); // [100, 250, 300]

// Convert to boxed List<Integer>
java.util.List<Integer> boxedList = tokens.boxed();
System.out.println("Boxed list: " + boxedList); // [100, 250, 300]

// Create with initial capacity for performance
IntArrayList largeTokens = new IntArrayList(1000);
for (int i = 0; i < 1000; i++) {
    largeTokens.add(i);
}

// Clear the list
tokens.clear();
System.out.println("After clear, size: " + tokens.size()); // 0
```

--------------------------------

### Register Custom Encoding Implementation in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

Implement the Encoding interface to create a custom tokenization logic. This example demonstrates a simple character-based encoding and its registration with JTokkit's EncodingRegistry. It is designed to be thread-safe.

```java
import com.knuddels.jtokkit.api.Encoding;
import com.knuddels.jtokkit.api.EncodingResult;
import com.knuddels.jtokkit.api.IntArrayList;

// Custom encoding implementation (thread-safe)
class SimpleCharEncoding implements Encoding {
    @Override
    public IntArrayList encode(String text) {
        IntArrayList tokens = new IntArrayList();
        for (char c : text.toCharArray()) {
            tokens.add((int) c);
        }
        return tokens;
    }

    @Override
    public EncodingResult encode(String text, int maxTokens) {
        IntArrayList tokens = new IntArrayList();
        int count = 0;
        for (char c : text.toCharArray()) {
            if (count >= maxTokens) break;
            tokens.add((int) c);
            count++;
        }
        return new EncodingResult(tokens, count < text.length());
    }

    @Override
    public IntArrayList encodeOrdinary(String text) {
        return encode(text);
    }

    @Override
    public EncodingResult encodeOrdinary(String text, int maxTokens) {
        return encode(text, maxTokens);
    }

    @Override
    public int countTokens(String text) {
        return text.length();
    }

    @Override
    public int countTokensOrdinary(String text) {
        return text.length();
    }

    @Override
    public String decode(IntArrayList tokens) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < tokens.size(); i++) {
            sb.append((char) tokens.get(i));
        }
        return sb.toString();
    }

    @Override
    public byte[] decodeBytes(IntArrayList tokens) {
        return decode(tokens).getBytes(java.nio.charset.StandardCharsets.UTF_8);
    }

    @Override
    public String getName() {
        return "simple-char-encoding";
    }
}

// Register custom implementation
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding customEncoding = new SimpleCharEncoding();
registry.registerCustomEncoding(customEncoding);

// Use custom encoding
java.util.Optional<Encoding> retrieved = registry.getEncoding("simple-char-encoding");
if (retrieved.isPresent()) {
    Encoding enc = retrieved.get();
    IntArrayList tokens = enc.encode("ABC");
    System.out.println("Custom tokens: " + tokens); // [65, 66, 67]
    String decoded = enc.decode(tokens);
    System.out.println("Decoded: " + decoded); // "ABC"
}
```

--------------------------------

### Get Encoding for Specific Models

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

Automatically retrieve the correct encoding for various OpenAI models based on their type or string name.

```APIDOC
## Get Encoding for Specific Models

### Description
Retrieve the appropriate encoding for a specific OpenAI model automatically.

### Method
N/A (Retrieval)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```java
import com.knuddels.jtokkit.api.ModelType;

// Assuming 'registry' is an initialized EncodingRegistry instance
// EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();

// Get encoding for GPT-4
Encoding gpt4Encoding = registry.getEncodingForModel(ModelType.GPT_4);
System.out.println("GPT-4 uses: " + gpt4Encoding.getName()); // cl100k_base
System.out.println("Max context length: " + ModelType.GPT_4.getMaxContextLength()); // 8192

// Get encoding for GPT-4o
Encoding gpt4oEncoding = registry.getEncodingForModel(ModelType.GPT_4O);
System.out.println("GPT-4o uses: " + gpt4oEncoding.getName()); // o200k_base

// Get encoding for GPT-3.5-Turbo
Encoding gpt35Encoding = registry.getEncodingForModel(ModelType.GPT_3_5_TURBO);

// Get encoding for text-embedding models
Encoding adaEncoding = registry.getEncodingForModel(ModelType.TEXT_EMBEDDING_ADA_002);

// Get encoding for model by string name (supports version snapshots)
java.util.Optional<Encoding> gpt4Snapshot = registry.getEncodingForModel("gpt-4-0314");
java.util.Optional<Encoding> gpt35Snapshot = registry.getEncodingForModel("gpt-3.5-turbo-0301");
```

### Response
#### Success Response (N/A)
N/A

#### Response Example
N/A
```

--------------------------------

### Encode Text to Tokens in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This Java snippet demonstrates encoding text strings into token IDs using JTokkit. It covers basic encoding, handling complex text with special characters and Unicode, and shows how to get the token count.

```java
import com.knuddels.jtokkit.api.IntArrayList;

Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Basic encoding
IntArrayList tokens = enc.encode("This is a sample sentence.");
System.out.println("Tokens: " + tokens); // [2028, 374, 264, 6205, 11914, 13]
System.out.println("Token count: " + tokens.size()); // 6

// Encoding with complex text
String complexText = "Many words map to one token, but some don't: indivisible.\n\n" +
                     "Unicode characters like emojis may be split into many tokens: 🤚🏾\n" +
                     "Sequences commonly found together may be grouped: 1234567890";
IntArrayList complexTokens = enc.encode(complexText);
System.out.println("Complex text token count: " + complexTokens.size());

// Encoding handles unicode properly
IntArrayList unicodeTokens = enc.encode("Mixed script: 你好 world! 🌍");
System.out.println("Unicode tokens: " + unicodeTokens);
```

--------------------------------

### Get Encoding from Registry in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/usage.md

Demonstrates various methods to retrieve an Encoding object from an EncodingRegistry. Encodings can be obtained using type-safe enums (EncodingType, ModelType), string names, or by specifying a model.

```java
// Get encoding via type-safe enum
Encoding encoding = registry.getEncoding(EncodingType.CL100K_BASE);
```

```java
// Get encoding via string name
Optional<Encoding> encoding = registry.getEncoding("cl100k_base");
```

```java
// Get encoding for a specific model via type-safe enum
Encoding encoding = registry.getEncodingForModel(ModelType.GPT_4);
```

```java
// Get encoding for a specific model via string name
Optional<Encoding> encoding = registry.getEncodingForModel("gpt_4");
```

--------------------------------

### Encode Text with Token Limit and Truncation in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This example demonstrates how to encode text using jtokkit with a specified maximum token limit. The `encode` method automatically truncates the text to fit within the limit while respecting character boundaries to avoid corrupting multi-byte characters like emojis. It returns an `EncodingResult` object containing the tokens, a flag indicating truncation, and the index of the last processed character.

```java
import com.knuddels.jtokkit.api.EncodingResult;

Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Encode with max token limit
EncodingResult result = enc.encode("This is a sample sentence.", 3);
System.out.println("Tokens: " + result.getTokens()); // [2028, 374, 264]
System.out.println("Was truncated: " + result.isTruncated()); // true
System.out.println("Last processed char index: " + result.getLastProcessedCharacterIndex());

String decoded = enc.decode(result.getTokens());
System.out.println("Decoded: " + decoded); // "This is a"

// Truncation respects character boundaries (emoji example)
EncodingResult emojiResult = enc.encode("I love 🍕", 4);
System.out.println("Tokens: " + emojiResult.getTokens()); // [40, 3021]
System.out.println("Was truncated: " + emojiResult.isTruncated()); // true

// The emoji is removed entirely rather than partially encoded
String emojiDecoded = enc.decode(emojiResult.getTokens());
System.out.println("Decoded: " + emojiDecoded); // "I love"

// Use encodeOrdinary variant with truncation for special tokens
EncodingResult ordinaryResult = enc.encodeOrdinary("hello <|endoftext|> world", 5);
System.out.println("Ordinary truncated tokens: " + ordinaryResult.getTokens());
System.out.println("Was truncated: " + ordinaryResult.isTruncated());

```

--------------------------------

### Count ChatML Message Tokens in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/recipes/chatml.md

Calculates the token count for a list of chat messages formatted according to ChatML. It uses the `jtokkit` library to get the appropriate encoding for the specified model and counts tokens for roles, content, names, and ChatML structural elements. This method is crucial for estimating API costs and managing token limits.

```java
private int countMessageTokens(
		EncodingRegistry registry,
		String model,
		List<ChatMessage> messages // consists of role, content and an optional name
) {
	Encoding encoding = registry.getEncodingForModel(model).orElseThrow();
	int tokensPerMessage;
	int tokensPerName
	if (model.startsWith("gpt-4")) {
		tokensPerMessage = 3;
		tokensPerName = 1;
	} else if (model.startsWith("gpt-3.5-turbo")) {
		tokensPerMessage = 4; // every message follows <|start|>{role/name}\n{content}<|end|>\n
		tokensPerName = -1; // if there's a name, the role is omitted
	} else {
		throw new IllegalArgumentException("Unsupported model: " + model);
	}

	int sum = 0;
	for (final var message : messages) {
		sum += tokensPerMessage;
		sum += encoding.countTokens(message.getContent());
		sum += encoding.countTokens(message.getRole());
		if (message.hasName()) {
			sum += encoding.countTokens(message.getName());
			sum += tokensPerName;
		}
	}

	sum += 3; // every reply is primed with <|start|>assistant<|message|>

	return sum;
}
```

--------------------------------

### Gradle Dependency for JTokkit

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/intro.md

Include this Groovy snippet in your Gradle project's build.gradle file to integrate the JTokkit library. This will make JTokkit's tokenization features available for your Java or Kotlin projects.

```groovy
dependencies {
	implementation 'com.knuddels:jtokkit:1.1.0'
}
```

--------------------------------

### Initialize Encoding Registry in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This snippet demonstrates how to initialize an EncodingRegistry in Java using JTokkit. It shows creating both default and lazy registries, and retrieving specific encodings by type or string name.

```java
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.api.EncodingRegistry;
import com.knuddels.jtokkit.api.EncodingType;
import com.knuddels.jtokkit.api.Encoding;

// Create default registry with all encodings pre-loaded
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();

// Or create lazy registry that loads encodings on first access
EncodingRegistry lazyRegistry = Encodings.newLazyEncodingRegistry();

// Get encoding by type
Encoding cl100k = registry.getEncoding(EncodingType.CL100K_BASE);
Encoding o200k = registry.getEncoding(EncodingType.O200K_BASE);
Encoding p50k = registry.getEncoding(EncodingType.P50K_BASE);
Encoding r50k = registry.getEncoding(EncodingType.R50K_BASE);

// Get encoding by string name
java.util.Optional<Encoding> optionalEncoding = registry.getEncoding("cl100k_base");
if (optionalEncoding.isPresent()) {
    Encoding encoding = optionalEncoding.get();
    System.out.println("Encoding name: " + encoding.getName());
}
```

--------------------------------

### Encoding Registry Management

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

Learn how to initialize and manage encoding registries, which are responsible for caching and providing access to different tokenization algorithms.

```APIDOC
## Encoding Registry Management

### Description
Initialize a registry that manages and caches encoding instances for different tokenization algorithms.

### Method
N/A (Initialization and retrieval)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```java
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.api.EncodingRegistry;
import com.knuddels.jtokkit.api.EncodingType;
import com.knuddels.jtokkit.api.Encoding;

// Create default registry with all encodings pre-loaded
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();

// Or create lazy registry that loads encodings on first access
EncodingRegistry lazyRegistry = Encodings.newLazyEncodingRegistry();

// Get encoding by type
Encoding cl100k = registry.getEncoding(EncodingType.CL100K_BASE);
Encoding o200k = registry.getEncoding(EncodingType.O200K_BASE);
Encoding p50k = registry.getEncoding(EncodingType.P50K_BASE);
Encoding r50k = registry.getEncoding(EncodingType.R50K_BASE);

// Get encoding by string name
java.util.Optional<Encoding> optionalEncoding = registry.getEncoding("cl100k_base");
if (optionalEncoding.isPresent()) {
    Encoding encoding = optionalEncoding.get();
    System.out.println("Encoding name: " + encoding.getName());
}
```

### Response
#### Success Response (N/A)
N/A

#### Response Example
N/A
```

--------------------------------

### Maven Dependency for JTokkit

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/intro.md

Add this XML snippet to your Maven project's pom.xml file to include the JTokkit library. This dependency allows you to use JTokkit's tokenization capabilities in your Java applications.

```xml
<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>
```

--------------------------------

### Create Default Encoding Registry in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/usage.md

Creates a default encoding registry that loads all vocabularies from the classpath upon creation. This registry is thread-safe and caches loaded encodings, making subsequent accesses efficient. Keep a reference to the registry as its creation is resource-intensive.

```java
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
```

--------------------------------

### Add New Byte Pair Encoding with Parameters in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/extending.md

This snippet demonstrates how to add a new byte pair encoding to JTokkit by providing necessary parameters. It includes creating `GptBytePairEncodingParams` with a name, pattern, encoding map, and special token encoding map, then registering it with the `EncodingRegistry`. Finally, it shows how to retrieve the newly added encoding.

```java
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
GptBytePairEncodingParams params = new GptBytePairEncodingParams(
        "custom-name",
        Pattern.compile("some custom pattern"),
        encodingMap,
        specialTokenEncodingMap
);
registry.registerGptBytePairEncoding(params);

// Get the encoding from the registry
Encoding encodingFromRegistry = registry.getEncoding("custom-name");
```

--------------------------------

### Decode Token IDs to Bytes with JTokkit

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This snippet demonstrates how to decode token IDs directly into byte arrays using the JTokkit library. It shows the conversion of token IDs to bytes and then to a UTF-8 string. It also illustrates handling binary data correctly.

```java
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Decode to bytes
IntArrayList tokens = new IntArrayList();
tokens.add(15339);
tokens.add(1917);
byte[] bytes = enc.decodeBytes(tokens);
System.out.println("Byte array: " + Arrays.toString(bytes));
// [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]

// Convert bytes to string manually if needed
String text = new String(bytes, java.nio.charset.StandardCharsets.UTF_8);
System.out.println("Text from bytes: " + text); // "hello world"

// Decode handles binary data correctly
IntArrayList encodedData = enc.encode("Binary: \u0000\u0001\u0002");
byte[] binaryBytes = enc.decodeBytes(encodedData);
System.out.println("Binary byte count: " + binaryBytes.length);
```

--------------------------------

### Create Lazy Loading Encoding Registry in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/usage.md

Creates an encoding registry that lazily loads vocabularies only when they are accessed for the first time. This approach is suitable when not all encodings are immediately needed, saving initial resource overhead. The registry is thread-safe and caches vocabularies after their first load.

```java
EncodingRegistry registry = Encodings.newLazyEncodingRegistry();
```

--------------------------------

### Implement Encoding Interface and Register in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/extending.md

This snippet shows how to implement the `Encoding` interface, create a custom encoding, and register it with the `EncodingRegistry`. It also demonstrates retrieving the registered encoding by its unique name. Ensure the custom encoding name is unique and the implementation is thread-safe.

```java
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding customEncoding = new CustomEncoding();
registry.register(customEncoding);

// Get the encoding from the registry
Encoding encodingFromRegistry = registry.getEncoding("custom-name");
```

--------------------------------

### Count Tokens Efficiently in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This Java snippet demonstrates efficient token counting using jtokkit's `countTokens` and `countTokensOrdinary` methods. These methods are faster than full encoding as they do not generate the token list, making them ideal for quick checks, such as validating token limits before sending requests to an API. It also shows how to verify the count against the `encode` method's result.

```java
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Count tokens (faster than encode)
int tokenCount = enc.countTokens("This is a sample sentence.");
System.out.println("Token count: " + tokenCount); // 6

// Count tokens for large text
String largeText = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. " +
                   "Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.";
int largeTokenCount = enc.countTokens(largeText);
System.out.println("Large text token count: " + largeTokenCount);

// Count ordinary tokens (handles special tokens)
int ordinaryCount = enc.countTokensOrdinary("hello <|endoftext|> world");
System.out.println("Ordinary token count: " + ordinaryCount); // 8

// Verify token count matches encode result
IntArrayList tokens = enc.encode("Test message");
assert tokens.size() == enc.countTokens("Test message");

// Use for checking token limits before API calls
String userPrompt = "Write a detailed essay about artificial intelligence.";
int promptTokens = enc.countTokens(userPrompt);
int maxCompletionTokens = 4096 - promptTokens;
System.out.println("Available tokens for completion: " + maxCompletionTokens);

```

--------------------------------

### Encode and Decode Text in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/usage.md

Shows how to encode a string into a list of integer token IDs and decode a list of token IDs back into a string using an Encoding object. The encoding process is thread-safe.

```java
IntArrayList encoded = encoding.encode("This is a sample sentence.");
// encoded = [2028, 374, 264, 6205, 11914, 13]
```

```java
String decoded = encoding.decode(encoded);
// decoded = "This is a sample sentence."
```

--------------------------------

### Register Custom BPE Encoding Parameters in JTokkit Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/README.md

This Java snippet shows how to add new parameters for the existing BPE algorithm in JTokkit. It involves creating a GptBytePairEncodingParams object with custom configurations and registering it with the EncodingRegistry. This enables using the BPE algorithm with custom patterns and dictionaries.

```java
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.EncodingRegistry;
import com.knuddels.jtokkit.gptBpe.GptBytePairEncodingParams;
import java.util.regex.Pattern;
import java.util.Map;

// Assume encodingMap and specialTokenEncodingMap are properly initialized Maps
// Map<Integer, String> encodingMap = ...;
// Map<String, Integer> specialTokenEncodingMap = ...;

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
GptBytePairEncodingParams params = new GptBytePairEncodingParams(
        "custom-name",
        Pattern.compile("some custom pattern"),
        encodingMap,
        specialTokenEncodingMap
);
registry.registerGptBytePairEncoding(params);

```

--------------------------------

### Initialize and Use Default Encoding Registry in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/README.md

This snippet demonstrates how to initialize the default EncodingRegistry and retrieve an encoding for a specific type (CL100K_BASE) or model (TEXT_EMBEDDING_ADA_002). It also shows basic encoding and decoding operations. The EncodingRegistry and Encoding classes are thread-safe.

```java
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.EncodingRegistry;
import com.knuddels.jtokkit.EncodingType;
import com.knuddels.jtokkit.ModelType;
import com.knuddels.jtokkit.Encoding;
import com.knuddels.jtokkit.intarray.IntArrayList;

// ...

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Example of encoding and decoding
IntArrayList encoded = enc.encode("This is a sample sentence.");
// encoded = [2028, 374, 264, 6205, 11914, 13]
String decoded = enc.decode(encoded);
// decoded = "This is a sample sentence."

// Or get the tokenizer based on the model type
Encoding secondEnc = registry.getEncodingForModel(ModelType.TEXT_EMBEDDING_ADA_002);
// enc == secondEnc

// Initial example from README
Encoding encReadme = registry.getEncoding(EncodingType.CL100K_BASE);
assert "hello world".equals(encReadme.decode(encReadme.encode("hello world")));

// Or get the tokenizer corresponding to a specific OpenAI model
Encoding encModel = registry.getEncodingForModel(ModelType.TEXT_EMBEDDING_ADA_002);

```

--------------------------------

### Encode Text with Special Tokens in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/usage.md

Illustrates how to handle special tokens during text encoding. The standard `encode` method throws an exception for special tokens, while `encodeOrdinary` treats them as regular text. This is useful for specific model functionalities like fill-in-the-middle.

```java
encoding.encode("hello <|endoftext|> world");
// raises an UnsupportedOperationException
```

```java
encoding.encodeOrdinary("hello <|endoftext|> world");
// returns [15339, 83739, 8862, 728, 428, 91, 29, 1917]
```

--------------------------------

### Encode Text to Tokens

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

Convert text strings into lists of token IDs, which is essential for preparing data for language models and counting tokens.

```APIDOC
## Encode Text to Tokens

### Description
Convert text strings into lists of token IDs for processing by language models.

### Method
N/A (Encoding)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```java
import com.knuddels.jtokkit.api.IntArrayList;

// Assuming 'registry' is an initialized EncodingRegistry instance
// EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Basic encoding
IntArrayList tokens = enc.encode("This is a sample sentence.");
System.out.println("Tokens: " + tokens); // [2028, 374, 264, 6205, 11914, 13]
System.out.println("Token count: " + tokens.size()); // 6

// Encoding with complex text
String complexText = "Many words map to one token, but some don't: indivisible.\n\n" +
                     "Unicode characters like emojis may be split into many tokens: 🤚🏾\n" +
                     "Sequences commonly found together may be grouped: 1234567890";
IntArrayList complexTokens = enc.encode(complexText);
System.out.println("Complex text token count: " + complexTokens.size());

// Encoding handles unicode properly
IntArrayList unicodeTokens = enc.encode("Mixed script: 你好 world! 🌍");
System.out.println("Unicode tokens: " + unicodeTokens);
```

### Response
#### Success Response (N/A)
N/A

#### Response Example
N/A
```

--------------------------------

### Register Custom Byte Pair Encoding in JTokkit

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This Java snippet illustrates how to extend JTokkit by registering a custom byte pair encoding. It involves creating `GptBytePairEncodingParams` with custom token mappings and a regex pattern, then registering it with the `EncodingRegistry` for subsequent use.

```java
import com.knuddels.jtokkit.api.GptBytePairEncodingParams;
import java.util.regex.Pattern;
import java.util.HashMap;
import java.util.Map;

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();

// Create encoding parameters
Map<byte[], Integer> encodingMap = new HashMap<>();
encodingMap.put("hello".getBytes(java.nio.charset.StandardCharsets.UTF_8), 1000);
encodingMap.put("world".getBytes(java.nio.charset.StandardCharsets.UTF_8), 1001);
encodingMap.put(" ".getBytes(java.nio.charset.StandardCharsets.UTF_8), 1002);

Map<String, Integer> specialTokensMap = new HashMap<>();
specialTokensMap.put("<|custom|>"), 9999);

GptBytePairEncodingParams params = new GptBytePairEncodingParams(
    "custom-encoding",
    Pattern.compile("\\w+|\\s+|[^\\w\\s]+"),
    encodingMap,
    specialTokensMap
);

// Register the custom encoding
registry.registerGptBytePairEncoding(params);

// Use the custom encoding
java.util.Optional<Encoding> customEncoding = registry.getEncoding("custom-encoding");
if (customEncoding.isPresent()) {
    Encoding enc = customEncoding.get();
    System.out.println("Custom encoding registered: " + enc.getName());

    // Use it like any other encoding
    IntArrayList tokens = enc.encodeOrdinary("hello world");
    System.out.println("Custom tokens: " + tokens);
}
```

--------------------------------

### Register Custom Encoding in JTokkit Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/README.md

This Java snippet illustrates how to extend JTokkit by implementing the Encoding interface and registering a custom encoding with the EncodingRegistry. This allows using custom tokenization logic within the library.

```java
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.EncodingRegistry;
import com.knuddels.jtokkit.Encoding;

// Assume CustomEncoding is an implementation of the Encoding interface
// Encoding customEncoding = new CustomEncoding(); 

EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
// registry.registerEncoding(customEncoding);

```

--------------------------------

### Count Tokens in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/usage.md

Provides efficient methods for counting the number of tokens a given text will encode to, without performing the actual encoding. `countTokens` is for standard encoding, while `countTokensOrdinary` handles special tokens as regular text.

```java
int tokenCount = encoding.countTokens("This is a sample sentence.");
// tokenCount = 6
```

```java
int tokenCount = encoding.countTokensOrdinary("hello <|endoftext|> world");
// tokenCount = 8
```

--------------------------------

### Encode Special Tokens as Ordinary Text in Java

Source: https://context7.com/knuddelsgmbh/jtokkit/llms.txt

This snippet shows how to encode text containing special tokens like '<|endoftext|>' as regular text using jtokkit. It contrasts the default behavior, which throws an exception for special tokens, with the `encodeOrdinary` method that treats them as normal characters. This is useful when special tokens should not be interpreted as control characters.

```java
Encoding enc = registry.getEncoding(EncodingType.CL100K_BASE);

// Regular encode throws exception on special tokens
try {
    IntArrayList tokens = enc.encode("hello <|endoftext|> world");
} catch (UnsupportedOperationException e) {
    System.out.println("Error: Special tokens not supported in encode()");
}

// Use encodeOrdinary to treat special tokens as normal text
IntArrayList ordinaryTokens = enc.encodeOrdinary("hello <|endoftext|> world");
System.out.println("Ordinary encoding: " + ordinaryTokens);
// [15339, 83739, 8862, 728, 428, 91, 29, 1917]

String decoded = enc.decode(ordinaryTokens);
System.out.println("Decoded: " + decoded); // "hello <|endoftext|> world"

```

--------------------------------

### Encode Text with Truncation in Java

Source: https://github.com/knuddelsgmbh/jtokkit/blob/main/docs/docs/getting-started/usage.md

Enables encoding text up to a specified maximum number of tokens. If the text exceeds `maxTokens`, it is truncated. The methods also handle incomplete Unicode characters caused by truncation by removing the corresponding tokens.

```java
IntArrayList encoded = encoding.encode("This is a sample sentence.", 3);
// encoded = [2028, 374, 264]
```

```java
String decoded = encoding.decode(encoded);
// decoded = "This is a"
```

```java
IntArrayList encoded = encoding.encode("I love 🍕", 4);
// encoded = [40, 3021]
```

```java
String decoded = encoding.decode(encoded);
// decoded = "I love"
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.