JTokkit

https://jtokkit.knuddels.de/
Admin
JTokkit is a fast and efficient Java tokenizer library designed for use with OpenAI models,...
Tokens:2,234
Snippets:27
Trust Score:9.9
Update:5 months ago
Show doc for...
Tokens:
Raw
### Get Encoding from Registry

Source: https://jtokkit.knuddels.de/docs/getting-started/usage

Retrieve an Encoding object from the registry using either a type-safe enum or a string name. You can also get encodings specific to a model by its type-safe enum or string name.

```java
// Get encoding via type-safe enum  
Encoding encoding = registry.getEncoding(EncodingType.CL100K_BASE);
```

```java
// Get encoding via string name  
Optional<Encoding> encoding = registry.getEncoding("cl100k_base");
```

```java
// Get encoding for a specific model via type-safe enum  
Encoding encoding = registry.getEncodingForModel(ModelType.GPT_4);
```

```java
// Get encoding for a specific model via string name  
Optional<Encoding> encoding = registry.getEncodingForModel("gpt_4");
```

--------------------------------

### Create Default Encoding Registry

Source: https://jtokkit.knuddels.de/docs/getting-started/usage

Instantiate a new EncodingRegistry to manage encodings. This is an expensive operation as it loads vocabularies from the classpath. Keep a reference to the registry for caching and thread-safe access.

```java
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();
```

--------------------------------

### Implement and Register Custom Encoding

Source: https://jtokkit.knuddels.de/docs/getting-started/extending

Implement the `Encoding` interface for custom encoding logic and register it with the `EncodingRegistry`. Ensure the encoding name is unique and the implementation is thread-safe.

```java
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();  
Encoding customEncoding = new CustomEncoding();  
registry.register(customEncoding);  
  
// Get the encoding from the registry  
Encoding encodingFromRegistry = registry.getEncoding("custom-name");  
```

--------------------------------

### Create Lazy Encoding Registry

Source: https://jtokkit.knuddels.de/docs/getting-started/usage

Instantiate a lazy EncodingRegistry that loads vocabularies only when they are accessed. This can be more efficient if not all encodings are used. The registry is thread-safe and caches vocabularies after first access.

```java
EncodingRegistry registry = Encodings.newLazyEncodingRegistry();
```

--------------------------------

### Gradle Dependency for JTokkit

Source: https://jtokkit.knuddels.de/docs/getting-started

Add this dependency to your Gradle project to include JTokkit.

```gradle
dependencies {
	implementation 'com.knuddels:jtokkit:1.1.0'
}
```

--------------------------------

### Maven Dependency for JTokkit

Source: https://jtokkit.knuddels.de/docs/getting-started

Add this dependency to your Maven project to include JTokkit.

```xml
<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>
```

--------------------------------

### Add New Byte Pair Encoding

Source: https://jtokkit.knuddels.de/docs/getting-started/extending

Add a new byte pair encoding by providing necessary parameters like name, pattern, and token maps to `GptBytePairEncodingParams`. Register this new encoding with the `EncodingRegistry`.

```java
EncodingRegistry registry = Encodings.newDefaultEncodingRegistry();  
GptBytePairEncodingParams params = new GptBytePairEncodingParams(  
        "custom-name",  
        Pattern.compile("some custom pattern"),  
        encodingMap,  
        specialTokenEncodingMap  
);
registry.registerGptBytePairEncoding(params);  
  
// Get the encoding from the registry  
Encoding encodingFromRegistry = registry.getEncoding("custom-name");  
```

--------------------------------

### Encode and Decode Text

Source: https://jtokkit.knuddels.de/docs/getting-started/usage

Use an Encoding object to convert text into a list of token integers and back. The encoding process is thread-safe. Note that special tokens are not supported by default encode.

```java
IntArrayList encoded = encoding.encode("This is a sample sentence.");  
// encoded = [2028, 374, 264, 6205, 11914, 13]  

String decoded = encoding.decode(encoded);  
// decoded = "This is a sample sentence."
```

--------------------------------

### Count ChatML Message Tokens in Java

Source: https://jtokkit.knuddels.de/docs/getting-started/recipes/chatml

Use this method to calculate the token count for messages formatted in ChatML, considering model-specific token overheads for messages and names. Ensure you have the EncodingRegistry and a list of ChatMessage objects.

```java
private int countMessageTokens(  
		EncodingRegistry registry,  
		String model,  
		List<ChatMessage> messages // consists of role, content and an optional name  
) {  
	Encoding encoding = registry.getEncodingForModel(model).orElseThrow();  
	int tokensPerMessage;  
	int tokensPerName  
	if (model.startsWith("gpt-4")) {  
		tokensPerMessage = 3;  
		tokensPerName = 1;  
	} else if (model.startsWith("gpt-3.5-turbo")) {  
		tokensPerMessage = 4; // every message follows <|start|>{role/name}\n{content}<|end|>\n  
		tokensPerName = -1; // if there's a name, the role is omitted  
	} else {  
		throw new IllegalArgumentException("Unsupported model: " + model);  
	}  
  
	int sum = 0;  
	for (final var message : messages) {  
		sum += tokensPerMessage;  
		sum += encoding.countTokens(message.getContent());  
		sum += encoding.countTokens(message.getRole());  
		if (message.hasName()) {  
			sum += encoding.countTokens(message.getName());  
			sum += tokensPerName;  
		}  
	}  
  
	sum += 3; // every reply is primed with <|start|>assistant<|message|>  
  
	return sum;  
}
```

--------------------------------

### Encode Text with Special Tokens

Source: https://jtokkit.knuddels.de/docs/getting-started/usage

When encountering special tokens like <|endoftext|>, the standard `encode` method throws an exception. Use `encodeOrdinary` to treat special tokens as regular text.

```java
encoding.encode("hello <|endoftext|> world");  
// raises an UnsupportedOperationException  

encoding.encodeOrdinary("hello <|endoftext|> world");  
// returns [15339, 83739, 8862, 728, 428, 91, 29, 1917]
```

--------------------------------

### Count Tokens in Text

Source: https://jtokkit.knuddels.de/docs/getting-started/usage

Efficiently count the number of tokens a given text will encode to without performing the full encoding. Use `countTokens` for standard text and `countTokensOrdinary` for text including special tokens.

```java
int tokenCount = encoding.countTokens("This is a sample sentence.");  
// tokenCount = 6  

int tokenCount = encoding.countTokensOrdinary("hello <|endoftext|> world");  
// tokenCount = 8
```

--------------------------------

### Encode Text with Truncation

Source: https://jtokkit.knuddels.de/docs/getting-started/usage

Encode text up to a specified maximum number of tokens using `encode(String, int)`. This method truncates the token list and handles split Unicode characters. The decoded result reflects the truncated text.

```java
IntArrayList encoded = encoding.encode("This is a sample sentence.", 3);  
// encoded = [2028, 374, 264]  

String decoded = encoding.decode(encoded);  
// decoded = "This is a"
```

```java
IntArrayList encoded = encoding.encode("I love 🍕", 4);  
// encoded = [40, 3021]  

String decoded = encoding.decode(encoded);  
// decoded = "I love"
```
Tokens:
Raw