### Get Encoding from Registry Source: https://jtokkit.knuddels.de/docs/getting-started/usage Retrieve an Encoding object from the registry using either a type-safe enum or a string name. You can also get encodings specific to a model by its type-safe enum or string name. ```java // Get encoding via type-safe enum Encoding encoding = registry.getEncoding(EncodingType.CL100K_BASE); ``` ```java // Get encoding via string name Optional encoding = registry.getEncoding("cl100k_base"); ``` ```java // Get encoding for a specific model via type-safe enum Encoding encoding = registry.getEncodingForModel(ModelType.GPT_4); ``` ```java // Get encoding for a specific model via string name Optional encoding = registry.getEncodingForModel("gpt_4"); ``` -------------------------------- ### Create Default Encoding Registry Source: https://jtokkit.knuddels.de/docs/getting-started/usage Instantiate a new EncodingRegistry to manage encodings. This is an expensive operation as it loads vocabularies from the classpath. Keep a reference to the registry for caching and thread-safe access. ```java EncodingRegistry registry = Encodings.newDefaultEncodingRegistry(); ``` -------------------------------- ### Implement and Register Custom Encoding Source: https://jtokkit.knuddels.de/docs/getting-started/extending Implement the `Encoding` interface for custom encoding logic and register it with the `EncodingRegistry`. Ensure the encoding name is unique and the implementation is thread-safe. ```java EncodingRegistry registry = Encodings.newDefaultEncodingRegistry(); Encoding customEncoding = new CustomEncoding(); registry.register(customEncoding); // Get the encoding from the registry Encoding encodingFromRegistry = registry.getEncoding("custom-name"); ``` -------------------------------- ### Create Lazy Encoding Registry Source: https://jtokkit.knuddels.de/docs/getting-started/usage Instantiate a lazy EncodingRegistry that loads vocabularies only when they are accessed. This can be more efficient if not all encodings are used. The registry is thread-safe and caches vocabularies after first access. ```java EncodingRegistry registry = Encodings.newLazyEncodingRegistry(); ``` -------------------------------- ### Gradle Dependency for JTokkit Source: https://jtokkit.knuddels.de/docs/getting-started Add this dependency to your Gradle project to include JTokkit. ```gradle dependencies { implementation 'com.knuddels:jtokkit:1.1.0' } ``` -------------------------------- ### Maven Dependency for JTokkit Source: https://jtokkit.knuddels.de/docs/getting-started Add this dependency to your Maven project to include JTokkit. ```xml com.knuddels jtokkit 1.1.0 ``` -------------------------------- ### Add New Byte Pair Encoding Source: https://jtokkit.knuddels.de/docs/getting-started/extending Add a new byte pair encoding by providing necessary parameters like name, pattern, and token maps to `GptBytePairEncodingParams`. Register this new encoding with the `EncodingRegistry`. ```java EncodingRegistry registry = Encodings.newDefaultEncodingRegistry(); GptBytePairEncodingParams params = new GptBytePairEncodingParams( "custom-name", Pattern.compile("some custom pattern"), encodingMap, specialTokenEncodingMap ); registry.registerGptBytePairEncoding(params); // Get the encoding from the registry Encoding encodingFromRegistry = registry.getEncoding("custom-name"); ``` -------------------------------- ### Encode and Decode Text Source: https://jtokkit.knuddels.de/docs/getting-started/usage Use an Encoding object to convert text into a list of token integers and back. The encoding process is thread-safe. Note that special tokens are not supported by default encode. ```java IntArrayList encoded = encoding.encode("This is a sample sentence."); // encoded = [2028, 374, 264, 6205, 11914, 13] String decoded = encoding.decode(encoded); // decoded = "This is a sample sentence." ``` -------------------------------- ### Count ChatML Message Tokens in Java Source: https://jtokkit.knuddels.de/docs/getting-started/recipes/chatml Use this method to calculate the token count for messages formatted in ChatML, considering model-specific token overheads for messages and names. Ensure you have the EncodingRegistry and a list of ChatMessage objects. ```java private int countMessageTokens( EncodingRegistry registry, String model, List messages // consists of role, content and an optional name ) { Encoding encoding = registry.getEncodingForModel(model).orElseThrow(); int tokensPerMessage; int tokensPerName if (model.startsWith("gpt-4")) { tokensPerMessage = 3; tokensPerName = 1; } else if (model.startsWith("gpt-3.5-turbo")) { tokensPerMessage = 4; // every message follows <|start|>{role/name}\n{content}<|end|>\n tokensPerName = -1; // if there's a name, the role is omitted } else { throw new IllegalArgumentException("Unsupported model: " + model); } int sum = 0; for (final var message : messages) { sum += tokensPerMessage; sum += encoding.countTokens(message.getContent()); sum += encoding.countTokens(message.getRole()); if (message.hasName()) { sum += encoding.countTokens(message.getName()); sum += tokensPerName; } } sum += 3; // every reply is primed with <|start|>assistant<|message|> return sum; } ``` -------------------------------- ### Encode Text with Special Tokens Source: https://jtokkit.knuddels.de/docs/getting-started/usage When encountering special tokens like <|endoftext|>, the standard `encode` method throws an exception. Use `encodeOrdinary` to treat special tokens as regular text. ```java encoding.encode("hello <|endoftext|> world"); // raises an UnsupportedOperationException encoding.encodeOrdinary("hello <|endoftext|> world"); // returns [15339, 83739, 8862, 728, 428, 91, 29, 1917] ``` -------------------------------- ### Count Tokens in Text Source: https://jtokkit.knuddels.de/docs/getting-started/usage Efficiently count the number of tokens a given text will encode to without performing the full encoding. Use `countTokens` for standard text and `countTokensOrdinary` for text including special tokens. ```java int tokenCount = encoding.countTokens("This is a sample sentence."); // tokenCount = 6 int tokenCount = encoding.countTokensOrdinary("hello <|endoftext|> world"); // tokenCount = 8 ``` -------------------------------- ### Encode Text with Truncation Source: https://jtokkit.knuddels.de/docs/getting-started/usage Encode text up to a specified maximum number of tokens using `encode(String, int)`. This method truncates the token list and handles split Unicode characters. The decoded result reflects the truncated text. ```java IntArrayList encoded = encoding.encode("This is a sample sentence.", 3); // encoded = [2028, 374, 264] String decoded = encoding.decode(encoded); // decoded = "This is a" ``` ```java IntArrayList encoded = encoding.encode("I love 🍕", 4); // encoded = [40, 3021] String decoded = encoding.decode(encoded); // decoded = "I love" ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.