### Install tiktoken_ruby Gem Source: https://github.com/iapark/tiktoken_ruby/blob/main/README.md Instructions for installing the tiktoken_ruby gem using Bundler for dependency management or directly via the `gem install` command. ```sh bundle add tiktoken_ruby ``` ```sh gem install tiktoken_ruby ``` -------------------------------- ### Set Up tiktoken_ruby Development Environment Source: https://github.com/iapark/tiktoken_ruby/blob/main/README.md Commands to clone the tiktoken_ruby repository, navigate into the directory, install Ruby dependencies, compile native extensions, and run the test suite for local development. ```sh git clone https://github.com/IAPark/tiktoken_ruby.git cd tiktoken_ruby bundle install bundle exec rake compile bundle exec rake spec ``` -------------------------------- ### Encode and Decode Text with tiktoken_ruby Source: https://github.com/iapark/tiktoken_ruby/blob/main/README.md Demonstrates how to initialize a Tiktoken encoder for a specific encoding (cl100k_base) and use it to encode a string into tokens, then decode the tokens back to the original string. ```ruby require 'tiktoken_ruby' enc = Tiktoken.get_encoding("cl100k_base") enc.decode(enc.encode("hello world")) #=> "hello world" ``` -------------------------------- ### Retrieve Tiktoken Encoder by Model Name Source: https://github.com/iapark/tiktoken_ruby/blob/main/README.md Shows how to obtain a Tiktoken encoder instance by specifying a GPT model name (e.g., 'gpt-4') and then use it to encode a string to determine its token count. ```ruby require 'tiktoken_ruby' enc = Tiktoken.encoding_for_model("gpt-4") enc.encode("hello world").length #=> 2 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.