### List Available Benchmarks Source: https://github.com/aiqinxuancai/tiktokensharp/blob/master/README.md List all available benchmarks in a flat format. This is useful for identifying specific tests to run. ```bash dotnet run --project TiktokenSharp.Benchmark/TiktokenSharp.Benchmark.csproj --framework net10.0 -c Release -- --list flat ``` -------------------------------- ### Run Benchmark Tests Source: https://github.com/aiqinxuancai/tiktokensharp/blob/master/README.md Execute the benchmark tests for TiktokenSharp. Ensure you are in the project directory. ```bash dotnet run --project TiktokenSharp.Benchmark/TiktokenSharp.Benchmark.csproj --framework net10.0 -c Release ``` -------------------------------- ### C# Benchmark Code for TiktokenSharp and SharpToken Source: https://github.com/aiqinxuancai/tiktokensharp/blob/master/README.md This C# code defines benchmark methods for both TiktokenSharp and SharpToken. It measures the time and memory used for encoding and decoding a long text string 10,000 times. Use this code within a benchmark project. ```csharp private GptEncoding _sharpToken = GptEncoding.GetEncoding("cl100k_base"); private TikToken _tikToken = TikToken.GetEncoding("cl100k_base"); private string _kLongText = "King Lear, one of Shakespeare's darkest and most savage plays, tells the story of the foolish and Job-like Lear, who divides his kingdom, as he does his affections, according to vanity and whim. Lear’s failure as a father engulfs himself and his world in turmoil and tragedy."; [Benchmark] public int SharpToken() { var sum = 0; for (var i = 0; i < 10000; i++) { var encoded = _sharpToken.Encode(_kLongText); var decoded = _sharpToken.Decode(encoded); sum += decoded.Length; } return sum; } [Benchmark] public int TiktokenSharp() { var sum = 0; for (var i = 0; i < 10000; i++) { var encoded = _tikToken.Encode(_kLongText); var decoded = _tikToken.Decode(encoded); sum += decoded.Length; } return sum; } ``` -------------------------------- ### Encode and Decode Text with TiktokenSharp Source: https://github.com/aiqinxuancai/tiktokensharp/blob/master/README.md Demonstrates how to encode text into token IDs and decode token IDs back into text using model names or encoding names. The first use of an encoder will download necessary files. ```csharp using TiktokenSharp; //use model name TikToken tikToken = TikToken.EncodingForModel("gpt-3.5-turbo"); var i = tikToken.Encode("hello world"); //[15339, 1917] var d = tikToken.Decode(i); //hello world //use encoding name TikToken tikToken = TikToken.GetEncoding("cl100k_base"); var i = tikToken.Encode("hello world"); //[15339, 1917] var d = tikToken.Decode(i); //hello world ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.