tiktoken is a fast Byte Pair Encoding (BPE) tokenizer for use with OpenAI's models, offering...

Tokens:36,866
Snippets:327
Trust Score:9.1
License:MIT
Update:2 weeks ago
Tokens:
Raw