rtiktoken0.0.7 package

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

decode_tokens

Decodes tokens back to text

get_token_count

Returns the number of tokens in a text

get_tokens

Converts text to tokens

model_to_tokenizer

Gets the name of the tokenizer used by a model

Download source packageRead PDF manual

A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.

  • Maintainer: David Zimmermann-Kollenda
  • License: MIT + file LICENSE
  • Last published: 2025-04-14

Useful links