Home
Packages
Datasets
Task Views
R resources
Packages
Toggle theme
Toggle Menu
Home
Packages
rtiktoken
model_to_tokenizer
model_to_tokenizer function
Gets the name of the tokenizer used by a model
Copy
rtiktoken package
Read PDF manual
Maintainer: David Zimmermann-Kollenda
License: MIT + file LICENSE
Last published: 2025-04-14
Useful links
https://github.com/DavZim/rtiktoken/issues
https://davzim.github.io/rtiktoken/
https://github.com/DavZim/rtiktoken/