bpe_encode function

Tokenise text alongside a Byte Pair Encoding model

tokenizers.bpe package Read PDF manual

Maintainer: Jan Wijffels
License: MPL-2.0
Last published: 2023-09-15
https://github.com/bnosac/tokenizers.bpe