tokenizers.bpe0.1.3 package

Byte Pair Encoding Text Tokenization

bpe

Construct a Byte Pair Encoding model

bpe_decode

Decode Byte Pair Encoding sequences to text

bpe_encode

Tokenise text alongside a Byte Pair Encoding model

bpe_load_model

Load a Byte Pair Encoding model

Download source packageRead PDF manual

Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.