LLamaVocabType
Namespace: LLama.Native
1 | |
Inheritance Object → ValueType → Enum → LLamaVocabType
Implements IComparable, ISpanFormattable, IFormattable, IConvertible
Remarks:
llama_vocab_type
Fields
| Name | Value | Description |
|---|---|---|
| None | 0 | For models without vocab |
| SentencePiece | 1 | LLaMA tokenizer based on byte-level BPE with byte fallback |
| BytePairEncoding | 2 | GPT-2 tokenizer based on byte-level BPE |
| WordPiece | 3 | BERT tokenizer based on WordPiece |
| Unigram | 4 | T5 tokenizer based on Unigram |
| RWKV | 5 | RWKV tokenizer based on greedy tokenization |