LLamaTokenType
Namespace: LLama.Native
Token Types
1 | |
Inheritance Object → ValueType → Enum → LLamaTokenType
Implements IComparable, IFormattable, IConvertible
Remarks:
C# equivalent of llama_token_get_type
Fields
| Name | Value | Description |
|---|---|---|
| LLAMA_TOKEN_TYPE_UNDEFINED | 0 | No specific type has been set for this token |
| LLAMA_TOKEN_TYPE_NORMAL | 1 | This is a "normal" token |
| LLAMA_TOKEN_TYPE_UNKNOWN | 2 | An "unknown" character/text token e.g. <unk> |
| LLAMA_TOKEN_TYPE_CONTROL | 3 | A special control token e.g. </s> |