LLamaContext
Namespace: LLama
A llama_context, which holds all the context required to interact with a model
public sealed class LLamaContext : System.IDisposable
Inheritance Object → LLamaContext
Implements IDisposable
Properties
VocabCount
Total number of tokens in vocabulary of this model
public int VocabCount { get; }
Property Value
ContextSize
Total number of tokens in the context
public int ContextSize { get; }
Property Value
EmbeddingSize
Dimension of embedding vectors
public int EmbeddingSize { get; }
Property Value
Params
The model params set for this model.
public IModelParams Params { get; set; }
Property Value
NativeHandle
The native handle, which is used to be passed to the native APIs
public SafeLLamaContextHandle NativeHandle { get; }
Property Value
Remarks:
Be careful how you use this!
Encoding
The encoding set for this model to deal with text input.
public Encoding Encoding { get; }
Property Value
EmbeddingLength
The embedding length of the model, also known as n_embed
public int EmbeddingLength { get; }
Property Value
Constructors
LLamaContext(IModelParams, ILLamaLogger)
Caution
Use the LLamaWeights.CreateContext instead
public LLamaContext(IModelParams params, ILLamaLogger logger)
Parameters
params IModelParams
Model params.
logger ILLamaLogger
The logger.
LLamaContext(LLamaWeights, IModelParams, ILLamaLogger)
Create a new LLamaContext for the given LLamaWeights
public LLamaContext(LLamaWeights model, IModelParams params, ILLamaLogger logger)
Parameters
model LLamaWeights
params IModelParams
logger ILLamaLogger
Exceptions
Methods
Clone()
Create a copy of the current state of this context
public LLamaContext Clone()
Returns
Tokenize(String, Boolean)
Tokenize a string.
public Int32[] Tokenize(string text, bool addBos)
Parameters
text String
addBos Boolean
Whether to add a bos to the text.
Returns
DeTokenize(IEnumerable<Int32>)
Detokenize the tokens to text.
public string DeTokenize(IEnumerable<int> tokens)
Parameters
tokens IEnumerable<Int32>
Returns
SaveState(String)
Save the state to specified path.
public void SaveState(string filename)
Parameters
filename String
GetStateData()
Caution
Use GetState instead, this supports larger states (over 2GB)
Get the state data as a byte array.
public Byte[] GetStateData()
Returns
GetState()
Get the state data as an opaque handle
public State GetState()
Returns
LoadState(String)
Load the state from specified path.
public void LoadState(string filename)
Parameters
filename String
Exceptions
LoadState(Byte[])
Load the state from memory.
public void LoadState(Byte[] stateData)
Parameters
stateData Byte[]
Exceptions
LoadState(State)
Load the state from memory.
public void LoadState(State state)
Parameters
state State
Exceptions
Sample(LLamaTokenDataArray, Nullable`1&, Single, MirostatType, Single, Single, Int32, Single, Single, Single, SafeLLamaGrammarHandle)
Perform the sampling. Please don't use it unless you fully know what it does.
public int Sample(LLamaTokenDataArray candidates, Nullable`1& mirostat_mu, float temperature, MirostatType mirostat, float mirostatTau, float mirostatEta, int topK, float topP, float tfsZ, float typicalP, SafeLLamaGrammarHandle grammar)
Parameters
candidates LLamaTokenDataArray
mirostat_mu Nullable`1&
temperature Single
mirostat MirostatType
mirostatTau Single
mirostatEta Single
topK Int32
topP Single
tfsZ Single
typicalP Single
grammar SafeLLamaGrammarHandle
Returns
ApplyPenalty(IEnumerable<Int32>, Dictionary<Int32, Single>, Int32, Single, Single, Single, Boolean)
Apply the penalty for the tokens. Please don't use it unless you fully know what it does.
public LLamaTokenDataArray ApplyPenalty(IEnumerable<int> lastTokens, Dictionary<int, float> logitBias, int repeatLastTokensCount, float repeatPenalty, float alphaFrequency, float alphaPresence, bool penalizeNL)
Parameters
lastTokens IEnumerable<Int32>
logitBias Dictionary<Int32, Single>
repeatLastTokensCount Int32
repeatPenalty Single
alphaFrequency Single
alphaPresence Single
penalizeNL Boolean
Returns
Eval(Int32[], Int32)
public int Eval(Int32[] tokens, int pastTokensCount)
Parameters
tokens Int32[]
pastTokensCount Int32
Returns
Int32
The updated pastTokensCount.
Exceptions
Eval(List<Int32>, Int32)
public int Eval(List<int> tokens, int pastTokensCount)
Parameters
tokens List<Int32>
pastTokensCount Int32
Returns
Int32
The updated pastTokensCount.
Exceptions
Eval(ReadOnlyMemory<Int32>, Int32)
public int Eval(ReadOnlyMemory<int> tokens, int pastTokensCount)
Parameters
tokens ReadOnlyMemory<Int32>
pastTokensCount Int32
Returns
Int32
The updated pastTokensCount.
Exceptions
Eval(ReadOnlySpan<Int32>, Int32)
public int Eval(ReadOnlySpan<int> tokens, int pastTokensCount)
Parameters
tokens ReadOnlySpan<Int32>
pastTokensCount Int32
Returns
Int32
The updated pastTokensCount.
Exceptions
GenerateResult(IEnumerable<Int32>)
internal IEnumerable<string> GenerateResult(IEnumerable<int> ids)
Parameters
Returns
TokenToString(Int32)
Convert a token into a string
public string TokenToString(int token)
Parameters
token Int32
Returns
Dispose()
public void Dispose()