SafeLLamaContextHandle
Namespace: LLama.Native
A safe wrapper around a llama_context
public sealed class SafeLLamaContextHandle : SafeLLamaHandleBase, System.IDisposable
Inheritance Object → CriticalFinalizerObject → SafeHandle → SafeLLamaHandleBase → SafeLLamaContextHandle
Implements IDisposable
Properties
VocabCount
Total number of tokens in vocabulary of this model
public int VocabCount { get; }
Property Value
ContextSize
Total number of tokens in the context
public int ContextSize { get; }
Property Value
EmbeddingSize
Dimension of embedding vectors
public int EmbeddingSize { get; }
Property Value
ModelHandle
Get the model which this context is using
public SafeLlamaModelHandle ModelHandle { get; }
Property Value
IsInvalid
public bool IsInvalid { get; }
Property Value
IsClosed
public bool IsClosed { get; }
Property Value
Constructors
SafeLLamaContextHandle(IntPtr, SafeLlamaModelHandle)
Create a new SafeLLamaContextHandle
public SafeLLamaContextHandle(IntPtr handle, SafeLlamaModelHandle model)
Parameters
handle IntPtr
pointer to an allocated llama_context
model SafeLlamaModelHandle
the model which this context was created from
Methods
ReleaseHandle()
protected bool ReleaseHandle()
Returns
Create(SafeLlamaModelHandle, LLamaContextParams)
Create a new llama_state for the given model
public static SafeLLamaContextHandle Create(SafeLlamaModelHandle model, LLamaContextParams lparams)
Parameters
model SafeLlamaModelHandle
lparams LLamaContextParams
Returns
Exceptions
Clone(LLamaContextParams)
Create a new llama context with a clone of the current llama context state
public SafeLLamaContextHandle Clone(LLamaContextParams lparams)
Parameters
lparams LLamaContextParams
Returns
Tokenize(String, Boolean, Encoding)
Convert the given text into tokens
public Int32[] Tokenize(string text, bool add_bos, Encoding encoding)
Parameters
text String
The text to tokenize
add_bos Boolean
Whether the "BOS" token should be added
encoding Encoding
Encoding to use for the text
Returns
Exceptions
GetLogits()
Token logits obtained from the last call to llama_eval()
The logits for the last token are stored in the last row
Can be mutated in order to change the probabilities of the next token.
Rows: n_tokens
Cols: n_vocab
public Span<float> GetLogits()
Returns
TokenToString(Int32, Encoding)
Convert a token into a string
public string TokenToString(int token, Encoding encoding)
Parameters
token Int32
Token to decode into a string
encoding Encoding
Returns
TokenToString(Int32, Encoding, StringBuilder)
Append a single llama token to a string builder
public void TokenToString(int token, Encoding encoding, StringBuilder dest)
Parameters
token Int32
Token to decode
encoding Encoding
dest StringBuilder
string builder to append the result to
TokenToSpan(Int32, Span<Byte>)
Convert a single llama token into bytes
public int TokenToSpan(int token, Span<byte> dest)
Parameters
token Int32
Token to decode
dest Span<Byte>
A span to attempt to write into. If this is too small nothing will be written
Returns
Int32
The size of this token. nothing will be written if this is larger than dest
Eval(ReadOnlySpan<Int32>, Int32, Int32)
Run the llama inference to obtain the logits and probabilities for the next token.
public bool Eval(ReadOnlySpan<int> tokens, int n_past, int n_threads)
Parameters
tokens ReadOnlySpan<Int32>
The provided batch of new tokens to process
n_past Int32
the number of tokens to use from previous eval calls
n_threads Int32
Returns
Boolean
Returns true on success
GetStateSize()
Get the size of the state, when saved as bytes
public ulong GetStateSize()
Returns
GetState(Byte*, UInt64)
Get the raw state of this context, encoded as bytes. Data is written into the dest pointer.
public ulong GetState(Byte* dest, ulong size)
Parameters
dest Byte*
Destination to write to
size UInt64
Number of bytes available to write to in dest (check required size with GetStateSize())
Returns
UInt64
The number of bytes written to dest
Exceptions
ArgumentOutOfRangeException
Thrown if dest is too small
GetState(IntPtr, UInt64)
Get the raw state of this context, encoded as bytes. Data is written into the dest pointer.
public ulong GetState(IntPtr dest, ulong size)
Parameters
dest IntPtr
Destination to write to
size UInt64
Number of bytes available to write to in dest (check required size with GetStateSize())
Returns
UInt64
The number of bytes written to dest
Exceptions
ArgumentOutOfRangeException
Thrown if dest is too small
SetState(Byte*)
Set the raw state of this context
public ulong SetState(Byte* src)
Parameters
src Byte*
The pointer to read the state from
Returns
UInt64
Number of bytes read from the src pointer
SetState(IntPtr)
Set the raw state of this context
public ulong SetState(IntPtr src)
Parameters
src IntPtr
The pointer to read the state from
Returns
UInt64
Number of bytes read from the src pointer