Conversation
Namespace: LLama.Batched
A single conversation thread that can be prompted (adding tokens from the user) or inferred (extracting a token from the LLM)
1 | |
Inheritance Object → Conversation
Implements IDisposable
Attributes NullableContextAttribute, NullableAttribute
Properties
Executor
The executor which this conversation belongs to
1 | |
Property Value
ConversationId
Unique ID for this conversation
1 | |
Property Value
TokenCount
Total number of tokens in this conversation, cannot exceed the context length.
1 | |
Property Value
IsDisposed
Indicates if this conversation has been disposed, nothing can be done with a disposed conversation
1 | |
Property Value
RequiresInference
Indicates if this conversation is waiting for inference to be run on the executor. "Prompt" and "Sample" cannot be called when this is true.
1 | |
Property Value
RequiresSampling
Indicates that this conversation should be sampled.
1 | |
Property Value
Methods
Finalize()
Finalizer for Conversation
1 | |
Dispose()
End this conversation, freeing all resources used by it
1 | |
Exceptions
Fork()
Create a copy of the current conversation
1 | |
Returns
Exceptions
Remarks:
The copy shares internal state, so consumes very little extra memory.
GetSampleIndex(Int32)
Get the index in the context which each token can be sampled from, the return value of this function get be used to retrieve logits (SafeLLamaContextHandle.GetLogitsIth(Int32)) or to sample a token (SafeLLamaSamplerChainHandle.Sample(SafeLLamaContextHandle, Int32).
1 | |
Parameters
offset Int32
How far from the end of the previous prompt should logits be sampled. Any value other than 0 requires
allLogits to have been set during prompting.
For example if 5 tokens were supplied in the last prompt call:
-
-
-
Returns
Exceptions
CannotSampleRequiresPromptException
Thrown if this conversation was not prompted before the previous call to infer
CannotSampleRequiresInferenceException
Thrown if Infer() must be called on the executor
Sample(Int32)
Get the logits from this conversation, ready for sampling
1 | |
Parameters
offset Int32
How far from the end of the previous prompt should logits be sampled. Any value other than 0 requires allLogits to have been set during prompting
Returns
Exceptions
CannotSampleRequiresPromptException
Thrown if this conversation was not prompted before the previous call to infer
CannotSampleRequiresInferenceException
Thrown if Infer() must be called on the executor
Prompt(List<LLamaToken>, Boolean)
Add tokens to this conversation
1 | |
Parameters
tokens List<LLamaToken>
allLogits Boolean
If true, generate logits for all tokens. If false, only generate logits for the last token.
Exceptions
AlreadyPromptedConversationException
Prompt(ReadOnlySpan<LLamaToken>, Boolean)
Add tokens to this conversation
1 | |
Parameters
tokens ReadOnlySpan<LLamaToken>
allLogits Boolean
If true, generate logits for all tokens. If false, only generate logits for the last token.
Exceptions
AlreadyPromptedConversationException
Prompt(LLamaToken)
Add a single token to this conversation
1 | |
Parameters
token LLamaToken
Exceptions
AlreadyPromptedConversationException
Prompt(SafeLlavaImageEmbedHandle)
Prompt this conversation with an image embedding
1 | |
Parameters
embedding SafeLlavaImageEmbedHandle
Prompt(ReadOnlySpan<Single>)
Prompt this conversation with embeddings
1 | |
Parameters
embeddings ReadOnlySpan<Single>
The raw values of the embeddings. This span must divide equally by the embedding size of this model.
Modify(ModifyKvCache)
Directly modify the KV cache of this conversation
1 | |
Parameters
modifier ModifyKvCache
Exceptions
CannotModifyWhileRequiresInferenceException
Thrown if this method is called while Conversation.RequiresInference == true
Save(String)
Save the complete state of this conversation to a file. if the file already exists it will be overwritten.
1 | |
Parameters
filepath String
Exceptions
CannotSaveWhileRequiresInferenceException
Save()
Save the complete state of this conversation in system memory.
1 | |