LLamaBatch
Namespace: LLama.Native
A batch allows submitting multiple tokens to multiple sequences simultaneously
public class LLamaBatch
Inheritance Object → LLamaBatch
Properties
TokenCount
The number of tokens in this batch
public int TokenCount { get; private set; }
Property Value
SequenceCapacity
Maximum number of sequences a token can be assigned to (automatically grows if exceeded)
public int SequenceCapacity { get; private set; }
Property Value
Constructors
LLamaBatch()
Create a new batch for submitting inputs to llama.cpp
public LLamaBatch()
Methods
ToNativeBatch(LLamaNativeBatch&)
internal GroupDisposable ToNativeBatch(LLamaNativeBatch& batch)
Parameters
batch LLamaNativeBatch&
Returns
Add(LLamaToken, LLamaPos, ReadOnlySpan<LLamaSeqId>, Boolean)
Add a single token to the batch at the same position in several sequences
public int Add(LLamaToken token, LLamaPos pos, ReadOnlySpan<LLamaSeqId> sequences, bool logits)
Parameters
token LLamaToken
The token to add
pos LLamaPos
The position to add it att
sequences ReadOnlySpan<LLamaSeqId>
The set of sequences to add this token to
logits Boolean
Returns
Int32
The index that the token was added at. Use this for GetLogitsIth
Remarks:
https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2
Add(LLamaToken, LLamaPos, List<LLamaSeqId>, Boolean)
Add a single token to the batch at the same position in several sequences
public int Add(LLamaToken token, LLamaPos pos, List<LLamaSeqId> sequences, bool logits)
Parameters
token LLamaToken
The token to add
pos LLamaPos
The position to add it att
sequences List<LLamaSeqId>
The set of sequences to add this token to
logits Boolean
Returns
Int32
The index that the token was added at. Use this for GetLogitsIth
Remarks:
https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2
Add(LLamaToken, LLamaPos, LLamaSeqId, Boolean)
Add a single token to the batch at a certain position for a single sequences
public int Add(LLamaToken token, LLamaPos pos, LLamaSeqId sequence, bool logits)
Parameters
token LLamaToken
The token to add
pos LLamaPos
The position to add it att
sequence LLamaSeqId
The sequence to add this token to
logits Boolean
Returns
Int32
The index that the token was added at. Use this for GetLogitsIth
Remarks:
https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2
AddRange(ReadOnlySpan<LLamaToken>, LLamaPos, LLamaSeqId, Boolean)
Add a range of tokens to a single sequence, start at the given position.
public int AddRange(ReadOnlySpan<LLamaToken> tokens, LLamaPos start, LLamaSeqId sequence, bool logitsLast)
Parameters
tokens ReadOnlySpan<LLamaToken>
The tokens to add
start LLamaPos
The starting position to add tokens at
sequence LLamaSeqId
The sequence to add this token to
logitsLast Boolean
Whether the final token should generate logits
Returns
Int32
The index that the final token was added at. Use this for GetLogitsIth
Clear()
Set TokenCount to zero for this batch
public void Clear()
GetLogitPositions(Span<ValueTuple<LLamaSeqId, Int32>>)
Get the positions where logits can be sampled from
internal Span<ValueTuple<LLamaSeqId, int>> GetLogitPositions(Span<ValueTuple<LLamaSeqId, int>> dest)
Parameters
dest Span<ValueTuple<LLamaSeqId, Int32>>