Skip to content

LLamaBatch

Namespace: LLama.Native

A batch allows submitting multiple tokens to multiple sequences simultaneously

1
public class LLamaBatch

Inheritance Object → LLamaBatch

Properties

TokenCount

The number of tokens in this batch

1
public int TokenCount { get; private set; }

Property Value

Int32

SequenceCapacity

Maximum number of sequences a token can be assigned to (automatically grows if exceeded)

1
public int SequenceCapacity { get; private set; }

Property Value

Int32

Constructors

LLamaBatch()

Create a new batch for submitting inputs to llama.cpp

1
public LLamaBatch()

Methods

ToNativeBatch(LLamaNativeBatch&)

1
internal GroupDisposable ToNativeBatch(LLamaNativeBatch& batch)

Parameters

batch LLamaNativeBatch&

Returns

GroupDisposable

Add(LLamaToken, LLamaPos, ReadOnlySpan<LLamaSeqId>, Boolean)

Add a single token to the batch at the same position in several sequences

1
public int Add(LLamaToken token, LLamaPos pos, ReadOnlySpan<LLamaSeqId> sequences, bool logits)

Parameters

token LLamaToken
The token to add

pos LLamaPos
The position to add it att

sequences ReadOnlySpan<LLamaSeqId>
The set of sequences to add this token to

logits Boolean

Returns

Int32
The index that the token was added at. Use this for GetLogitsIth

Remarks:

https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2

Add(LLamaToken, LLamaPos, List<LLamaSeqId>, Boolean)

Add a single token to the batch at the same position in several sequences

1
public int Add(LLamaToken token, LLamaPos pos, List<LLamaSeqId> sequences, bool logits)

Parameters

token LLamaToken
The token to add

pos LLamaPos
The position to add it att

sequences List<LLamaSeqId>
The set of sequences to add this token to

logits Boolean

Returns

Int32
The index that the token was added at. Use this for GetLogitsIth

Remarks:

https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2

Add(LLamaToken, LLamaPos, LLamaSeqId, Boolean)

Add a single token to the batch at a certain position for a single sequences

1
public int Add(LLamaToken token, LLamaPos pos, LLamaSeqId sequence, bool logits)

Parameters

token LLamaToken
The token to add

pos LLamaPos
The position to add it att

sequence LLamaSeqId
The sequence to add this token to

logits Boolean

Returns

Int32
The index that the token was added at. Use this for GetLogitsIth

Remarks:

https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2

AddRange(ReadOnlySpan<LLamaToken>, LLamaPos, LLamaSeqId, Boolean)

Add a range of tokens to a single sequence, start at the given position.

1
public int AddRange(ReadOnlySpan<LLamaToken> tokens, LLamaPos start, LLamaSeqId sequence, bool logitsLast)

Parameters

tokens ReadOnlySpan<LLamaToken>
The tokens to add

start LLamaPos
The starting position to add tokens at

sequence LLamaSeqId
The sequence to add this token to

logitsLast Boolean
Whether the final token should generate logits

Returns

Int32
The index that the final token was added at. Use this for GetLogitsIth

Clear()

Set TokenCount to zero for this batch

1
public void Clear()

GetLogitPositions(Span<ValueTuple<LLamaSeqId, Int32>>)

Get the positions where logits can be sampled from

1
internal Span<ValueTuple<LLamaSeqId, int>> GetLogitPositions(Span<ValueTuple<LLamaSeqId, int>> dest)

Parameters

dest Span<ValueTuple<LLamaSeqId, Int32>>

Returns

Span<ValueTuple<LLamaSeqId, Int32>>