Skip to content

< Back

LLamaBatchEmbeddings

Namespace: LLama.Native

An embeddings batch allows submitting embeddings to multiple sequences simultaneously

1
public class LLamaBatchEmbeddings

Inheritance Object → LLamaBatchEmbeddings
Attributes NullableContextAttribute, NullableAttribute

Properties

EmbeddingDimensions

Size of an individual embedding

1
public int EmbeddingDimensions { get; }

Property Value

Int32

EmbeddingsCount

The number of items in this batch

1
public int EmbeddingsCount { get; private set; }

Property Value

Int32

SequenceCapacity

Maximum number of sequences an item can be assigned to (automatically grows if exceeded)

1
public int SequenceCapacity { get; private set; }

Property Value

Int32

Constructors

LLamaBatchEmbeddings(Int32)

Create a new batch for submitting inputs to llama.cpp

1
public LLamaBatchEmbeddings(int embeddingDimensions)

Parameters

embeddingDimensions Int32

Methods

Add(ReadOnlySpan<Single>, LLamaPos, ReadOnlySpan<LLamaSeqId>, Boolean)

Add a single embedding to the batch at the same position in several sequences

1
public int Add(ReadOnlySpan<float> embedding, LLamaPos pos, ReadOnlySpan<LLamaSeqId> sequences, bool logits)

Parameters

embedding ReadOnlySpan<Single>
The embedding to add

pos LLamaPos
The position to add it att

sequences ReadOnlySpan<LLamaSeqId>
The set of sequences to add this token to

logits Boolean

Returns

Int32
The index that the token was added at. Use this for GetLogitsIth

Remarks:

https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2

Add(ReadOnlySpan<Single>, LLamaPos, LLamaSeqId, Boolean)

Add a single embedding to the batch for a single sequence

1
public int Add(ReadOnlySpan<float> embedding, LLamaPos pos, LLamaSeqId sequence, bool logits)

Parameters

embedding ReadOnlySpan<Single>

pos LLamaPos

sequence LLamaSeqId

logits Boolean

Returns

Int32
The index that the token was added at. Use this for GetLogitsIth

Add<TParam>(TParam, WriteEmbeddingsDelegate<TParam>, LLamaPos, ReadOnlySpan<LLamaSeqId>, Boolean)

Add a single embedding to the batch at the same position in several sequences

1
public int Add<TParam>(TParam parameter, WriteEmbeddingsDelegate<TParam> write, LLamaPos pos, ReadOnlySpan<LLamaSeqId> sequences, bool logits)

Type Parameters

TParam
Type of userdata passed to write delegate

Parameters

parameter TParam
Userdata passed to write delegate

write WriteEmbeddingsDelegate<TParam>
Delegate called once to write data into a span

pos LLamaPos
Position to write this embedding to

sequences ReadOnlySpan<LLamaSeqId>
All sequences to assign this embedding to

logits Boolean
Whether logits should be generated for this embedding

Returns

Int32
The index that the token was added at. Use this for GetLogitsIth

Remarks:

https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2

Add<TParam>(TParam, WriteEmbeddingsDelegate<TParam>, LLamaPos, LLamaSeqId, Boolean)

Add a single embedding to the batch at a position for one sequence

1
public int Add<TParam>(TParam parameter, WriteEmbeddingsDelegate<TParam> write, LLamaPos pos, LLamaSeqId sequence, bool logits)

Type Parameters

TParam
Type of userdata passed to write delegate

Parameters

parameter TParam
Userdata passed to write delegate

write WriteEmbeddingsDelegate<TParam>
Delegate called once to write data into a span

pos LLamaPos
Position to write this embedding to

sequence LLamaSeqId
Sequence to assign this embedding to

logits Boolean
Whether logits should be generated for this embedding

Returns

Int32
The index that the token was added at. Use this for GetLogitsIth

Remarks:

https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/common/common.cpp#L829C2-L829C2

Clear()

Set EmbeddingsCount to zero for this batch

1
public void Clear()

< Back