LLamaTokenDataArray

Namespace: LLama.Native

Contains an array of LLamaTokenData, potentially sorted.

public struct LLamaTokenDataArray

Fields

The LLamaTokenData

public Memory<LLamaTokenData> Data;

Indicates if data is sorted by logits in descending order. If this is false the token data is in no particular order.

public bool Sorted;

Create a new LLamaTokenDataArray

LLamaTokenDataArray(Memory<LLamaTokenData> tokens, bool isSorted)

isSorted Boolean

Create a new LLamaTokenDataArray, copying the data from the given logits

LLamaTokenDataArray Create(ReadOnlySpan<float> logits)

Create a new LLamaTokenDataArray, copying the data from the given logits into temporary memory.

LLamaTokenDataArray Create(ReadOnlySpan<float> logits, Memory<LLamaTokenData> buffer)

buffer Memory<LLamaTokenData>
Temporary memory which will be used to work on these logits. Must be at least as large as logits array

Remarks:

The memory must not be modified while this LLamaTokenDataArray is in use.

Overwrite the logit values for all given tokens

void OverwriteLogits(ReadOnlySpan<ValueTuple<LLamaToken, float>> values)

values ReadOnlySpan<ValueTuple<LLamaToken, Single>>
tuples of token and logit value to overwrite

Sorts candidate tokens by their logits in descending order and calculate probabilities based on logits.

void Softmax()