BaseSamplingPipeline

Namespace: LLama.Sampling

Base class for implementing custom sampling pipelines. This provides a helpful framework for implementing ISamplingPipeline.

public abstract class BaseSamplingPipeline : ISamplingPipeline, System.IDisposable

Properties

Grammar to constrain valid tokens

public SafeLLamaGrammarHandle Grammar { get; set; }

public LLamaToken Sample(SafeLLamaContextHandle ctx, Span<float> logits, ReadOnlySpan<LLamaToken> lastTokens)

public void Accept(SafeLLamaContextHandle ctx, LLamaToken token)

Process the raw logit values

protected abstract void ProcessLogits(SafeLLamaContextHandle ctx, Span<float> logits, ReadOnlySpan<LLamaToken> lastTokens)

ctx SafeLLamaContextHandle
The context being sampled from

logits Span<Single>
The logits produced by the model

lastTokens ReadOnlySpan<LLamaToken>
A list of tokens recently returned by the model

Process the LLamaTokenDataArray and select a single token

protected abstract LLamaToken ProcessTokenDataArray(SafeLLamaContextHandle ctx, LLamaTokenDataArray candidates, ReadOnlySpan<LLamaToken> lastTokens)

ctx SafeLLamaContextHandle
The context being sampled from

candidates LLamaTokenDataArray
The LLamaTokenDataArray data produced by the model

lastTokens ReadOnlySpan<LLamaToken>
A list of tokens recently returned by the model

public void Reset()

public abstract ISamplingPipeline Clone()

public void Dispose()