BaseSamplingPipeline
Namespace: LLama.Sampling
Base class for implementing custom sampling pipelines. This provides a helpful framework for implementing ISamplingPipeline.
public abstract class BaseSamplingPipeline : ISamplingPipeline, System.IDisposable
Inheritance Object → BaseSamplingPipeline
Implements ISamplingPipeline, IDisposable
Properties
Grammar
Grammar to constrain valid tokens
public SafeLLamaGrammarHandle Grammar { get; set; }
Property Value
Methods
Sample(SafeLLamaContextHandle, Span<Single>, ReadOnlySpan<LLamaToken>)
public LLamaToken Sample(SafeLLamaContextHandle ctx, Span<float> logits, ReadOnlySpan<LLamaToken> lastTokens)
Parameters
logits Span<Single>
lastTokens ReadOnlySpan<LLamaToken>
Returns
Accept(SafeLLamaContextHandle, LLamaToken)
public void Accept(SafeLLamaContextHandle ctx, LLamaToken token)
Parameters
token LLamaToken
ProcessLogits(SafeLLamaContextHandle, Span<Single>, ReadOnlySpan<LLamaToken>)
Process the raw logit values
protected abstract void ProcessLogits(SafeLLamaContextHandle ctx, Span<float> logits, ReadOnlySpan<LLamaToken> lastTokens)
Parameters
ctx SafeLLamaContextHandle
The context being sampled from
logits Span<Single>
The logits produced by the model
lastTokens ReadOnlySpan<LLamaToken>
A list of tokens recently returned by the model
ProcessTokenDataArray(SafeLLamaContextHandle, LLamaTokenDataArray, ReadOnlySpan<LLamaToken>)
Process the LLamaTokenDataArray and select a single token
protected abstract LLamaToken ProcessTokenDataArray(SafeLLamaContextHandle ctx, LLamaTokenDataArray candidates, ReadOnlySpan<LLamaToken> lastTokens)
Parameters
ctx SafeLLamaContextHandle
The context being sampled from
candidates LLamaTokenDataArray
The LLamaTokenDataArray data produced by the model
lastTokens ReadOnlySpan<LLamaToken>
A list of tokens recently returned by the model
Returns
Reset()
public void Reset()
Clone()
public abstract ISamplingPipeline Clone()
Returns
Dispose()
public void Dispose()