Skip to content

< Back

LLamaContextParams

Namespace: LLama.Native

A C# representation of the llama.cpp llama_context_params struct

1
public struct LLamaContextParams

Inheritance Object → ValueType → LLamaContextParams

Remarks:

changing the default values of parameters marked as [EXPERIMENTAL] may cause crashes or incorrect results in certain configurations https://github.com/ggerganov/llama.cpp/pull/7544

Fields

n_ctx

text context, 0 = from model

1
public uint n_ctx;

n_batch

logical maximum batch size that can be submitted to llama_decode

1
public uint n_batch;

n_ubatch

physical maximum batch size

1
public uint n_ubatch;

n_seq_max

max number of sequences (i.e. distinct states for recurrent models)

1
public uint n_seq_max;

n_threads

number of threads to use for generation

1
public int n_threads;

n_threads_batch

number of threads to use for batch processing

1
public int n_threads_batch;

rope_scaling_type

RoPE scaling type, from enum llama_rope_scaling_type

1
public RopeScalingType rope_scaling_type;

llama_pooling_type

whether to pool (sum) embedding results by sequence id

1
public LLamaPoolingType llama_pooling_type;

attention_type

Attention type to use for embeddings

1
public LLamaAttentionType attention_type;

rope_freq_base

RoPE base frequency, 0 = from model

1
public float rope_freq_base;

rope_freq_scale

RoPE frequency scaling factor, 0 = from model

1
public float rope_freq_scale;

yarn_ext_factor

YaRN extrapolation mix factor, negative = from model

1
public float yarn_ext_factor;

yarn_attn_factor

YaRN magnitude scaling factor

1
public float yarn_attn_factor;

yarn_beta_fast

YaRN low correction dim

1
public float yarn_beta_fast;

yarn_beta_slow

YaRN high correction dim

1
public float yarn_beta_slow;

yarn_orig_ctx

YaRN original context size

1
public uint yarn_orig_ctx;

defrag_threshold

defragment the KV cache if holes/size > defrag_threshold, Set to < 0 to disable (default)

1
public float defrag_threshold;

cb_eval

ggml_backend_sched_eval_callback

1
public IntPtr cb_eval;

cb_eval_user_data

User data passed into cb_eval

1
public IntPtr cb_eval_user_data;

type_k

data type for K cache. EXPERIMENTAL

1
public GGMLType type_k;

type_v

data type for V cache. EXPERIMENTAL

1
public GGMLType type_v;

abort_callback

ggml_abort_callback

1
public IntPtr abort_callback;

abort_callback_user_data

User data passed into abort_callback

1
public IntPtr abort_callback_user_data;

Properties

embeddings

if true, extract embeddings (together with logits)

1
public bool embeddings { get; set; }

Property Value

Boolean

offload_kqv

whether to offload the KQV ops (including the KV cache) to GPU

1
public bool offload_kqv { get; set; }

Property Value

Boolean

flash_attention

whether to use flash attention. EXPERIMENTAL

1
public bool flash_attention { get; set; }

Property Value

Boolean

no_perf

whether to measure performance timings

1
public bool no_perf { get; set; }

Property Value

Boolean

Methods

Default()

Get the default LLamaContextParams

1
LLamaContextParams Default()

Returns

LLamaContextParams

< Back