Skip to content

< Back

IModelParams

Namespace: LLama.Abstractions

The parameters for initializing a LLama model.

1
public interface IModelParams

Attributes NullableContextAttribute

Properties

MainGpu

main_gpu interpretation depends on split_mode:

  • None - The GPU that is used for the entire mode.
  • Row - The GPU that is used for small tensors and intermediate results.
  • Layer - Ignored.
1
public abstract int MainGpu { get; set; }

Property Value

Int32

SplitMode

How to split the model across multiple GPUs

1
public abstract Nullable<GPUSplitMode> SplitMode { get; }

Property Value

Nullable<GPUSplitMode>

TensorBufferOverrides

Buffer type overrides for specific tensor patterns, allowing you to specify hardware devices to use for individual tensors or sets of tensors. Equivalent to --override-tensor or -ot on the llama.cpp command line or tensor_buft_overrides internally.

1
public abstract List<TensorBufferOverride> TensorBufferOverrides { get; }

Property Value

List<TensorBufferOverride>

GpuLayerCount

Number of layers to run in VRAM / GPU memory (n_gpu_layers)

1
public abstract int GpuLayerCount { get; }

Property Value

Int32

UseMemorymap

Use mmap for faster loads (use_mmap)

1
public abstract bool UseMemorymap { get; }

Property Value

Boolean

UseMemoryLock

Use mlock to keep model in memory (use_mlock)

1
public abstract bool UseMemoryLock { get; }

Property Value

Boolean

ModelPath

Model path (model)

1
public abstract string ModelPath { get; }

Property Value

String

TensorSplits

how split tensors should be distributed across GPUs

1
public abstract TensorSplitsCollection TensorSplits { get; }

Property Value

TensorSplitsCollection

VocabOnly

Load vocab only (no weights)

1
public abstract bool VocabOnly { get; }

Property Value

Boolean

CheckTensors

Validate model tensor data before loading

1
public abstract bool CheckTensors { get; }

Property Value

Boolean

MetadataOverrides

Override specific metadata items in the model

1
public abstract List<MetadataOverride> MetadataOverrides { get; }

Property Value

List<MetadataOverride>

< Back