Table of Contents

Namespace NumSharp.Backends.Kernels

Classes

DirectILKernelGenerator

Binary operations (same-type) - contiguous kernels and generic helpers.

GeneratedDelegates

Central registry for every runtime-generated kernel cache. Each cache exposes a public read-only *Count (live cache size) and an internal reset.

ILKernelGenerator

Generates per-chunk IL kernels for NDIter-driven execution.

Kernels emitted here are called as the inner loop of an NDIter iteration — once per chunk, with dataptrs/strides/count provided by the iterator. The kernel does no axis or stride walking of its own.

Add new kernel families in ILKernelGenerator.<Op>.cs partial files. See DirectILKernelGenerator for the legacy whole-array kernels currently being migrated to this model.

ReductionOpExtensions

Extension methods for ReductionOp.

ReductionTypeExtensions

Extension methods for NPTypeCode related to reductions.

SimdDot

SIMD fused multiply-accumulate dot product for contiguous float / double vectors. Computes sum(a[i] * b[i]) in a single pass — no temporary product array (contrast with left * right followed by ReduceAdd, which materializes an n-element temp and walks the data twice).

Four independent Vector256 accumulators give the out-of-order core enough instruction-level parallelism to hide FMA latency; a scalar tail handles the remainder. Accumulation type matches the element type (double in double, float in float) so the result dtype mirrors NumPy's np.dot.

Callers route only contiguous (stride == 1) same-type operands here; strided views take a scalar strided loop, and non-float dtypes take the INumber<T> path — both in Default.Dot.Fused.cs.

SimdMatMul

High-performance SIMD matrix multiplication with cache blocking and panel packing. Single-threaded implementation achieving ~20 GFLOPS on modern CPUs.

Key optimizations:

  • GEBP (General Block Panel) algorithm with cache blocking
  • Full panel packing: A as [kc][MR] panels, B as [kc][NR] panels
  • 8x16 micro-kernel with 16 Vector256 accumulators
  • FMA (Fused Multiply-Add) for 2x FLOP throughput
  • 4x k-loop unrolling for instruction-level parallelism

Stride-aware variants (see SimdMatMul.Strided.cs / SimdMatMul.Double.cs) accept (stride0, stride1) for each operand so transposed / sliced NDArray views can be matmul'd without materializing contiguous copies.

StrideDetector

Stride-based pattern detection for selecting optimal SIMD execution paths. All methods are aggressively inlined for minimal dispatch overhead.

Structs

AxisReductionKernelKey

Cache key for axis reduction kernels. Reduces along a specific axis, producing an array with one fewer dimension.

BinaryScalarKernelKey

Cache key for binary scalar operation kernels. Identifies a unique kernel by LHS type, RHS type, result type, and operation.

ComparisonKernelKey

Cache key for comparison operation kernels. Identifies a unique kernel by LHS type, RHS type, operation, and execution path. Result type is always bool (NPTypeCode.Boolean).

ComparisonScalarKernelKey

Cache key for comparison scalar operation kernels. Identifies a unique kernel by LHS type, RHS type, and comparison operation. Result type is always bool.

CopyKernelKey
CumulativeAxisKernelKey

Cache key for cumulative axis reduction kernels (cumsum along axis, etc.). Output has same shape as input, cumulative accumulation along specified axis.

CumulativeKernelKey

Cache key for cumulative reduction kernels (cumsum, etc.). Output has same shape as input, each element is accumulation of elements before it.

DirectILKernelGenerator.WeightedSumKernelKey
ElementReductionKernelKey

Cache key for element-wise (full array) reduction kernels. Reduces all elements to a single scalar value.

ILKernelGenerator.ReduceKernelKey

Cache key for a per-chunk reduction kernel: operation + input dtype + accumulator (output) dtype. Layout is handled at runtime inside the kernel body (pinned vs slab, contiguous vs strided inner loop), so it is NOT part of the key.

ILKernelGenerator.ScanAxisAux

Scan-axis geometry passed to a cumulative inner-loop kernel via auxdata. Byte strides + element count for the (removed) scan axis; the iterator supplies the per-operand byte strides for the remaining innermost axis through the kernel's strides argument.

IndexCollector

A growable buffer for collecting long indices, backed by NDArray storage. Replaces LongIndexBuffer - uses NumSharp's existing unmanaged memory infrastructure.

InnerLoopKernelKey

Compact identity of a production inner-loop kernel (the Tier-3B kernels the ufunc routes drive through NDIterRef.ExecuteElementWise*): the ufunc family, its op and up to three dtypes packed into one 64-bit value.

MixedTypeKernelKey

Cache key for mixed-type binary operation kernels. Identifies a unique kernel by LHS type, RHS type, result type, operation, and execution path.

UnaryKernelKey

Cache key for unary operation kernels. Identifies a unique kernel by input type, output type, operation, and whether contiguous.

UnaryScalarKernelKey

Cache key for unary scalar operation kernels. Identifies a unique kernel by input type, output type, and operation.

Enums

BinaryOp

Binary operations supported by kernel providers.

ComparisonOp

Comparison operations supported by kernel providers. All comparison operations return bool (NPTypeCode.Boolean).

CopyExecutionPath
DirectILKernelGenerator.ClipBoundsKind
DirectILKernelGenerator.ClipMode
ExecutionPath

Execution paths for binary operations, selected based on stride analysis.

InnerLoopKernelKey.KernelFamily

The ufunc family a key belongs to (selects the string format and op enum).

ReductionOp

Reduction operations supported by kernel providers.

UnaryOp

Unary operations supported by kernel providers.

Delegates

ArgwhereCountKernel

IL-emitted SIMD popcount: returns the count of non-zero elements in a contiguous T-typed buffer. size is in elements (not bytes).

ArgwhereExpandKernel

IL-emitted coord-expand: converts a flat-index buffer (monotonic ascending C-order) to the (count, ndim) row-major argwhere result via incremental coord advance — no per-element divmod. Dtype-agnostic (operates on long*).

ArgwhereFlatKernel

IL-emitted SIMD bit-scan: writes flat element indices of non-zero T positions into outBuf (caller must pre-size via ArgwhereCountKernel). Returns the number written.

AxisReductionKernel

Delegate for axis reduction kernels. Reduces along a specific axis, writing to output array.

ChooseFlatKernel

Flat np.choose kernel: every operand is C-contiguous at the result shape, so element flat is at choiceBases[mi] + flatelem and ((long)idxBase)[flat].

ChooseStridedKernel

Strided np.choose kernel: an odometer over the C-contiguous result; the index and the selected choice are read through their own per-dimension BYTE strides (0 = broadcast).

ComparisonKernel

Comparison operation kernel signature using void pointers. LHS and RHS may have different types, but result is always bool. Type conversion is handled internally by the generated IL.

ContiguousKernel<T>

Delegate for contiguous (SimdFull) binary operations. Simplified signature - no strides needed since both arrays are contiguous.

CopyKernel
CumulativeAxisKernel

Delegate for cumulative axis reduction kernels. Computes running accumulation along a specific axis.

CumulativeKernel

Delegate for cumulative reduction kernels (cumsum, etc.). Output has same shape as input.

DiagWriteKernel

IL-emitted strided store: writes dst[idstStride] = src[isrcStride] for each i in [0, count). A srcByteStride of 0 broadcasts a single source element (scalar fill). Pointers are pre-offset by the caller; strides are in bytes and may be negative.

DirectILKernelGenerator.AdjacentDiffKernel

Fused adjacent-difference kernel: for each of rows rows, dst[i] = src[i+1] - src[i] over the row's inLen contiguous elements (the output row has inLen-1 elements). src and dst are raw base pointers to C-contiguous buffers.

DirectILKernelGenerator.CastKernel

Cross-dtype contiguous copy kernel. Both src and dst must be contiguous.

DirectILKernelGenerator.ClipKernel

Universal clip kernel signature: read from src, clamp to lo / hi, write to dst. size is element count. Bound pointers are interpreted per the mode the kernel was generated for (scalar = single value, array = size values). Bound pointers for unused sides are ignored by the generated IL.

DirectILKernelGenerator.DivModKernel

Fused divmod kernel over contiguous same-dtype buffers: for each i, q[i] = floor(a[i]/b[i]) and r[i] = a[i] - q[i]b[i] (floored), both produced by a single NDDivision Divmod call.

DirectILKernelGenerator.GemvKernel

Matrix·vector kernel: y[i] = dot(A[i], x) for row-major A whose rows are contiguous (K along stride 1) and start aRowStride elements apart, and contiguous x/y.

DirectILKernelGenerator.GramKernel

Computes the symmetric Gram matrix G = A @ B^T for row-major A, B each shaped (m, k), into the (m, m) buffer g. The result is symmetric (g[i,j] == g[j,i]); the upper triangle is computed and mirrored. For the unweighted covariance a == b (the same buffer); the weighted form passes b = Xc*w.

DirectILKernelGenerator.HistogramCountKernel

Fused unweighted histogram count kernel: for each sample compute its bin index (as in DirectILKernelGenerator.HistogramIndicesKernel) and increment the count directly — no int64 indices array and no separate bincount pass. Out-of-range/NaN samples are skipped.

DirectILKernelGenerator.HistogramIndicesKernel

Fused histogram bin-index kernel: maps each sample (pre-cast to the bin dtype B) to its equal-width bin index, or to the sentinel bin nbins when out of range.

DirectILKernelGenerator.HistogramSearchCountKernel

Fused non-uniform histogram count kernel: for each sample, binary-search its bin among arbitrary (monotonic) edges and increment the count — replacing a whole-array searchsorted + comparison

  • astype + bincount chain (5 passes and several full-size temps) with ONE pass.
DirectILKernelGenerator.HistogramSearchIndexKernel

Fused per-dimension bin-index kernel for histogramdd: for each sample writes searchsorted(edges, x, 'right') with a value on the last edge pulled one bin left — the exact Ncount histogramdd needs — using the same branchless SIMD/binary search as the count kernels. Replaces the general (slow) np.searchsorted call in the D-dimensional hot loop.

DirectILKernelGenerator.HistogramWeightedKernel

Fused weighted histogram scatter for equal-width bins: for each in-range sample, add its (float64) weight to the bin's running float64 total. Runs over ONE block; the caller casts the total to the weight dtype and folds it into the result, replicating NumPy's per-block n += bincount(block, w).astype(ntype) — no int64 indices array, no separate bincount pass.

DirectILKernelGenerator.InnerCastLoop

Convert count elements src→dst, advancing each pointer by its byte stride per element. One direct call Converts.To{Dst} body.

DirectILKernelGenerator.MaskedCastKernel

where-masked cross-dtype copy kernel delegate.

DirectILKernelGenerator.NanToNumKernel

Universal nan_to_num kernel signature: read from src, replace non-finite values with the scalar fills, write to dst. size is the element count. nan / posinf / neginf each point to ONE value of the kernel dtype.

DirectILKernelGenerator.QuantileKernel

The emitted quantile kernel: one call computes every row (reduced line) of the staged data.

DirectILKernelGenerator.RepeatBroadcastKernel

Broadcast variant — every j uses the same count.

DirectILKernelGenerator.RepeatPerJKernel

Per-j variant — counts[j] varies; must have length n.

DirectILKernelGenerator.SearchSortedKernel

SearchSorted kernel delegate.

DirectILKernelGenerator.ShiftArrayKernel<T>

Delegate for shift operation with per-element shift amounts. This is the scalar loop path for element-wise shifts.

DirectILKernelGenerator.ShiftScalarKernel<T>

Delegate for shift operation with scalar shift amount. This is the SIMD-optimized path for uniform shifts.

DirectILKernelGenerator.StridedCastKernel

Cross-dtype strided/broadcast copy kernel.

ElementCopyKernel

An IL-emitted copy of opaque fixed-width elements over a 2-D grid of byte strides: for every (r, c), the destination element at dst + r * dstRowStride + c * dstColStride receives the kernel's COPY bytes from src + r * srcRowStride + c * srcColStride, followed by ZERO NUL bytes.

ElementReductionKernel

Delegate for element-wise reduction kernels. Reduces all elements of an array to a single value.

FilterAxisKernel

IL-emitted fused mask-driven gather. Reads src at each True position in mask and emits one slab per outer block into dst. The runtime innerSize is honoured by the "bulk" variant; the typed variants (1/2/4/8/16-byte) ignore it (the size is baked into IL).

IndicesKernel

IL-emitted slab filler for np.indices. Writes the multi-axis coordinate of each output position directly via blockwise SIMD memsets; no per-element divmod or coord advance.

MaskGatherKernel

Copies, in order, every slab of src whose mask byte is non-zero into consecutive slabs of dst; returns the number of slabs written (see the file header for the contract).

MaskedAssignKernel

Where-masked bit copy of one element width over prepared, broadcast operands (np.copyto's element loop): dst[i] = src[i] wherever mask[i] != 0.

MatMul2DKernel<T>

Kernel delegate for 2D matrix multiplication: C = A * B A is [M x K], B is [K x N], C is [M x N] All matrices are row-major contiguous.

MixedTypeKernel

Mixed-type binary operation kernel signature using void pointers. Handles operations where LHS, RHS, and result may have different types. Type conversion is handled internally by the generated IL.

ND2DElementwiseKernel

A whole-2-D-block element-wise kernel: it loops the outer (strided) axis itself, running an inner SIMD loop over innerCount elements per row and advancing each operand pointer by outerByteStrides[op]. One call covers the whole coalesced 2-D block.

NonZeroPerDimKernel

IL-emitted coord expand for np.nonzero: converts a flat-index buffer (monotonic ascending C-order) into ndim separate per-dim coordinate columns via incremental coord advance — no per-element divmod. Dtype-agnostic (operates on long*).

outCols is a pointer to an array of ndim long* pointers, one per output dimension. outCols[d][i] receives the coordinate along dim d for the i'th non-zero element.

PiecewiseScalarKernel

Fused np.piecewise (scalar-funclist) kernel. conds is n boolean base pointers (the real conditions), funcVals is n + 1 scalar values of the result dtype (funcVals[0..n-1] the per-condition scalars, funcVals[n] the seed — the default, or 0), and result is C-contiguous of count elements. result[i] is funcVals[k] for the LARGEST k in [0,n) whose cond is true, else the seed.

PlaceKernel

IL-emitted mask-driven scatter for np.place. For each i in [0, maskSize) with mask[i] true, writes dst[i] = values[j % valuesCount] and advances j.

PolyCalcKernel

One numpy.polynomial calculus kernel call (see the file header): loads/scales and differentiates (orders fused orders) or integrates (one order) a C-contiguous working buffer in place.

PolyCastKernel

np.array(src[:n], copy=True, dtype=T) for a 1-D source: r[i] = (T)p[i] with the house (NumPy astype) conversion, strided source into a contiguous destination.

PolyChebMulxKernel

One fused chebmulx call (see DirectILKernelGenerator's PolyAlgebra file header): NumPy's prd for a series of n ≥ 2 terms of one dtype, computed in one forward pass.

PolyChebNodesKernel

One Chebyshev node fill (see NumSharp.Backends.Kernels.PolyChebNodes for each formula): count float64 results written contiguously from dst, element k computed from the exact ramp value first + k*step (or k itself, for NumSharp.Backends.Kernels.PolyChebNodes.Pts2).

PolyCombineKernel

The body of NumPy's polyutils._add / _sub after as_series: writes the in-place-updated target series r[0..na) (contiguous, the result dtype) from the target a (the operand NumPy updates in place) and the other operand b, both read through their own strides and dtypes, then runs trimseq on r.

PolyLastAboveKernel

trimcoef's scan (NumPy 2.4.2 polyutils.trimcoef): 1 + ind[-1] for [ind] = np.nonzero(np.abs(c) > tol), or 0 when no element exceeds tol.

PolyMapParmsKernel

polyutils.mapparms on four NumPy scalars of ONE dtype — the whole of NumPy's six scalarmath operations in one call: oldlen = o1 - o0; newlen = n1 - n0; off = (o1n0 - o0n1) / oldlen; scl = newlen / oldlen.

PolyNormedHermiteKernel

One normalised Hermite recurrence of degree steps + 1 (at least 1) at count contiguous float64 points: dst[k] = h(x[k]) with NumPy's statements (see the file header), reading row j of table at step j.

PolyNormedHermiteStepsKernel

Writes the step table of one normalised Hermite recurrence: row j (two doubles at table + 2*j) holds NumPy's np.sqrt((nd - 1.) / nd) and np.sqrt(2. / nd) (NumSharp.Backends.Kernels.PolyNormedHermite.HermiteE: np.sqrt(1. / nd)) for nd = first - j, as NumPy's loop computes them before its nd = nd - 1.0.

PolyRampKernel

NumPy's np.arange over int64 as a raw fill: dst[k] = start + k*step for k in [0, n), the running value advanced by addition (no per-element multiply).

PolyScalarBinaryKernel

One NumPy scalar binary op: *r = *a OP *b, both operands already in the loop dtype.

PolyScalarBoxKernel

Boxes one element as its CLR type (double, Half, Complex, …).

PolyScalarPredicateKernel

One NumPy scalar comparison against the Python int 0.

PolyScalarUnaryKernel

One NumPy scalar unary op (np.negative): r = -a.

PolyTrimLenKernel

trimseq's scan (NumPy 2.4.2 polyutils.trimseq), with NumPy's two return shapes kept apart: seq ITSELF when it is empty or its last element is nonzero, otherwise the SLICE seq[:i+1] for the last i with seq[i] != 0 (at least one element) — a slice even when it keeps every element ([0.]), which is observable as a view.

PolyVanderKernel

One numpy.polynomial Vandermonde kernel call (see the file header): the pseudo-Vandermonde matrix of one basis over cols points, 1-D or the fused 2-D / 3-D outer product.

PutAlongAxisScatterKernel

IL-emitted per-element scatter kernel for np.put_along_axis. Walks the iteration odometer, reads one value through the (broadcast) values strides and writes it into arr at the resolved axis position. Runs ONLY after PutAlongAxisValidateKernel has passed, so it applies the negative-index wrap but performs NO bounds check in the hot loop.

PutAlongAxisValidateKernel

IL-emitted index-validation pass for np.put_along_axis. Walks the iteration odometer, applies the advanced-index negative wrap, and bounds-checks each int64 index against the axis length — WITHOUT writing anything, so a caller can guarantee NumPy's all-or-nothing assignment (a single out-of-range index leaves the destination untouched). Dtype-agnostic (indices are always int64), hence a single cached instance.

PutFlatKernel

Lean 1-D scatter: dst[indices[j]] = values[j mod valuesCount] (a wrapping values cursor: valuesCount == 1 broadcasts a scalar, == count is a straight copy). RAISE semantics as TakeFlatKernel; returns count or the first failing position — elements before it HAVE been written.

PutKernel

IL-emitted scatter kernel for np.put. Writes dst[apply_mode(indices[i])] = values[i % valuesCount] for each i in [0, indicesCount).

PutMaskKernel

IL-emitted mask-driven scatter for np.putmask. For each i in [0, maskSize) with mask[i] true, writes dst[i] = values[i % valuesCount]. The values cursor advances on every element (NumPy's npy_fastputmask), unlike PlaceKernel.

RavelMultiIndexKernel

IL-emitted multi-coord→flat-index folder for np.ravel_multi_index. Caller pre-casts each coord array to contig int64 and computes ravelStrides (C or F order baked into the strides); the kernel reads coords[d][i] linearly, applies the per-axis modes clipping/wrapping, and writes the summed flat index into outIndices.

SelectFusedKernel

Fused np.select kernel. conds is n boolean base pointers, choices is n choice base pointers (all the result dtype, C-contiguous, full result size). defPtr is the default — a single element when the kernel was generated for a scalar default, otherwise a full-size array. result is C-contiguous of count elements.

StridedMoveKernel

IL-emitted strided copy of fixed-width elements: dst[i * dstByteStride] = src[i * srcByteStride] for each i in [0, count).

StridedUnaryKernel

Fused strided-source unary kernel: builds each SIMD vector directly from a strided 1-D source via lane-count scalar gathers, applies the unary op, and stores contiguously — single pass, no scratch buffer, no per-tile dispatch.

TakeAlongAxisKernel

IL-emitted per-element gather kernel for np.take_along_axis. Walks the C-contiguous result with an incremental strided odometer, reading one int64 index and gathering one elemBytes-wide element of the source per step.

TakeFlatKernel

Lean 1-D gather: dst[j] = src[indices[j]] over count elements of one compile-time width, RAISE semantics (a negative index is normalised once, then bounds-checked). indices addresses int32 or int64 values, per the kernel's generation. Returns count, or the first failing position.

TakeKernel

IL-emitted gather kernel for np.take. The source is treated as a 3-D layout (outerSize, maxItem, innerSize-bytes). For each (outer, j) pair the kernel reads indices[j], applies mode, and copies innerSize bytes from the source slab to the destination position.

TakeLeadingKernel

The fancy-index gather along a non-leading axis, in NumPy's MapIter buffer order: for every index j and outer row o, copies innerSize bytes from src[(o * maxItem + indices[j]) * innerSize] to dst[(j * outerSize + o) * innerSize].

TraceKernel

Walks the diagonal of one or more 2-D sub-arrays and accumulates each into the promoted dtype. Per-src dtype kernel; accum / result dtypes are baked into the IL at emit time.

TypedElementReductionKernel<TResult>

Delegate for typed element-wise reduction kernels. Returns the reduced value directly without boxing.

UnaryKernel

Unary operation kernel signature using void pointers. Handles operations where input and output may have different types. Type conversion is handled internally by the generated IL.

UnravelIndexKernel

IL-emitted flat→multi-coord expander for np.unravel_index. Caller pre-casts indices to int64 and computes unravelSize = product of dims; the kernel reads indices linearly, validates each value against [0, unravelSize), and writes element i's coordinates into the row out[indim .. indim+ndim-1] of the interleaved buffer out, using the C-order or F-order extraction direction selected by idxStart / idxStep.

UnravelIndexMagicKernel

The reciprocal twin of UnravelIndexKernel: the same contract, but each peeled dimension is divided by a precomputed invariant reciprocal (Granlund & Montgomery, "Division by Invariant Integers using Multiplication", Fig. 4.1) instead of the hardware divide. Exact for every non-negative index and every POSITIVE dimension; a negative dimension (C's truncating division of a negative running quotient) must take UnravelIndexKernel.

WhereBytesKernel

An IL-emitted element select for opaque elements of a fixed byte width (the width is baked into the kernel): result[i] = cond[i] ? x[i] : y[i] over count contiguous elements.

WhereBytesStepKernel

An IL-emitted element select for opaque elements of a fixed byte width whose INPUTS each advance by their own byte step: result[i] = cond[i·condStep] ? x[i·xStep] : y[i·yStep] over count elements written contiguously. A step of 0 broadcasts a one-element operand; a negative step walks a reversed view.

WhereKernel<T>

Delegate for where operation kernels.

WhereScalarXKernel<T>
WhereScalarXYKernel<T>
WhereScalarYKernel<T>