Table of Contents

Class SizeBucketedBufferPool

Namespace
NumSharp.Backends.Unmanaged.Pooling
Assembly
NumSharp.dll

Thread-safe pool of recently-freed unmanaged buffers, bucketed by exact byte size. Acts as a tcache-like front for Alloc(nuint) / Free(void*): a successful Take is just a pop from a per-size ConcurrentStack<T>; a failed Take falls through to NativeMemory.Alloc.

WHY THIS EXISTS

Profiling NumSharp's binary-op pipeline shows ~500 µs of every 1024×1024 float32 a + b is spent on first-touch overhead of the fresh output buffer — page-faulting each cache line on first write plus the kernel-mode cost of Alloc(nuint) reaching out to the OS for a fresh chunk. NumPy hides the same cost via glibc tcache reuse: a buffer freed by the previous op is handed back warm to the next call. This pool replicates that behaviour at the NumSharp layer.

GC PACING — the promptness gap vs NumPy (Wave 2.5)

NumPy frees a dropped temporary the INSTANT its refcount hits zero, so the very next allocation reuses the same warm block. NumSharp's dropped-but-undisposed NDArray returns its buffer only via the block Disposer's FINALIZER, which runs only after a GC collects the wrapper graph. In a tight allocating loop the managed garbage per op is tiny (~a few hundred bytes), so gen-0 collections are thousands of calls apart — the pool starves and every call pays a cold NativeMemory.Alloc plus first-touch page faults (measured: 1K float32 unary 16% hit-rate; 100K ~35 µs/op of soft faults — 7× the kernel's own time).

The pool therefore PACES the GC: when a Take MISSES (or a bucket is about to run dry) after at least PaceThresholdBytes of buffers were handed out since the last paced collection, it requests ONE forced, blocking, non-compacting gen-0 collection (~4-5 µs on a small heap). That collects the dropped wrapper graphs (allocated since the last pace → still gen 0), the finalizer thread drains, and the returns refill the buckets — the .NET stand-in for CPython's refcount-prompt free, amortized to at most one gen-0 pause per PaceThresholdBytes of allocation. Deterministically-disposing workloads (Dispose / NDScope) keep their buckets stocked and never trip the trigger. Opt-out: NUMSHARP_POOL_GC_PACING=0; window: NUMSHARP_POOL_GC_PACING_MB (default 16).

GC PRESSURE — coarse chunks, not per-call (Wave 2.5)

Wave 2.4 registered AddMemoryPressure(long) on every Take and removed it on every Return. Two measured pathologies: the Add/Remove pair costs ~55 ns per op, and the churn drove the GC into CONSTANT full gen-2 collections in allocating loops (100K float32 unary: a full GC every ~24 calls — ~12 µs/op of GC time; every collection observed was gen 2). Pressure now tracks the pool's OUTSTANDING bytes (taken − returned, i.e. what user code actually holds) in PressureChunkBytes chunks: the hot path pays two interlocked adds, and the GC only hears about net growth/shrink at 64 MiB granularity — enough for the issue #501 protection (a process holding gigabytes of native arrays still pushes the GC to collect) without per-op churn. Reclamation promptness inside the window is the pacing trigger's job, not pressure's.

SIZING POLICY

• The window is MinPoolableBytes (1 B) to MaxPoolableBytes (64 MiB) — Wave 2.4 opened both ends: the 1000-element float32 result (4000 B) missed the old 4 KiB floor by 96 bytes, and every 4M-element output (16–32 MiB) missed the old 1 MiB cap, paying ~2× in demand-zero page faults per call (in-place toggle-verified: P1 contig add 4M 3.37→1.74 ms). • Above the cap: no pooling. Huge buffers are rare and the memory cost of keeping them around dwarfs the alloc-cost savings. • Per-bucket caps are DYNAMIC (Wave 2.5): finalizer-driven returns arrive in one burst per collection — up to a whole pacing window of buffers at once. The old flat cap of 8 discarded ~95% of each burst (measured: 20K-call 1K loop → 3.3K pooled, 16K freed) and re-starved the loop. A bucket now holds up to one pacing window of bytes: cap = PaceThresholdBytes / size, clamped to [MaxBuffersPerBucket .. MaxBuffersPerDynamicBucket] below LargeBucketThreshold and to [MaxBuffersPerLargeBucket .. window] at/above it — so per-bucket resident stays ≈ the pacing window (16 MiB default) for mid sizes and keeps the old floor of 2 for huge buffers. • Bucket key is the EXACT byte count requested (no rounding). Same-size repeated allocs are the dominant pattern in element- wise ops; rounding to power-of-2 would waste memory and break exact-fit reuse for typical workloads (e.g. 4 MiB float32 1K×1K).

CORRECTNESS

• Stored buffers are NOT zero-filled. Callers that need zeroed memory must zero on Take (the same contract NativeMemory.Alloc has). • Buffer ownership transfers fully on Take: the pool no longer references the pointer, so subsequent Return calls aren't at risk of double-pop. • Return is best-effort: when the bucket is full or the size falls outside the pool's window the pointer is freed immediately via Free(void*). • Return must be given the size the buffer was taken with: the size selects the bucket AND tells whether the buffer has an alignment header (see ALIGNMENT).

ALIGNMENT (2026-10-10)

A buffer of AlignedMinBytes (one page) or more starts on a BufferAlignment-byte boundary, whether it is fresh, zeroed, reused from a bucket or outside the pooled window. The C allocator aligns to 16 bytes, so half of all buffers used to start in the middle of a 32-byte vector, where every second vector load or store of a SIMD kernel crosses a cache line. A kernel can align ONE of its streams with a scalar prefix; only the allocator can align all of them. Measured on the comparison benchmark (two arrays of 100,000 elements, the result fresh): float64 20.0 -> 15.1 us, float32 10.6 -> 8.5, int32 11.4 -> 9.4, int16 6.4 -> 5.7; 82 of 614 cells at that size ran 10 percent or more faster, and the cells at 1,000 and at 10,000,000 elements did not move. The cost is AlignedOverheadBytes (80) bytes a buffer. The boundary is a cache line, NOT a page: buffers that all start at one page offset put every stream of a many-operand kernel in the same cache set at each step.

public static class SizeBucketedBufferPool
Inheritance
SizeBucketedBufferPool
Inherited Members

Fields

AlignedMinBytes

Byte size from which a buffer is handed out on a BufferAlignment-byte boundary: one page. A smaller buffer stays in the L1 cache while a kernel works on it, where a vector load that crosses a cache line costs nothing that can be measured, and the header of an aligned buffer (AlignedOverheadBytes) would be a large part of it.

public const long AlignedMinBytes = 4096

Field Value

long

Remarks

WHY ALIGN AT ALL. The C allocator returns 16-byte aligned blocks, so half of all buffers start in the middle of a 32-byte vector and every second vector load or store of a kernel then crosses a cache line. Measured on the comparison kernels with the operands in the L2 cache, ns an element, buffer on a 64-byte boundary against 16 bytes past one: float64 two arrays 0.141 and 0.250, float32 0.074 and 0.126, uint8 against a value 0.026 and 0.036. Each array won or lost 20 to 78 percent by its address. NumPy's buffers are 16-byte aligned too (measured: ctypes.data % 64 is uniform over 0 / 16 / 32 / 48 up to 800 KB), so this is a place where NumSharp is the faster library by construction, not by chance.

AlignedOverheadBytes

Extra bytes one aligned buffer costs: the header and the distance to the next boundary (at most BufferAlignment). 80 bytes on a buffer of at least 4096: under 2 percent.

public const int AlignedOverheadBytes = 80

Field Value

int

BufferAlignment

Alignment, in bytes, of every buffer of AlignedMinBytes or more: one cache line, which is also a whole number of 128-, 256- and 512-bit vectors.

public const int BufferAlignment = 64

Field Value

int

GcPacingEnabled

Whether pool-initiated gen-0 pacing is enabled (env NUMSHARP_POOL_GC_PACING, default on).

public static readonly bool GcPacingEnabled

Field Value

bool

GuardPagesEnabled

Opt-in diagnostic page-heap mode (env NUMSHARP_DEBUG_GUARD_PAGES=1, Windows only). When on, every Take(long)/TakeZeroed(long) hands back a buffer whose last byte abuts an inaccessible guard page (AllocGuarded(long, out nint)), pooling is bypassed, and any one-past-the-end write faults INSTANTLY at the offending access — used to localise an indexing out-of-bounds write to the exact case/site. Default OFF (the field is read once at startup) so production paths are untouched.

public static readonly bool GuardPagesEnabled

Field Value

bool

LargeBucketThreshold

Bucket sizes at/above this use MaxBuffersPerLargeBucket as their cap FLOOR.

public const long LargeBucketThreshold = 1048576

Field Value

long

MaxBuffersPerBucket

FLOOR of the dynamic per-bucket cap below LargeBucketThreshold (the pre-Wave-2.5 flat cap). The effective cap is clamp(PaceThresholdBytes / size, MaxBuffersPerBucket, MaxBuffersPerDynamicBucket) — see the SIZING POLICY note on burst-sized buckets.

public const int MaxBuffersPerBucket = 8

Field Value

int

MaxBuffersPerDynamicBucket

Ceiling of the dynamic per-bucket cap — bounds tiny-size buckets (entries, not bytes).

public const int MaxBuffersPerDynamicBucket = 4096

Field Value

int

MaxBuffersPerLargeBucket

Per-bucket cap floor for large (≥ 1 MiB) buckets — bounds peak resident memory.

public const int MaxBuffersPerLargeBucket = 2

Field Value

int

MaxPoolableBytes

Maximum allocation size to pool (bytes). Wave 2.4 raised this from 1 MiB to 64 MiB: the dominant benchmark/e2e shapes (4M elements = 16 MiB float32 / 32 MiB float64 outputs) all missed the old cap and paid ~0.3–0.4 ms of first-touch page faults per call — the "allocator tax" residual on every measured e2e strided row. NumPy gets the same reuse for free from glibc's arena caching. Resident growth is bounded by the per-bucket cap.

public const long MaxPoolableBytes = 67108864

Field Value

long

MinPoolableBytes

Minimum allocation size to pool (bytes). Wave 2.4 lowered this from 4096 to 1: the small-N hot path (e.g. a 1000-element float32 ufunc result = 4000 bytes) sat just under the old threshold and paid a fresh NativeMemory.Alloc + GC memory pressure pair on EVERY call. Tiny buckets cost almost nothing resident and a pool hit skips the cold alloc entirely.

public const long MinPoolableBytes = 1

Field Value

long

PaceThresholdBytes

The pacing window in bytes (env NUMSHARP_POOL_GC_PACING_MB, default 16 MiB, clamped 1..1024 MiB): at most one pool-initiated gen-0 collection per this many bytes handed out, and the per-bucket warm-set byte budget.

public static readonly long PaceThresholdBytes

Field Value

long

PressureChunkBytes

Granularity of the pool's AddMemoryPressure(long) accounting. The pool tells the GC about net OUTSTANDING native bytes (taken − returned) in whole chunks of this size, so the per-op hot path never calls the pressure APIs (see the GC PRESSURE header note).

public const long PressureChunkBytes = 67108864

Field Value

long

Properties

Hits

How many Take calls served from the pool.

public static long Hits { get; }

Property Value

long

Misses

How many Take calls fell through to NativeMemory.Alloc.

public static long Misses { get; }

Property Value

long

PacedCollections

How many gen-0 collections the pool has requested (pacing; see the class header).

public static long PacedCollections { get; }

Property Value

long

Returns

How many Return calls accepted the buffer into the pool.

public static long Returns { get; }

Property Value

long

ReturnsFreed

How many Return calls freed the buffer (bucket full / out-of-range).

public static long ReturnsFreed { get; }

Property Value

long

ZeroedAllocs

How many TakeZeroed calls went straight to calloc (the np.zeros fast path).

public static long ZeroedAllocs { get; }

Property Value

long

Methods

BookForeign(long)

Books native memory that NumSharp keeps alive or hands out but did NOT allocate - an ONNX Runtime tensor a caller received from NumSharp, a numpy or ONNX Runtime buffer under a NumSharp view - so the GC hears about it the way it hears about NumSharp's own buffers: through the coarse PressureChunkBytes chunks, and through ONE paced gen-0 collection each time the booked bytes GROW by a pacing window (PaceThresholdBytes) since the last such collection.

public static void BookForeign(long bytes)

Parameters

bytes long

The bytes to book (nothing for zero or less). Give exactly this count back with ReturnForeign(long) when the memory is no longer kept alive by NumSharp.

Remarks

Why growth, not volume. A holder that the caller disposes gives its booking back at once, so code that disposes what it books never grows the balance and never pays for a collection. A holder dropped undisposed keeps its booking until its finalizer runs, which needs a collection the GC would not otherwise make (the holder is a few dozen managed bytes): the growth trigger makes that collection after a window's worth of dropped bytes, and the finalizers it queues give the bookings back.

Why not AddMemoryPressure(long) per holder. The GC counts every addition toward its next full collection even after the matching removal, so a loop that books and gives back at once still ran 45 gen-2 collections per 5,000 views of 602 KB (ONNX Runtime owning views, measured 2026-11-06); booked here, none.

Not a hand-out. Unlike AccountForeignAllocation(long), a booking does not count toward the pool's own pacing window (the bytes handed out since the last paced collection): a booking given back at once is not memory the pool should collect for, and counting it made the next ordinary pool miss pace a collection in a loop that disposed every view (caught by the pythonnet bridge's lease test). Only the outstanding total and its pressure chunks move.

The growth test is a heuristic over two interlocked counters: concurrent bookings can move WHICH call paces, never whether the books balance.

For a bridge package. Book when your holder (a view, a lease, a wrapper) starts keeping the foreign memory alive and give the same count back exactly once when it stops - in Dispose AND in its finalizer, guarded so only one of them returns it. Footgun: a booking never given back stays in the outstanding total for the life of the process, keeps its pressure chunks registered and makes every later growth window pace a collection sooner; a give-back without its booking drives the counters below the truth.

Clear()

Drain every pooled buffer immediately (testing / memory pressure). Calls Free(void*) on each. No pressure adjustment — pooled buffers carry none (they are not outstanding).

public static void Clear()

ResetCounters()

Reset all counters. Diagnostic only.

public static void ResetCounters()

Return(nint, long)

Return a buffer to the pool. Caller transfers ownership; do NOT touch the pointer after the call.

If the size falls outside the pool window or the bucket is already at capacity, the buffer is freed via Free(void*) instead of being kept.

public static void Return(nint ptr, long bytes)

Parameters

ptr nint

Pointer obtained from Take(long) or TakeZeroed(long). A pointer from another allocator must not be returned here: from AlignedMinBytes up the pool reads its own header in front of the buffer.

bytes long

Size in bytes originally requested. It must be the size passed to the Take call: it selects the bucket and tells whether the buffer has an alignment header.

ReturnForeign(long)

Gives back one BookForeign(long) booking. Touches only counters (no lock, no allocation), so it is safe from a finalizer thread and from native callbacks.

public static void ReturnForeign(long bytes)

Parameters

bytes long

The count the matching BookForeign(long) booked (nothing for zero or less).

Remarks

Call it once per booking: the counters cannot tell a second give-back from a real one, so a double return (a Dispose and a finalizer both returning) leaves the outstanding total below what is really held.

Take(long)

Take a buffer of the given byte size. Returns either a reused warm buffer or a fresh allocation; either way the caller owns it and must eventually Return(nint, long) it with the same byte size (never free it directly: from AlignedMinBytes up the pointer is not the C allocator's own). The memory is NOT zeroed.

public static nint Take(long bytes)

Parameters

bytes long

Byte size of the buffer. Must be > 0.

Returns

nint

TakeZeroed(long)

Take a ZERO-INITIALIZED buffer of the given byte size. This is the np.zeros / fill-with-default fast path and the analogue of NumPy's npy_alloc_cache_zero / PyDataMem_NEW_ZEROED.

WHY calloc INSTEAD OF Take + memset

AllocZeroed(nuint) calls the CRT calloc. For any non-trivial size the CRT/OS serves the request from fresh, copy-on-write zero pages (Windows VirtualAlloc / Linux mmap(MAP_ANONYMOUS)): the pages are only physically committed and zeroed by the kernel lazily, on first write. So zeroing a 10M-element (80 MB) block costs ~0.01 ms instead of the ~14 ms an explicit element fill — or even a Clear(void*, nuint) memset (~21 ms) — pays to touch every page up front.

The dirty same-size bucket cache used by Take(long) is intentionally NOT consulted: a recycled buffer is dirty and would force a full memset, touching every page and throwing away the entire lazy-zero advantage for exactly the large sizes that matter. NumPy makes the same call — its zero cache only engages below 1 KiB, a regime where the CRT's own low- fragmentation heap already supplies that small-block reuse for our calloc.

OWNERSHIP & PRESSURE

The caller owns the returned pointer and must eventually Return(nint, long) it with the same byte size. A calloc'd buffer is laid out like one of Take(long) (aligned, with the same header, from AlignedMinBytes up), so Return may pool it for later (non-zero) reuse by Take(long) or free it. Outstanding-byte accounting is registered here exactly as Take(long) does, so the paired Return(nint, long) balances it, and the pacing trigger keeps dropped np.zeros graphs collected promptly.

public static nint TakeZeroed(long bytes)

Parameters

bytes long

Byte size of the buffer. Must be >= 0.

Returns

nint