Class SizeBucketedBufferPool
Thread-safe pool of recently-freed unmanaged buffers, bucketed by exact byte size. Acts as a tcache-like front for Alloc(nuint) / Free(void*): a successful Take is just a pop from a per-size ConcurrentStack<T>; a failed Take falls through to NativeMemory.Alloc.
WHY THIS EXISTS
Profiling NumSharp's binary-op pipeline shows ~500 µs of every
1024×1024 float32 a + b is spent on first-touch overhead of the
fresh output buffer — page-faulting each cache line on first write
plus the kernel-mode cost of Alloc(nuint)
reaching out to the OS for a fresh chunk. NumPy hides the same
cost via glibc tcache reuse: a buffer freed by the previous op is
handed back warm to the next call. This pool replicates that
behaviour at the NumSharp layer.
GC PACING — the promptness gap vs NumPy (Wave 2.5)
NumPy frees a dropped temporary the INSTANT its refcount hits zero,
so the very next allocation reuses the same warm block. NumSharp's
dropped-but-undisposed NDArray returns its buffer only via
the block Disposer's FINALIZER, which runs only after a GC collects
the wrapper graph. In a tight allocating loop the managed garbage
per op is tiny (~a few hundred bytes), so gen-0 collections are
thousands of calls apart — the pool starves and every call pays a
cold NativeMemory.Alloc plus first-touch page faults
(measured: 1K float32 unary 16% hit-rate; 100K ~35 µs/op of soft
faults — 7× the kernel's own time).
The pool therefore PACES the GC: when a Take MISSES (or a bucket is
about to run dry) after at least PaceThresholdBytes
of buffers were handed out since the last paced collection, it
requests ONE forced, blocking, non-compacting gen-0
collection (~4-5 µs on a small heap). That collects the dropped
wrapper graphs (allocated since the last pace → still gen 0), the
finalizer thread drains, and the returns refill the buckets — the
.NET stand-in for CPython's refcount-prompt free, amortized to at
most one gen-0 pause per PaceThresholdBytes of
allocation. Deterministically-disposing workloads (Dispose /
NDScope) keep their buckets stocked and never trip the trigger.
Opt-out: NUMSHARP_POOL_GC_PACING=0; window:
NUMSHARP_POOL_GC_PACING_MB (default 16).
GC PRESSURE — coarse chunks, not per-call (Wave 2.5)
Wave 2.4 registered AddMemoryPressure(long) on every Take and removed it on every Return. Two measured pathologies: the Add/Remove pair costs ~55 ns per op, and the churn drove the GC into CONSTANT full gen-2 collections in allocating loops (100K float32 unary: a full GC every ~24 calls — ~12 µs/op of GC time; every collection observed was gen 2). Pressure now tracks the pool's OUTSTANDING bytes (taken − returned, i.e. what user code actually holds) in PressureChunkBytes chunks: the hot path pays two interlocked adds, and the GC only hears about net growth/shrink at 64 MiB granularity — enough for the issue #501 protection (a process holding gigabytes of native arrays still pushes the GC to collect) without per-op churn. Reclamation promptness inside the window is the pacing trigger's job, not pressure's.
SIZING POLICY
• The window is MinPoolableBytes (1 B) to
MaxPoolableBytes (64 MiB) — Wave 2.4 opened both
ends: the 1000-element float32 result (4000 B) missed the old
4 KiB floor by 96 bytes, and every 4M-element output (16–32 MiB)
missed the old 1 MiB cap, paying ~2× in demand-zero page faults
per call (in-place toggle-verified: P1 contig add 4M 3.37→1.74 ms).
• Above the cap: no pooling. Huge buffers are rare and the memory
cost of keeping them around dwarfs the alloc-cost savings.
• Per-bucket caps are DYNAMIC (Wave 2.5): finalizer-driven returns
arrive in one burst per collection — up to a whole pacing window
of buffers at once. The old flat cap of 8 discarded ~95% of each
burst (measured: 20K-call 1K loop → 3.3K pooled, 16K freed) and
re-starved the loop. A bucket now holds up to one pacing window
of bytes: cap = PaceThresholdBytes / size, clamped to
[MaxBuffersPerBucket .. MaxBuffersPerDynamicBucket]
below LargeBucketThreshold and to
[MaxBuffersPerLargeBucket .. window] at/above it —
so per-bucket resident stays ≈ the pacing window (16 MiB default)
for mid sizes and keeps the old floor of 2 for huge buffers.
• Bucket key is the EXACT byte count requested (no rounding).
Same-size repeated allocs are the dominant pattern in element-
wise ops; rounding to power-of-2 would waste memory and break
exact-fit reuse for typical workloads (e.g. 4 MiB float32 1K×1K).
CORRECTNESS
• Stored buffers are NOT zero-filled. Callers that need zeroed memory must zero on Take (the same contract NativeMemory.Alloc has). • Buffer ownership transfers fully on Take: the pool no longer references the pointer, so subsequent Return calls aren't at risk of double-pop. • Return is best-effort: when the bucket is full or the size falls outside the pool's window the pointer is freed immediately via Free(void*). • Return must be given the size the buffer was taken with: the size selects the bucket AND tells whether the buffer has an alignment header (see ALIGNMENT).
ALIGNMENT (2026-10-10)
A buffer of AlignedMinBytes (one page) or more starts on a BufferAlignment-byte boundary, whether it is fresh, zeroed, reused from a bucket or outside the pooled window. The C allocator aligns to 16 bytes, so half of all buffers used to start in the middle of a 32-byte vector, where every second vector load or store of a SIMD kernel crosses a cache line. A kernel can align ONE of its streams with a scalar prefix; only the allocator can align all of them. Measured on the comparison benchmark (two arrays of 100,000 elements, the result fresh): float64 20.0 -> 15.1 us, float32 10.6 -> 8.5, int32 11.4 -> 9.4, int16 6.4 -> 5.7; 82 of 614 cells at that size ran 10 percent or more faster, and the cells at 1,000 and at 10,000,000 elements did not move. The cost is AlignedOverheadBytes (80) bytes a buffer. The boundary is a cache line, NOT a page: buffers that all start at one page offset put every stream of a many-operand kernel in the same cache set at each step.
public static class SizeBucketedBufferPool
- Inheritance
-
SizeBucketedBufferPool
- Inherited Members
Fields
AlignedMinBytes
Byte size from which a buffer is handed out on a BufferAlignment-byte boundary: one page. A smaller buffer stays in the L1 cache while a kernel works on it, where a vector load that crosses a cache line costs nothing that can be measured, and the header of an aligned buffer (AlignedOverheadBytes) would be a large part of it.
public const long AlignedMinBytes = 4096
Field Value
Remarks
WHY ALIGN AT ALL. The C allocator returns 16-byte aligned blocks, so half of all buffers start in the
middle of a 32-byte vector and every second vector load or store of a kernel then crosses a cache
line. Measured on the comparison kernels with the operands in the L2 cache, ns an element, buffer on
a 64-byte boundary against 16 bytes past one: float64 two arrays 0.141 and 0.250, float32 0.074 and
0.126, uint8 against a value 0.026 and 0.036. Each array won or lost 20 to 78 percent by its address.
NumPy's buffers are 16-byte aligned too (measured: ctypes.data % 64 is uniform over
0 / 16 / 32 / 48 up to 800 KB), so this is a place where NumSharp is the faster library by
construction, not by chance.
AlignedOverheadBytes
Extra bytes one aligned buffer costs: the header and the distance to the next boundary (at most BufferAlignment). 80 bytes on a buffer of at least 4096: under 2 percent.
public const int AlignedOverheadBytes = 80
Field Value
BufferAlignment
Alignment, in bytes, of every buffer of AlignedMinBytes or more: one cache line, which is also a whole number of 128-, 256- and 512-bit vectors.
public const int BufferAlignment = 64
Field Value
GcPacingEnabled
Whether pool-initiated gen-0 pacing is enabled (env NUMSHARP_POOL_GC_PACING, default on).
public static readonly bool GcPacingEnabled
Field Value
GuardPagesEnabled
Opt-in diagnostic page-heap mode (env NUMSHARP_DEBUG_GUARD_PAGES=1, Windows only).
When on, every Take(long)/TakeZeroed(long) hands back a buffer whose
last byte abuts an inaccessible guard page (AllocGuarded(long, out nint)),
pooling is bypassed, and any one-past-the-end write faults INSTANTLY at the offending
access — used to localise an indexing out-of-bounds write to the exact case/site.
Default OFF (the field is read once at startup) so production paths are untouched.
public static readonly bool GuardPagesEnabled
Field Value
LargeBucketThreshold
Bucket sizes at/above this use MaxBuffersPerLargeBucket as their cap FLOOR.
public const long LargeBucketThreshold = 1048576
Field Value
MaxBuffersPerBucket
FLOOR of the dynamic per-bucket cap below LargeBucketThreshold
(the pre-Wave-2.5 flat cap). The effective cap is
clamp(PaceThresholdBytes / size, MaxBuffersPerBucket, MaxBuffersPerDynamicBucket)
— see the SIZING POLICY note on burst-sized buckets.
public const int MaxBuffersPerBucket = 8
Field Value
MaxBuffersPerDynamicBucket
Ceiling of the dynamic per-bucket cap — bounds tiny-size buckets (entries, not bytes).
public const int MaxBuffersPerDynamicBucket = 4096
Field Value
MaxBuffersPerLargeBucket
Per-bucket cap floor for large (≥ 1 MiB) buckets — bounds peak resident memory.
public const int MaxBuffersPerLargeBucket = 2
Field Value
MaxPoolableBytes
Maximum allocation size to pool (bytes). Wave 2.4 raised this from 1 MiB to 64 MiB: the dominant benchmark/e2e shapes (4M elements = 16 MiB float32 / 32 MiB float64 outputs) all missed the old cap and paid ~0.3–0.4 ms of first-touch page faults per call — the "allocator tax" residual on every measured e2e strided row. NumPy gets the same reuse for free from glibc's arena caching. Resident growth is bounded by the per-bucket cap.
public const long MaxPoolableBytes = 67108864
Field Value
MinPoolableBytes
Minimum allocation size to pool (bytes). Wave 2.4 lowered this from 4096 to 1: the small-N hot path (e.g. a 1000-element float32 ufunc result = 4000 bytes) sat just under the old threshold and paid a fresh NativeMemory.Alloc + GC memory pressure pair on EVERY call. Tiny buckets cost almost nothing resident and a pool hit skips the cold alloc entirely.
public const long MinPoolableBytes = 1
Field Value
PaceThresholdBytes
The pacing window in bytes (env NUMSHARP_POOL_GC_PACING_MB, default 16 MiB, clamped
1..1024 MiB): at most one pool-initiated gen-0 collection per this many bytes handed out,
and the per-bucket warm-set byte budget.
public static readonly long PaceThresholdBytes
Field Value
PressureChunkBytes
Granularity of the pool's AddMemoryPressure(long) accounting. The pool tells the GC about net OUTSTANDING native bytes (taken − returned) in whole chunks of this size, so the per-op hot path never calls the pressure APIs (see the GC PRESSURE header note).
public const long PressureChunkBytes = 67108864
Field Value
Properties
Hits
How many Take calls served from the pool.
public static long Hits { get; }
Property Value
Misses
How many Take calls fell through to NativeMemory.Alloc.
public static long Misses { get; }
Property Value
PacedCollections
How many gen-0 collections the pool has requested (pacing; see the class header).
public static long PacedCollections { get; }
Property Value
Returns
How many Return calls accepted the buffer into the pool.
public static long Returns { get; }
Property Value
ReturnsFreed
How many Return calls freed the buffer (bucket full / out-of-range).
public static long ReturnsFreed { get; }
Property Value
ZeroedAllocs
How many TakeZeroed calls went straight to calloc (the np.zeros fast path).
public static long ZeroedAllocs { get; }
Property Value
Methods
BookForeign(long)
Books native memory that NumSharp keeps alive or hands out but did NOT allocate - an ONNX Runtime tensor a caller received from NumSharp, a numpy or ONNX Runtime buffer under a NumSharp view - so the GC hears about it the way it hears about NumSharp's own buffers: through the coarse PressureChunkBytes chunks, and through ONE paced gen-0 collection each time the booked bytes GROW by a pacing window (PaceThresholdBytes) since the last such collection.
public static void BookForeign(long bytes)
Parameters
byteslongThe bytes to book (nothing for zero or less). Give exactly this count back with ReturnForeign(long) when the memory is no longer kept alive by NumSharp.
Remarks
Why growth, not volume. A holder that the caller disposes gives its booking back at once, so code that disposes what it books never grows the balance and never pays for a collection. A holder dropped undisposed keeps its booking until its finalizer runs, which needs a collection the GC would not otherwise make (the holder is a few dozen managed bytes): the growth trigger makes that collection after a window's worth of dropped bytes, and the finalizers it queues give the bookings back.
Why not AddMemoryPressure(long) per holder. The GC counts every addition toward its next full collection even after the matching removal, so a loop that books and gives back at once still ran 45 gen-2 collections per 5,000 views of 602 KB (ONNX Runtime owning views, measured 2026-11-06); booked here, none.
Not a hand-out. Unlike AccountForeignAllocation(long), a booking does not count toward the pool's own pacing window (the bytes handed out since the last paced collection): a booking given back at once is not memory the pool should collect for, and counting it made the next ordinary pool miss pace a collection in a loop that disposed every view (caught by the pythonnet bridge's lease test). Only the outstanding total and its pressure chunks move.
The growth test is a heuristic over two interlocked counters: concurrent bookings can move WHICH call paces, never whether the books balance.
For a bridge package. Book when your holder (a view, a lease, a wrapper) starts keeping the foreign
memory alive and give the same count back exactly once when it stops - in Dispose AND in its finalizer,
guarded so only one of them returns it. Footgun: a booking never given back stays in the outstanding total
for the life of the process, keeps its pressure chunks registered and makes every later growth window pace a
collection sooner; a give-back without its booking drives the counters below the truth.
Clear()
Drain every pooled buffer immediately (testing / memory pressure). Calls Free(void*) on each. No pressure adjustment — pooled buffers carry none (they are not outstanding).
public static void Clear()
ResetCounters()
Reset all counters. Diagnostic only.
public static void ResetCounters()
Return(nint, long)
Return a buffer to the pool. Caller transfers ownership; do NOT touch the pointer after the call.
If the size falls outside the pool window or the bucket is already at capacity, the buffer is freed via Free(void*) instead of being kept.
public static void Return(nint ptr, long bytes)
Parameters
ptrnintPointer obtained from Take(long) or TakeZeroed(long). A pointer from another allocator must not be returned here: from AlignedMinBytes up the pool reads its own header in front of the buffer.
byteslongSize in bytes originally requested. It must be the size passed to the Take call: it selects the bucket and tells whether the buffer has an alignment header.
ReturnForeign(long)
Gives back one BookForeign(long) booking. Touches only counters (no lock, no allocation), so it is safe from a finalizer thread and from native callbacks.
public static void ReturnForeign(long bytes)
Parameters
byteslongThe count the matching BookForeign(long) booked (nothing for zero or less).
Remarks
Call it once per booking: the counters cannot tell a second give-back from a real one, so a double return
(a Dispose and a finalizer both returning) leaves the outstanding total below what is really held.
Take(long)
Take a buffer of the given byte size. Returns either a reused warm buffer or a fresh allocation; either way the caller owns it and must eventually Return(nint, long) it with the same byte size (never free it directly: from AlignedMinBytes up the pointer is not the C allocator's own). The memory is NOT zeroed.
public static nint Take(long bytes)
Parameters
byteslongByte size of the buffer. Must be > 0.
Returns
TakeZeroed(long)
Take a ZERO-INITIALIZED buffer of the given byte size. This is
the np.zeros / fill-with-default fast path and the analogue of
NumPy's npy_alloc_cache_zero / PyDataMem_NEW_ZEROED.
WHY calloc INSTEAD OF Take + memset
AllocZeroed(nuint) calls the CRT
calloc. For any non-trivial size the CRT/OS serves the
request from fresh, copy-on-write zero pages (Windows
VirtualAlloc / Linux mmap(MAP_ANONYMOUS)): the
pages are only physically committed and zeroed by the kernel
lazily, on first write. So zeroing a 10M-element (80 MB) block
costs ~0.01 ms instead of the ~14 ms an explicit element fill —
or even a Clear(void*, nuint) memset (~21 ms) —
pays to touch every page up front.
The dirty same-size bucket cache used by Take(long) is intentionally NOT consulted: a recycled buffer is dirty and would force a full memset, touching every page and throwing away the entire lazy-zero advantage for exactly the large sizes that matter. NumPy makes the same call — its zero cache only engages below 1 KiB, a regime where the CRT's own low- fragmentation heap already supplies that small-block reuse for our calloc.
OWNERSHIP & PRESSURE
The caller owns the returned pointer and must eventually Return(nint, long) it with the same byte size. A calloc'd buffer is laid out like one of Take(long) (aligned, with the same header, from AlignedMinBytes up), so Return may pool it for later (non-zero) reuse by Take(long) or free it. Outstanding-byte accounting is registered here exactly as Take(long) does, so the paired Return(nint, long) balances it, and the pacing trigger keeps dropped np.zeros graphs collected promptly.
public static nint TakeZeroed(long bytes)
Parameters
byteslongByte size of the buffer. Must be >= 0.