Class SizeBucketedBufferPool
Thread-safe pool of recently-freed unmanaged buffers, bucketed by exact byte size. Acts as a tcache-like front for Alloc(nuint) / Free(void*): a successful Take is just a pop from a per-size ConcurrentStack<T>; a failed Take falls through to NativeMemory.Alloc.
WHY THIS EXISTS
Profiling NumSharp's binary-op pipeline shows ~500 µs of every
1024×1024 float32 a + b is spent on first-touch overhead of the
fresh output buffer — page-faulting each cache line on first write
plus the kernel-mode cost of Alloc(nuint)
reaching out to the OS for a fresh chunk. NumPy hides the same
cost via glibc tcache reuse: a buffer freed by the previous op is
handed back warm to the next call. This pool replicates that
behaviour at the NumSharp layer.
GC PACING — the promptness gap vs NumPy (Wave 2.5)
NumPy frees a dropped temporary the INSTANT its refcount hits zero,
so the very next allocation reuses the same warm block. NumSharp's
dropped-but-undisposed NDArray returns its buffer only via
the block Disposer's FINALIZER, which runs only after a GC collects
the wrapper graph. In a tight allocating loop the managed garbage
per op is tiny (~a few hundred bytes), so gen-0 collections are
thousands of calls apart — the pool starves and every call pays a
cold NativeMemory.Alloc plus first-touch page faults
(measured: 1K float32 unary 16% hit-rate; 100K ~35 µs/op of soft
faults — 7× the kernel's own time).
The pool therefore PACES the GC: when a Take MISSES (or a bucket is
about to run dry) after at least PaceThresholdBytes
of buffers were handed out since the last paced collection, it
requests ONE forced, blocking, non-compacting gen-0
collection (~4-5 µs on a small heap). That collects the dropped
wrapper graphs (allocated since the last pace → still gen 0), the
finalizer thread drains, and the returns refill the buckets — the
.NET stand-in for CPython's refcount-prompt free, amortized to at
most one gen-0 pause per PaceThresholdBytes of
allocation. Deterministically-disposing workloads (Dispose /
NDScope) keep their buckets stocked and never trip the trigger.
Opt-out: NUMSHARP_POOL_GC_PACING=0; window:
NUMSHARP_POOL_GC_PACING_MB (default 16).
GC PRESSURE — coarse chunks, not per-call (Wave 2.5)
Wave 2.4 registered AddMemoryPressure(long) on every Take and removed it on every Return. Two measured pathologies: the Add/Remove pair costs ~55 ns per op, and the churn drove the GC into CONSTANT full gen-2 collections in allocating loops (100K float32 unary: a full GC every ~24 calls — ~12 µs/op of GC time; every collection observed was gen 2). Pressure now tracks the pool's OUTSTANDING bytes (taken − returned, i.e. what user code actually holds) in PressureChunkBytes chunks: the hot path pays two interlocked adds, and the GC only hears about net growth/shrink at 64 MiB granularity — enough for the issue #501 protection (a process holding gigabytes of native arrays still pushes the GC to collect) without per-op churn. Reclamation promptness inside the window is the pacing trigger's job, not pressure's.
SIZING POLICY
• The window is MinPoolableBytes (1 B) to
MaxPoolableBytes (64 MiB) — Wave 2.4 opened both
ends: the 1000-element float32 result (4000 B) missed the old
4 KiB floor by 96 bytes, and every 4M-element output (16–32 MiB)
missed the old 1 MiB cap, paying ~2× in demand-zero page faults
per call (in-place toggle-verified: P1 contig add 4M 3.37→1.74 ms).
• Above the cap: no pooling. Huge buffers are rare and the memory
cost of keeping them around dwarfs the alloc-cost savings.
• Per-bucket caps are DYNAMIC (Wave 2.5): finalizer-driven returns
arrive in one burst per collection — up to a whole pacing window
of buffers at once. The old flat cap of 8 discarded ~95% of each
burst (measured: 20K-call 1K loop → 3.3K pooled, 16K freed) and
re-starved the loop. A bucket now holds up to one pacing window
of bytes: cap = PaceThresholdBytes / size, clamped to
[MaxBuffersPerBucket .. MaxBuffersPerDynamicBucket]
below LargeBucketThreshold and to
[MaxBuffersPerLargeBucket .. window] at/above it —
so per-bucket resident stays ≈ the pacing window (16 MiB default)
for mid sizes and keeps the old floor of 2 for huge buffers.
• Bucket key is the EXACT byte count requested (no rounding).
Same-size repeated allocs are the dominant pattern in element-
wise ops; rounding to power-of-2 would waste memory and break
exact-fit reuse for typical workloads (e.g. 4 MiB float32 1K×1K).
CORRECTNESS
• Stored buffers are NOT zero-filled. Callers that need zeroed memory must zero on Take (the same contract NativeMemory.Alloc has). • Buffer ownership transfers fully on Take: the pool no longer references the pointer, so subsequent Return calls aren't at risk of double-pop. • Return is best-effort: when the bucket is full or the size falls outside the pool's window the pointer is freed immediately via Free(void*).
public static class SizeBucketedBufferPool- Inheritance
-
SizeBucketedBufferPool
- Inherited Members
Fields
GcPacingEnabled
Whether pool-initiated gen-0 pacing is enabled (env NUMSHARP_POOL_GC_PACING, default on).
public static readonly bool GcPacingEnabledField Value
GuardPagesEnabled
Opt-in diagnostic page-heap mode (env NUMSHARP_DEBUG_GUARD_PAGES=1, Windows only).
When on, every Take(long)/TakeZeroed(long) hands back a buffer whose
last byte abuts an inaccessible guard page (AllocGuarded(long, out nint)),
pooling is bypassed, and any one-past-the-end write faults INSTANTLY at the offending
access — used to localise an indexing out-of-bounds write to the exact case/site.
Default OFF (the field is read once at startup) so production paths are untouched.
public static readonly bool GuardPagesEnabledField Value
LargeBucketThreshold
Bucket sizes at/above this use MaxBuffersPerLargeBucket as their cap FLOOR.
public const long LargeBucketThreshold = 1048576Field Value
MaxBuffersPerBucket
FLOOR of the dynamic per-bucket cap below LargeBucketThreshold
(the pre-Wave-2.5 flat cap). The effective cap is
clamp(PaceThresholdBytes / size, MaxBuffersPerBucket, MaxBuffersPerDynamicBucket)
— see the SIZING POLICY note on burst-sized buckets.
public const int MaxBuffersPerBucket = 8Field Value
MaxBuffersPerDynamicBucket
Ceiling of the dynamic per-bucket cap — bounds tiny-size buckets (entries, not bytes).
public const int MaxBuffersPerDynamicBucket = 4096Field Value
MaxBuffersPerLargeBucket
Per-bucket cap floor for large (≥ 1 MiB) buckets — bounds peak resident memory.
public const int MaxBuffersPerLargeBucket = 2Field Value
MaxPoolableBytes
Maximum allocation size to pool (bytes). Wave 2.4 raised this from 1 MiB to 64 MiB: the dominant benchmark/e2e shapes (4M elements = 16 MiB float32 / 32 MiB float64 outputs) all missed the old cap and paid ~0.3–0.4 ms of first-touch page faults per call — the "allocator tax" residual on every measured e2e strided row. NumPy gets the same reuse for free from glibc's arena caching. Resident growth is bounded by the per-bucket cap.
public const long MaxPoolableBytes = 67108864Field Value
MinPoolableBytes
Minimum allocation size to pool (bytes). Wave 2.4 lowered this from 4096 to 1: the small-N hot path (e.g. a 1000-element float32 ufunc result = 4000 bytes) sat just under the old threshold and paid a fresh NativeMemory.Alloc + GC memory pressure pair on EVERY call. Tiny buckets cost almost nothing resident and a pool hit skips the cold alloc entirely.
public const long MinPoolableBytes = 1Field Value
PaceThresholdBytes
The pacing window in bytes (env NUMSHARP_POOL_GC_PACING_MB, default 16 MiB, clamped
1..1024 MiB): at most one pool-initiated gen-0 collection per this many bytes handed out,
and the per-bucket warm-set byte budget.
public static readonly long PaceThresholdBytesField Value
PressureChunkBytes
Granularity of the pool's AddMemoryPressure(long) accounting. The pool tells the GC about net OUTSTANDING native bytes (taken − returned) in whole chunks of this size, so the per-op hot path never calls the pressure APIs (see the GC PRESSURE header note).
public const long PressureChunkBytes = 67108864Field Value
Properties
Hits
How many Take calls served from the pool.
public static long Hits { get; }Property Value
Misses
How many Take calls fell through to NativeMemory.Alloc.
public static long Misses { get; }Property Value
PacedCollections
How many gen-0 collections the pool has requested (pacing; see the class header).
public static long PacedCollections { get; }Property Value
Returns
How many Return calls accepted the buffer into the pool.
public static long Returns { get; }Property Value
ReturnsFreed
How many Return calls freed the buffer (bucket full / out-of-range).
public static long ReturnsFreed { get; }Property Value
ZeroedAllocs
How many TakeZeroed calls went straight to calloc (the np.zeros fast path).
public static long ZeroedAllocs { get; }Property Value
Methods
Clear()
Drain every pooled buffer immediately (testing / memory pressure). Calls Free(void*) on each. No pressure adjustment — pooled buffers carry none (they are not outstanding).
public static void Clear()ResetCounters()
Reset all counters. Diagnostic only.
public static void ResetCounters()Return(nint, long)
Return a buffer to the pool. Caller transfers ownership; do NOT touch the pointer after the call.
If the size falls outside the pool window or the bucket is already at capacity, the buffer is freed via Free(void*) instead of being kept.
public static void Return(nint ptr, long bytes)Parameters
ptrnintPointer obtained from Take(long) or a paired NativeMemory.Alloc.
byteslongSize in bytes originally requested.
Take(long)
Take a buffer of the given byte size. Returns either a reused warm buffer or a fresh allocation; either way the caller owns it and must eventually Return or Free it. The memory is NOT zeroed.
public static nint Take(long bytes)Parameters
byteslongByte size of the buffer. Must be > 0.
Returns
TakeZeroed(long)
Take a ZERO-INITIALIZED buffer of the given byte size. This is
the np.zeros / fill-with-default fast path and the analogue of
NumPy's npy_alloc_cache_zero / PyDataMem_NEW_ZEROED.
WHY calloc INSTEAD OF Take + memset
AllocZeroed(nuint) calls the CRT
calloc. For any non-trivial size the CRT/OS serves the
request from fresh, copy-on-write zero pages (Windows
VirtualAlloc / Linux mmap(MAP_ANONYMOUS)): the
pages are only physically committed and zeroed by the kernel
lazily, on first write. So zeroing a 10M-element (80 MB) block
costs ~0.01 ms instead of the ~14 ms an explicit element fill —
or even a Clear(void*, nuint) memset (~21 ms) —
pays to touch every page up front.
The dirty same-size bucket cache used by Take(long) is intentionally NOT consulted: a recycled buffer is dirty and would force a full memset, touching every page and throwing away the entire lazy-zero advantage for exactly the large sizes that matter. NumPy makes the same call — its zero cache only engages below 1 KiB, a regime where the CRT's own low- fragmentation heap already supplies that small-block reuse for our calloc.
OWNERSHIP & PRESSURE
The caller owns the returned pointer and must eventually Return(nint, long) or free it; a calloc'd pointer is a normal NativeMemory allocation, so Return may pool it for later (non-zero) reuse by Take(long) or free it via Free(void*). Outstanding-byte accounting is registered here exactly as Take(long) does, so the paired Return(nint, long) balances it, and the pacing trigger keeps dropped np.zeros graphs collected promptly.
public static nint TakeZeroed(long bytes)Parameters
byteslongByte size of the buffer. Must be >= 0.