Namespace NumSharp.Backends.Unmanaged.Pooling

Classes

Thread-safe pool of recently-freed unmanaged buffers, bucketed by exact byte size. Acts as a tcache-like front for Alloc(nuint) / Free(void*): a successful Take is just a pop from a per-size ConcurrentStack<T>; a failed Take falls through to NativeMemory.Alloc.

WHY THIS EXISTS

Profiling NumSharp's binary-op pipeline shows ~500 µs of every 1024×1024 float32 a + b is spent on first-touch overhead of the fresh output buffer — page-faulting each cache line on first write plus the kernel-mode cost of Alloc(nuint) reaching out to the OS for a fresh chunk. NumPy hides the same cost via glibc tcache reuse: a buffer freed by the previous op is handed back warm to the next call. This pool replicates that behaviour at the NumSharp layer.

GC PACING — the promptness gap vs NumPy (Wave 2.5)

NumPy frees a dropped temporary the INSTANT its refcount hits zero, so the very next allocation reuses the same warm block. NumSharp's dropped-but-undisposed NDArray returns its buffer only via the block Disposer's FINALIZER, which runs only after a GC collects the wrapper graph. In a tight allocating loop the managed garbage per op is tiny (~a few hundred bytes), so gen-0 collections are thousands of calls apart — the pool starves and every call pays a cold NativeMemory.Alloc plus first-touch page faults (measured: 1K float32 unary 16% hit-rate; 100K ~35 µs/op of soft faults — 7× the kernel's own time).

The pool therefore PACES the GC: when a Take MISSES (or a bucket is about to run dry) after at least PaceThresholdBytes of buffers were handed out since the last paced collection, it requests ONE forced, blocking, non-compacting gen-0 collection (~4-5 µs on a small heap). That collects the dropped wrapper graphs (allocated since the last pace → still gen 0), the finalizer thread drains, and the returns refill the buckets — the .NET stand-in for CPython's refcount-prompt free, amortized to at most one gen-0 pause per PaceThresholdBytes of allocation. Deterministically-disposing workloads (Dispose / NDScope) keep their buckets stocked and never trip the trigger. Opt-out: NUMSHARP_POOL_GC_PACING=0; window: NUMSHARP_POOL_GC_PACING_MB (default 16).

GC PRESSURE — coarse chunks, not per-call (Wave 2.5)

Wave 2.4 registered AddMemoryPressure(long) on every Take and removed it on every Return. Two measured pathologies: the Add/Remove pair costs ~55 ns per op, and the churn drove the GC into CONSTANT full gen-2 collections in allocating loops (100K float32 unary: a full GC every ~24 calls — ~12 µs/op of GC time; every collection observed was gen 2). Pressure now tracks the pool's OUTSTANDING bytes (taken − returned, i.e. what user code actually holds) in PressureChunkBytes chunks: the hot path pays two interlocked adds, and the GC only hears about net growth/shrink at 64 MiB granularity — enough for the issue #501 protection (a process holding gigabytes of native arrays still pushes the GC to collect) without per-op churn. Reclamation promptness inside the window is the pacing trigger's job, not pressure's.

SIZING POLICY

• The window is MinPoolableBytes (1 B) to MaxPoolableBytes (64 MiB) — Wave 2.4 opened both ends: the 1000-element float32 result (4000 B) missed the old 4 KiB floor by 96 bytes, and every 4M-element output (16–32 MiB) missed the old 1 MiB cap, paying ~2× in demand-zero page faults per call (in-place toggle-verified: P1 contig add 4M 3.37→1.74 ms). • Above the cap: no pooling. Huge buffers are rare and the memory cost of keeping them around dwarfs the alloc-cost savings. • Per-bucket caps are DYNAMIC (Wave 2.5): finalizer-driven returns arrive in one burst per collection — up to a whole pacing window of buffers at once. The old flat cap of 8 discarded ~95% of each burst (measured: 20K-call 1K loop → 3.3K pooled, 16K freed) and re-starved the loop. A bucket now holds up to one pacing window of bytes: cap = PaceThresholdBytes / size, clamped to [MaxBuffersPerBucket .. MaxBuffersPerDynamicBucket] below LargeBucketThreshold and to [MaxBuffersPerLargeBucket .. window] at/above it — so per-bucket resident stays ≈ the pacing window (16 MiB default) for mid sizes and keeps the old floor of 2 for huge buffers. • Bucket key is the EXACT byte count requested (no rounding). Same-size repeated allocs are the dominant pattern in element- wise ops; rounding to power-of-2 would waste memory and break exact-fit reuse for typical workloads (e.g. 4 MiB float32 1K×1K).

CORRECTNESS

• Stored buffers are NOT zero-filled. Callers that need zeroed memory must zero on Take (the same contract NativeMemory.Alloc has). • Buffer ownership transfers fully on Take: the pool no longer references the pointer, so subsequent Return calls aren't at risk of double-pop. • Return is best-effort: when the bucket is full or the size falls outside the pool's window the pointer is freed immediately via Free(void*).