Class VStringAllocator
The string allocator of ONE NEP 55 StringDType buffer — a statement-for-statement port of NumPy 2.4.2's
numpy/_core/src/multiarray/stringdtype/static_string.c: the 16-byte packed cell of every element, the
per-buffer ARENA that holds the text of elements longer than 15 bytes, the HEAP strings that a later, longer
write moves an element to, and the mutex that guards them.
public sealed class VStringAllocator
- Inheritance
-
VStringAllocator
- Inherited Members
Remarks
The cell (npy_packed_static_string, little-endian 64-bit layout). Bytes 0–7 hold an arena OFFSET or
a heap POINTER; bytes 8–15 hold the size, whose highest byte (byte 15) is the FLAGS byte:
NPY_STRING_MISSING (an NA value), NPY_STRING_INITIALIZED (written at least once),
NPY_STRING_OUTSIDE_ARENA (short or heap), NPY_STRING_LONG (an arena string of more
than 255 bytes, or a heap string). A string of up to NPY_SHORT_STRING_MAX_SIZE bytes lives IN the
cell: bytes 0–14 are the text and the low nibble of byte 15 is its length, with the flags
NPY_STRING_INITIALIZED | NPY_STRING_OUTSIDE_ARENA. An all-zero cell is an uninitialized empty string, so a
zero-filled buffer is a buffer of '' for free (NumPy's NPY_NEEDS_INIT).
The arena. A medium string (16–255 bytes) is stored after a one-byte size prefix, a long one after an eight-byte prefix; the cell holds the offset of the TEXT (after the prefix). The arena grows by reallocation to 125 % of what it needs, and its new tail is zeroed. A write that fits the old allocation reuses it (the prefix records the allocation's capacity); a longer one goes to the heap. A string set to NA keeps its allocation's metadata (only the NPY_STRING_MISSING bit is added), so a later write can reuse its arena space. After a short write the arena information is gone, and a later long write always goes to the heap.
NumPy's names. The constants carry the names of static_string.c's macros (NPY_STRING_*,
NPY_SHORT_STRING_*, NPY_MEDIUM_STRING_MAX_SIZE, NPY_MAX_STRING_SIZE; the cell size is
stringdtype/dtype.h's SIZEOF_NPY_PACKED_STATIC_STRING), and the members that port the
NpyString_* functions carry the function's name after the prefix (load(byte*, out byte*, out long), pack(byte*, byte*, long),
pack_null(byte*), free(byte*), dup(byte*, byte*, VStringAllocator, VStringAllocator), …). The two NumPy functions that take a list of
allocators, NpyString_acquire_allocators and NpyString_release_allocators, have a two-allocator
overload each. The ports of static_string.c's unprefixed helpers (is_short_string,
arena_malloc, heap_or_arena_allocate, vstring_buffer, …) keep NumSharp names, as do the
members NumPy has no function for. The test DTypeOracleTests.Members_ExistUnderNumPysNames holds the class
to the names and values it reads from NumPy's source.
Ownership. One allocator per BUFFER (NumPy gives every array that owns data its own descriptor instance
and allocator, stringdtype_finalize_descr): views and slices of the buffer share it, a copy gets a new one.
The cells' memory block owns the allocator — its release runs ClearAndFree(byte*, long) (NumPy's clear loop,
which frees the heap strings, then the arena), so the allocator lives exactly as long as the last view of its
cells. Cells are never valid with another buffer's allocator: an offset reads someone else's arena, and a heap
pointer copied into another buffer would be freed twice. Every movement of StringDType elements therefore
goes through dup(byte*, byte*, VStringAllocator, VStringAllocator), never through a raw copy of the cells.
Locking. Like NumPy's allocator mutex: a caller takes the lock ONCE per operation (acquire_allocator() / release_allocator(), or acquire_allocators(VStringAllocator, VStringAllocator) for two allocators), then calls the per-cell members, which assume it is held. The allocator does not lock per element.
Errors. NumPy's members return -1 and the caller raises MemoryError with its own text; here the
members return the same codes (0 success, -1 failure, 1 = "is null" for load(byte*, out byte*, out long)) and the callers
throw OutOfMemoryException with NumPy's text. The arena is NativeMemory memory
(NumPy's PyMem_RawMalloc / PyMem_RawRealloc).
Heap strings come from pages, not from one malloc each (the one place this port leaves NumPy's
memory calls; the cells are NumPy's byte for byte, but the heap pointer they hold is an address either way). NumPy
PyMem_RawMallocs every text that outgrows its arena slot, and that call was half of an assignment of
longer texts over a held array: 44 ns a text of 20–60 bytes, 149 ns of 300–500 (measured, pinned). A text of at
most NumSharp.Backends.Strings.VStringAllocator.HeapPoolMaxBytes bytes is carved instead from a page of its SIZE CLASS (block strides of 32
to 1,040 bytes in steps of 16): pages of 1 to 64 KiB taken from SizeBucketedBufferPool
(warm memory, recycled across buffers, accounted there), a free list and a bump pointer per page, an 8-byte header
in front of every text that names its page. Freeing a text pushes it on its page's free list; a page whose last
text is freed goes back to the buffer pool (the allocator keeps ONE emptied page as a spare for its next page), so
the memory a buffer holds follows its live texts (as pymalloc's pools do), not its peak. A longer text is a block
of its own from the C allocator (a null page in its header). The clear loop returns the pages whole and visits the
cells only while such a block exists. Measured (pinned) on np.copyto(dst, b) of 100,000 texts over a held
array: 62.2 → 39.6 ns a text at 20–60 bytes and 189 → 89 at 300–500 (NumPy 64.1 and 154).
Public surface. NumPy's documented C API for StringDType cells is public here under its names -
acquire_allocator(), release_allocator(), load(byte*, out byte*, out long), pack(byte*, byte*, long) and
pack_null(byte*) - so a bridge package can read and write an array's texts without a copy, as NumPy's C
extensions do. Reach an array's allocator through VStringSlice
(.Allocator); every other member stays internal, as NumPy keeps its counterparts private. Footguns: hold
the lock across every call, and touch only cells of the buffer the allocator belongs to.
Fields
NPY_MAX_STRING_SIZE
NPY_MAX_STRING_SIZE: one byte of the size field is the flags, so a string has at most 2^56 − 1 bytes.
public const long NPY_MAX_STRING_SIZE = 72057594037927935
Field Value
NPY_MEDIUM_STRING_MAX_SIZE
NPY_MEDIUM_STRING_MAX_SIZE: the longest arena string with a one-byte size prefix.
public const int NPY_MEDIUM_STRING_MAX_SIZE = 255
Field Value
NPY_SHORT_STRING_MAX_SIZE
NPY_SHORT_STRING_MAX_SIZE: the longest text a cell holds itself (15 bytes).
public const int NPY_SHORT_STRING_MAX_SIZE = 15
Field Value
NPY_SHORT_STRING_SIZE_MASK
NPY_SHORT_STRING_SIZE_MASK: the low nibble of byte 15, a short string's length.
public const byte NPY_SHORT_STRING_SIZE_MASK = 15
Field Value
NPY_STRING_FLAG_MASK
NPY_STRING_FLAG_MASK: the high nibble of byte 15.
public const byte NPY_STRING_FLAG_MASK = 240
Field Value
NPY_STRING_INITIALIZED
NPY_STRING_INITIALIZED: the element has been written at least once.
public const byte NPY_STRING_INITIALIZED = 64
Field Value
NPY_STRING_LONG
NPY_STRING_LONG: an arena string longer than 255 bytes (eight-byte prefix), or a heap string.
public const byte NPY_STRING_LONG = 16
Field Value
NPY_STRING_MISSING
NPY_STRING_MISSING: the cell is an NA value (its other flags keep the old allocation's metadata).
public const byte NPY_STRING_MISSING = 128
Field Value
NPY_STRING_OUTSIDE_ARENA
NPY_STRING_OUTSIDE_ARENA: the text is in the cell (short) or on the heap (with NPY_STRING_LONG).
public const byte NPY_STRING_OUTSIDE_ARENA = 32
Field Value
SIZEOF_NPY_PACKED_STATIC_STRING
NumPy's SIZEOF_NPY_PACKED_STATIC_STRING (stringdtype/dtype.h): the size of one packed cell
(sizeof(npy_packed_static_string)), so the stride between cells and the descriptor's itemsize. It
is the same constant as SIZEOF_NPY_PACKED_STATIC_STRING; the dtype oracle
test holds both to NumPy's value, so the two cannot drift apart.
public const int SIZEOF_NPY_PACKED_STATIC_STRING = 16
Field Value
Methods
acquire_allocator()
Acquires the allocator mutex (NpyString_acquire_allocator); pair it with exactly one release_allocator().
public void acquire_allocator()
Remarks
A Monitor lock: re-entrant on the same thread, and released by the thread that took it. Footgun:
take it in a try / finally that releases it - an exception between the two otherwise leaves every
array over this buffer blocked for good. Taking two allocators one after the other can deadlock against a
thread that takes them in the other order; NumPy's two-allocator form exists for that.
load(byte*, out byte*, out long)
Unpacks a cell (NpyString_load): its text pointer and size, read-only, valid while the lock is held and the
cell is not written.
public int load(byte* packed, out byte* buf, out long size)
Parameters
packedbyte*The cell.
bufbyte*Receives the first text byte (null for an NA value; a non-null pointer to nothing for
'').sizelongReceives the text size in bytes.
Returns
- int
1 for an NA value, 0 for a string, -1 when an arena string's arena is missing (a corrupt cell).
Remarks
Footgun: buf points into this allocator's arena or heap pages - it dangles once the lock
is released and another thread writes the cell, or once the array is disposed. Copy the bytes out before
releasing; never write through it.
pack(byte*, byte*, long)
Packs a UTF-8 text into a cell (NpyString_pack): frees the old text, then allocates and copies.
public int pack(byte* s, byte* buf, long size)
Parameters
sbyte*The cell.
bufbyte*The UTF-8 text (not inside this allocator's arena).
sizelongIts size in bytes.
Returns
- int
0, or -1 on failure.
Remarks
The lock must be held, and s must be a cell of this allocator's buffer. Footgun:
buf must not point into this allocator's own memory (a text load(byte*, out byte*, out long) returned) -
the old text is freed before the copy, so packing a cell's own text, or a neighbour's arena text, copies freed
bytes. Copy it out first.
pack_null(byte*)
Packs an NA value into a cell (NpyString_pack_null): the old text is freed, and the flags keep the old
allocation's metadata so a later write can reuse its arena space.
public int pack_null(byte* s)
Parameters
sbyte*The cell.
Returns
- int
0, or -1 on failure.
Remarks
The lock must be held, and s must be a cell of this allocator's buffer: a cell of
another buffer has its text freed by the wrong allocator.
release_allocator()
Releases the allocator mutex (NpyString_release_allocator).
public void release_allocator()
Exceptions
- SynchronizationLockException
The calling thread does not hold the lock (a release without its acquire_allocator(), or from another thread).