Table of Contents

Class VStringAllocator

Namespace
NumSharp.Backends.Strings
Assembly
NumSharp.dll

The string allocator of ONE NEP 55 StringDType buffer — a statement-for-statement port of NumPy 2.4.2's numpy/_core/src/multiarray/stringdtype/static_string.c: the 16-byte packed cell of every element, the per-buffer ARENA that holds the text of elements longer than 15 bytes, the HEAP strings that a later, longer write moves an element to, and the mutex that guards them.

public sealed class VStringAllocator
Inheritance
VStringAllocator
Inherited Members

Remarks

The cell (npy_packed_static_string, little-endian 64-bit layout). Bytes 0–7 hold an arena OFFSET or a heap POINTER; bytes 8–15 hold the size, whose highest byte (byte 15) is the FLAGS byte: NPY_STRING_MISSING (an NA value), NPY_STRING_INITIALIZED (written at least once), NPY_STRING_OUTSIDE_ARENA (short or heap), NPY_STRING_LONG (an arena string of more than 255 bytes, or a heap string). A string of up to NPY_SHORT_STRING_MAX_SIZE bytes lives IN the cell: bytes 0–14 are the text and the low nibble of byte 15 is its length, with the flags NPY_STRING_INITIALIZED | NPY_STRING_OUTSIDE_ARENA. An all-zero cell is an uninitialized empty string, so a zero-filled buffer is a buffer of '' for free (NumPy's NPY_NEEDS_INIT).

The arena. A medium string (16–255 bytes) is stored after a one-byte size prefix, a long one after an eight-byte prefix; the cell holds the offset of the TEXT (after the prefix). The arena grows by reallocation to 125 % of what it needs, and its new tail is zeroed. A write that fits the old allocation reuses it (the prefix records the allocation's capacity); a longer one goes to the heap. A string set to NA keeps its allocation's metadata (only the NPY_STRING_MISSING bit is added), so a later write can reuse its arena space. After a short write the arena information is gone, and a later long write always goes to the heap.

NumPy's names. The constants carry the names of static_string.c's macros (NPY_STRING_*, NPY_SHORT_STRING_*, NPY_MEDIUM_STRING_MAX_SIZE, NPY_MAX_STRING_SIZE; the cell size is stringdtype/dtype.h's SIZEOF_NPY_PACKED_STATIC_STRING), and the members that port the NpyString_* functions carry the function's name after the prefix (load(byte*, out byte*, out long), pack(byte*, byte*, long), pack_null(byte*), free(byte*), dup(byte*, byte*, VStringAllocator, VStringAllocator), …). The two NumPy functions that take a list of allocators, NpyString_acquire_allocators and NpyString_release_allocators, have a two-allocator overload each. The ports of static_string.c's unprefixed helpers (is_short_string, arena_malloc, heap_or_arena_allocate, vstring_buffer, …) keep NumSharp names, as do the members NumPy has no function for. The test DTypeOracleTests.Members_ExistUnderNumPysNames holds the class to the names and values it reads from NumPy's source.

Ownership. One allocator per BUFFER (NumPy gives every array that owns data its own descriptor instance and allocator, stringdtype_finalize_descr): views and slices of the buffer share it, a copy gets a new one. The cells' memory block owns the allocator — its release runs ClearAndFree(byte*, long) (NumPy's clear loop, which frees the heap strings, then the arena), so the allocator lives exactly as long as the last view of its cells. Cells are never valid with another buffer's allocator: an offset reads someone else's arena, and a heap pointer copied into another buffer would be freed twice. Every movement of StringDType elements therefore goes through dup(byte*, byte*, VStringAllocator, VStringAllocator), never through a raw copy of the cells.

Locking. Like NumPy's allocator mutex: a caller takes the lock ONCE per operation (acquire_allocator() / release_allocator(), or acquire_allocators(VStringAllocator, VStringAllocator) for two allocators), then calls the per-cell members, which assume it is held. The allocator does not lock per element.

Errors. NumPy's members return -1 and the caller raises MemoryError with its own text; here the members return the same codes (0 success, -1 failure, 1 = "is null" for load(byte*, out byte*, out long)) and the callers throw OutOfMemoryException with NumPy's text. The arena is NativeMemory memory (NumPy's PyMem_RawMalloc / PyMem_RawRealloc).

Heap strings come from pages, not from one malloc each (the one place this port leaves NumPy's memory calls; the cells are NumPy's byte for byte, but the heap pointer they hold is an address either way). NumPy PyMem_RawMallocs every text that outgrows its arena slot, and that call was half of an assignment of longer texts over a held array: 44 ns a text of 20–60 bytes, 149 ns of 300–500 (measured, pinned). A text of at most NumSharp.Backends.Strings.VStringAllocator.HeapPoolMaxBytes bytes is carved instead from a page of its SIZE CLASS (block strides of 32 to 1,040 bytes in steps of 16): pages of 1 to 64 KiB taken from SizeBucketedBufferPool (warm memory, recycled across buffers, accounted there), a free list and a bump pointer per page, an 8-byte header in front of every text that names its page. Freeing a text pushes it on its page's free list; a page whose last text is freed goes back to the buffer pool (the allocator keeps ONE emptied page as a spare for its next page), so the memory a buffer holds follows its live texts (as pymalloc's pools do), not its peak. A longer text is a block of its own from the C allocator (a null page in its header). The clear loop returns the pages whole and visits the cells only while such a block exists. Measured (pinned) on np.copyto(dst, b) of 100,000 texts over a held array: 62.2 → 39.6 ns a text at 20–60 bytes and 189 → 89 at 300–500 (NumPy 64.1 and 154).

Public surface. NumPy's documented C API for StringDType cells is public here under its names - acquire_allocator(), release_allocator(), load(byte*, out byte*, out long), pack(byte*, byte*, long) and pack_null(byte*) - so a bridge package can read and write an array's texts without a copy, as NumPy's C extensions do. Reach an array's allocator through VStringSlice (.Allocator); every other member stays internal, as NumPy keeps its counterparts private. Footguns: hold the lock across every call, and touch only cells of the buffer the allocator belongs to.

Fields

NPY_MAX_STRING_SIZE

NPY_MAX_STRING_SIZE: one byte of the size field is the flags, so a string has at most 2^56 − 1 bytes.

public const long NPY_MAX_STRING_SIZE = 72057594037927935

Field Value

long

NPY_MEDIUM_STRING_MAX_SIZE

NPY_MEDIUM_STRING_MAX_SIZE: the longest arena string with a one-byte size prefix.

public const int NPY_MEDIUM_STRING_MAX_SIZE = 255

Field Value

int

NPY_SHORT_STRING_MAX_SIZE

NPY_SHORT_STRING_MAX_SIZE: the longest text a cell holds itself (15 bytes).

public const int NPY_SHORT_STRING_MAX_SIZE = 15

Field Value

int

NPY_SHORT_STRING_SIZE_MASK

NPY_SHORT_STRING_SIZE_MASK: the low nibble of byte 15, a short string's length.

public const byte NPY_SHORT_STRING_SIZE_MASK = 15

Field Value

byte

NPY_STRING_FLAG_MASK

NPY_STRING_FLAG_MASK: the high nibble of byte 15.

public const byte NPY_STRING_FLAG_MASK = 240

Field Value

byte

NPY_STRING_INITIALIZED

NPY_STRING_INITIALIZED: the element has been written at least once.

public const byte NPY_STRING_INITIALIZED = 64

Field Value

byte

NPY_STRING_LONG

NPY_STRING_LONG: an arena string longer than 255 bytes (eight-byte prefix), or a heap string.

public const byte NPY_STRING_LONG = 16

Field Value

byte

NPY_STRING_MISSING

NPY_STRING_MISSING: the cell is an NA value (its other flags keep the old allocation's metadata).

public const byte NPY_STRING_MISSING = 128

Field Value

byte

NPY_STRING_OUTSIDE_ARENA

NPY_STRING_OUTSIDE_ARENA: the text is in the cell (short) or on the heap (with NPY_STRING_LONG).

public const byte NPY_STRING_OUTSIDE_ARENA = 32

Field Value

byte

SIZEOF_NPY_PACKED_STATIC_STRING

NumPy's SIZEOF_NPY_PACKED_STATIC_STRING (stringdtype/dtype.h): the size of one packed cell (sizeof(npy_packed_static_string)), so the stride between cells and the descriptor's itemsize. It is the same constant as SIZEOF_NPY_PACKED_STATIC_STRING; the dtype oracle test holds both to NumPy's value, so the two cannot drift apart.

public const int SIZEOF_NPY_PACKED_STATIC_STRING = 16

Field Value

int

Methods

acquire_allocator()

Acquires the allocator mutex (NpyString_acquire_allocator); pair it with exactly one release_allocator().

public void acquire_allocator()

Remarks

A Monitor lock: re-entrant on the same thread, and released by the thread that took it. Footgun: take it in a try / finally that releases it - an exception between the two otherwise leaves every array over this buffer blocked for good. Taking two allocators one after the other can deadlock against a thread that takes them in the other order; NumPy's two-allocator form exists for that.

load(byte*, out byte*, out long)

Unpacks a cell (NpyString_load): its text pointer and size, read-only, valid while the lock is held and the cell is not written.

public int load(byte* packed, out byte* buf, out long size)

Parameters

packed byte*

The cell.

buf byte*

Receives the first text byte (null for an NA value; a non-null pointer to nothing for '').

size long

Receives the text size in bytes.

Returns

int

1 for an NA value, 0 for a string, -1 when an arena string's arena is missing (a corrupt cell).

Remarks

Footgun: buf points into this allocator's arena or heap pages - it dangles once the lock is released and another thread writes the cell, or once the array is disposed. Copy the bytes out before releasing; never write through it.

pack(byte*, byte*, long)

Packs a UTF-8 text into a cell (NpyString_pack): frees the old text, then allocates and copies.

public int pack(byte* s, byte* buf, long size)

Parameters

s byte*

The cell.

buf byte*

The UTF-8 text (not inside this allocator's arena).

size long

Its size in bytes.

Returns

int

0, or -1 on failure.

Remarks

The lock must be held, and s must be a cell of this allocator's buffer. Footgun: buf must not point into this allocator's own memory (a text load(byte*, out byte*, out long) returned) - the old text is freed before the copy, so packing a cell's own text, or a neighbour's arena text, copies freed bytes. Copy it out first.

pack_null(byte*)

Packs an NA value into a cell (NpyString_pack_null): the old text is freed, and the flags keep the old allocation's metadata so a later write can reuse its arena space.

public int pack_null(byte* s)

Parameters

s byte*

The cell.

Returns

int

0, or -1 on failure.

Remarks

The lock must be held, and s must be a cell of this allocator's buffer: a cell of another buffer has its text freed by the wrong allocator.

release_allocator()

Releases the allocator mutex (NpyString_release_allocator).

public void release_allocator()

Exceptions

SynchronizationLockException

The calling thread does not hold the lock (a release without its acquire_allocator(), or from another thread).