Table of Contents

Class NpyFormat

Namespace
NumSharp.IO
Assembly
NumSharp.dll

The NumPy .npy binary format (NEP-01) — reader and writer for versions 1.0, 2.0 and 3.0.

public static class NpyFormat
Inheritance
NpyFormat
Inherited Members

Remarks

A port of numpy/lib/_format_impl.py (NumPy 2.4.2), structured to mirror it function for function so the two can be diffed. Byte-for-byte identical output is a tested contract: see test/oracle/gen_npy_oracle.py, which has real NumPy write every case this class claims to support, and NpyOracleTests, which replays them.

File layout:

\x93NUMPY          6 bytes   magic (\x93 is byte 147, NOT the character '?')
major, minor       2 bytes   format version
HEADER_LEN         2 bytes   uint16 little-endian  (v1.0)
                   4 bytes   uint32 little-endian  (v2.0 / v3.0)
header             HEADER_LEN bytes — a Python dict literal, space-padded, '\n'-terminated,
                   sized so the data starts on a 64-byte (ARRAY_ALIGN) boundary
data               shape-product * itemsize raw bytes, C- or Fortran-order

Divergences from NumPy, all forced by NumSharp's type system and each covered by a test:

  • Big-endian files are byte-swapped to native on read. NumPy instead keeps the raw bytes behind a byte-swapped dtype; NumSharp has no such dtype, so it converts. Values match; the byte-order attribute is not preserved.
  • Char (2-byte UTF-16) maps to NumPy's '<U1' (4-byte UCS-4), converting in both directions. '<U' widths above 1 have no NumSharp analog.
  • '<c8' (complex64) widens to Complex on read; writing always emits '<c16'.
  • Decimal has no NumPy dtype and cannot be written.
  • Object arrays, structured dtypes, datetime64/timedelta64, byte strings and void are parsed but rejected with a precise message (see the issue's "What We're Not Doing").

Fields

ArrayAlign

The header is padded so the data begins on a multiple of this. 64 bytes lets the data be memory-mapped and read with aligned SIMD loads.

public const int ArrayAlign = 64

Field Value

int

BufferSize

Chunk size for streamed reads, matching NumPy's BUFFER_SIZE (256 KB).

public const int BufferSize = 262144

Field Value

int

GrowthAxisMaxDigits

Spare spaces reserved after the dict so the shape can be rewritten in place when a file is grown along its first (C-order) or last (F-order) axis. len(str(8 * 2**64 - 1)).

public const int GrowthAxisMaxDigits = 21

Field Value

int

MagicLen

Length of the magic string plus the two version bytes.

public const int MagicLen = 8

Field Value

int

MaxHeaderSize

Default cap on header size. Parsing an arbitrarily large header is a denial-of-service vector, and no legitimate array needs more than this.

public const int MaxHeaderSize = 10000

Field Value

int

Properties

MagicPrefix

The magic string every .npy file starts with: \x93NUMPY.

public static ReadOnlySpan<byte> MagicPrefix { get; }

Property Value

ReadOnlySpan<byte>

Methods

CheckVersion(FormatVersion)

Reject versions this implementation does not know — NumPy's _check_version().

public static void CheckVersion(NpyFormat.FormatVersion version)

Parameters

version NpyFormat.FormatVersion

Exceptions

FormatException

The version is not (1,0), (2,0) or (3,0).

DtypeToDescr(NPTypeCode)

The descr string for a NumSharp dtype — the inverse of DescrToDtype(object) and the equivalent of NumPy's dtype_to_descr for the types NumSharp has.

public static string DtypeToDescr(NPTypeCode typeCode)

Parameters

typeCode NPTypeCode

Returns

string

Remarks

Single-byte types take the '|' (not-applicable) prefix, everything else '<': NumSharp is native-order internally and every platform it runs on is little-endian.

Exceptions

NotSupportedException

The dtype has no NumPy equivalent.

HeaderDataFromArray(NDArray)

Build the header dict for an array — NumPy's header_data_from_array_1_0().

public static Dictionary<string, object> HeaderDataFromArray(NDArray array)

Parameters

array NDArray

Returns

Dictionary<string, object>

Remarks

C-contiguity is tested first because a 1-D, 0-d or empty array is BOTH C- and F-contiguous, and those must come out as fortran_order: False. An array that is neither is copied to C-order at write time, so it too reports False.

IsNpyFile(Stream)

Whether the stream begins with the .npy magic. The position is restored.

public static bool IsNpyFile(Stream stream)

Parameters

stream Stream

Returns

bool

IsNpzFile(Stream)

Whether the stream begins with a ZIP signature — a .npz. The position is restored.

public static bool IsNpzFile(Stream stream)

Parameters

stream Stream

Returns

bool

Magic(FormatVersion)

The 8 magic bytes for a version — NumPy's magic().

public static byte[] Magic(NpyFormat.FormatVersion version)

Parameters

version NpyFormat.FormatVersion

Returns

byte[]

OpenMemmap(string, string, long)

Open a .npy file as a memory-mapped array — NumPy's open_memmap, reached via np.load(path, mmap_mode=…). The array shares the file's bytes: nothing is copied, and for r+ writes flush back to disk.

public static NDArray OpenMemmap(string path, string mode, long maxHeaderSize = 10000)

Parameters

path string

Path to a .npy file — mmap needs a real file, not a stream.

mode string

A NumPy mmap mode. Through np.load only r/readonly (read-only), r+ (read-write, persists) and c (copy-on-write, not persisted) actually work; w+/write/copyonwrite/readwrite validate but fail downstream in NumPy too, and ResolveMmapMode(string) reproduces those exact errors.

maxHeaderSize long

Reject headers larger than this.

Returns

NDArray

An NDArray viewing the mapped file. It holds the mapping alive; the map, view and file handle are released when the array (and every view of it) is disposed or collected — the same lifecycle a normally-owned array frees on.

Exceptions

NotSupportedException

The dtype cannot be zero-copy mapped: NumSharp byte-swaps big-endian files and converts <U1/<c8 on read, none of which a shared view allows.

ReadArray(Stream, bool, long)

Read an array — NumPy's read_array().

public static NDArray ReadArray(Stream stream, bool allowPickle = false, long maxHeaderSize = 10000)

Parameters

stream Stream

A stream positioned at the magic string.

allowPickle bool

Whether the caller trusts this file. NumSharp cannot unpickle either way, but this selects which error an object array produces and, per NumPy, lifts maxHeaderSize.

maxHeaderSize long

Reject headers larger than this. Ignored when allowPickle.

Returns

NDArray

ReadMagic(Stream)

Read and validate the magic string, returning the file's format version — NumPy's read_magic(). Leaves the stream on the header-length field.

public static NpyFormat.FormatVersion ReadMagic(Stream stream)

Parameters

stream Stream

Returns

NpyFormat.FormatVersion

Exceptions

FormatException

The magic string is wrong or the stream ends inside it.

WriteArray(Stream, NDArray, FormatVersion?, bool)

Write an array with its header — NumPy's write_array().

public static void WriteArray(Stream stream, NDArray array, NpyFormat.FormatVersion? version = null, bool allowPickle = true)

Parameters

stream Stream

An open, writable stream. Written from its current position.

array NDArray

The array to write. Any layout; non-contiguous is materialized C-order.

version NpyFormat.FormatVersion?

The format version, or null (default) for the oldest that fits.

allowPickle bool

Present for NumPy parity. NumSharp has no object dtype, so no array can reach the pickle path and this parameter never changes the outcome.

Exceptions

NotSupportedException

The dtype has no NumPy equivalent (Decimal).

WriteArrayHeader(Stream, Dictionary<string, object>, FormatVersion?)

Write an array header — NumPy's _write_array_header(). Leaves the stream exactly at the (64-byte aligned) start of the data.

public static void WriteArrayHeader(Stream stream, Dictionary<string, object> d, NpyFormat.FormatVersion? version = null)

Parameters

stream Stream

Target stream.

d Dictionary<string, object>

Header dict: descr, fortran_order and shape.

version NpyFormat.FormatVersion?

The version to write, or null to pick the oldest that fits.

Edit this page