Class NpyFormat
The NumPy .npy binary format (NEP-01) — reader and writer for versions 1.0, 2.0 and 3.0.
public static class NpyFormat
- Inheritance
-
NpyFormat
- Inherited Members
Remarks
A port of numpy/lib/_format_impl.py (NumPy 2.4.2), structured to mirror it function for
function so the two can be diffed. Byte-for-byte identical output is a tested contract: see
test/oracle/gen_npy_oracle.py, which has real NumPy write every case this class claims
to support, and NpyOracleTests, which replays them.
File layout:
\x93NUMPY 6 bytes magic (\x93 is byte 147, NOT the character '?')
major, minor 2 bytes format version
HEADER_LEN 2 bytes uint16 little-endian (v1.0)
4 bytes uint32 little-endian (v2.0 / v3.0)
header HEADER_LEN bytes — a Python dict literal, space-padded, '\n'-terminated,
sized so the data starts on a 64-byte (ARRAY_ALIGN) boundary
data shape-product * itemsize raw bytes, C- or Fortran-order
Divergences from NumPy, all forced by NumSharp's type system and each covered by a test:
- Big-endian files are byte-swapped to native on read. NumPy instead keeps the raw bytes behind a byte-swapped dtype; NumSharp has no such dtype, so it converts. Values match; the byte-order attribute is not preserved.
- Fixed-width strings (
'|S5','<U3') read and write as NumSharp's string arrays, byte for byte (a'>U3'file is byte-swapped per code point, as every big-endian file is). A zero-width'|S0'/'<U0'file is refused: NumSharp's string buffers hold at least one character per element. Char(2-byte UTF-16, no NumPy analog) is WRITTEN as NumPy's'<U1'(4-byte UCS-4); a'<U1'file reads back as NumPy's own'<U1'string array, not asChar(the mapping it had before the string dtypes existed).'<c8'(complex64) widens to Complex on read; writing always emits'<c16'.Decimalhas no NumPy dtype and cannot be written.- A NEP 55
StringDTypearray is written as NumPy writes it: the header says'|O'and the data is NumPy's protocol-4 pickle of the array, byte for byte; such a file (and NumPy's) loads withallow_pickle=True(NumSharp.IO.NpyPickle). - Other object arrays, structured dtypes, datetime64/timedelta64 and void are parsed but rejected with a precise message.
Fields
ArrayAlign
The header is padded so the data begins on a multiple of this. 64 bytes lets the data be memory-mapped and read with aligned SIMD loads.
public const int ArrayAlign = 64
Field Value
BufferSize
Chunk size for streamed reads, matching NumPy's BUFFER_SIZE (256 KB).
public const int BufferSize = 262144
Field Value
GrowthAxisMaxDigits
Spare spaces reserved after the dict so the shape can be rewritten in place when a file is
grown along its first (C-order) or last (F-order) axis. len(str(8 * 2**64 - 1)).
public const int GrowthAxisMaxDigits = 21
Field Value
MagicLen
Length of the magic string plus the two version bytes.
public const int MagicLen = 8
Field Value
MaxHeaderSize
Default cap on header size. Parsing an arbitrarily large header is a denial-of-service vector, and no legitimate array needs more than this.
public const int MaxHeaderSize = 10000
Field Value
Properties
MagicPrefix
The magic string every .npy file starts with: \x93NUMPY.
public static ReadOnlySpan<byte> MagicPrefix { get; }
Property Value
Methods
CheckVersion(FormatVersion)
Reject versions this implementation does not know — NumPy's _check_version().
public static void CheckVersion(NpyFormat.FormatVersion version)
Parameters
versionNpyFormat.FormatVersion
Exceptions
- FormatException
The version is not (1,0), (2,0) or (3,0).
DtypeToDescr(DType)
The descr string for a NumSharp descriptor — NumPy's dtype_to_descr: a fixed-width string
descriptor writes its own str ('|S5', '<U3', native order as NumSharp stores it), a
NEP 55 StringDType descriptor '|O' (NumPy "lies that the dtype is object" and pickles the array,
because a new-style dtype has no array-interface descr), every other dtype the descr of its storage lane
(DtypeToDescr(NPTypeCode)).
public static string DtypeToDescr(DType dtype)
Parameters
dtypeDTypeThe descriptor of the array being written.
Returns
- string
The header's descr.
Remarks
NumPy also warns UserWarning: Custom dtypes are saved as python objects using the pickle
protocol for a StringDType; NumSharp has no warnings channel and writes the same file silently.
Exceptions
- ArgumentNullException
dtypeis null.- NotSupportedException
The dtype has no NumPy equivalent (Decimal).
DtypeToDescr(NPTypeCode)
The descr string for a NumSharp dtype — the inverse of DescrToDtype(object)
and the equivalent of NumPy's dtype_to_descr for the types NumSharp has.
public static string DtypeToDescr(NPTypeCode typeCode)
Parameters
typeCodeNPTypeCode
Returns
Remarks
Single-byte types take the '|' (not-applicable) prefix, everything else '<':
NumSharp is native-order internally and every platform it runs on is little-endian.
Exceptions
- NotSupportedException
The dtype has no NumPy equivalent.
HeaderDataFromArray(NDArray)
Build the header dict for an array — NumPy's header_data_from_array_1_0().
public static Dictionary<string, object> HeaderDataFromArray(NDArray array)
Parameters
arrayNDArray
Returns
Remarks
C-contiguity is tested first because a 1-D, 0-d or empty array is BOTH C- and
F-contiguous, and those must come out as fortran_order: False. An array that is
neither is copied to C-order at write time, so it too reports False.
IsNpyFile(Stream)
Whether the stream begins with the .npy magic. The position is restored.
public static bool IsNpyFile(Stream stream)
Parameters
streamStream
Returns
IsNpzFile(Stream)
Whether the stream begins with a ZIP signature — a .npz. The position is restored.
public static bool IsNpzFile(Stream stream)
Parameters
streamStream
Returns
Magic(FormatVersion)
The 8 magic bytes for a version — NumPy's magic().
public static byte[] Magic(NpyFormat.FormatVersion version)
Parameters
versionNpyFormat.FormatVersion
Returns
- byte[]
OpenMemmap(string, string, long)
Open a .npy file as a memory-mapped array — NumPy's open_memmap, reached via
np.load(path, mmap_mode=…). The array shares the file's bytes: nothing is copied, and
for r+ writes flush back to disk.
public static NDArray OpenMemmap(string path, string mode, long maxHeaderSize = 10000)
Parameters
pathstringPath to a
.npyfile — mmap needs a real file, not a stream.modestringA NumPy mmap mode. Through
np.loadonlyr/readonly(read-only),r+(read-write, persists) andc(copy-on-write, not persisted) actually work;w+/write/copyonwrite/readwritevalidate but fail downstream in NumPy too, and ResolveMmapMode(string) reproduces those exact errors.maxHeaderSizelongReject headers larger than this.
Returns
- NDArray
An NDArray viewing the mapped file. It holds the mapping alive; the map, view and file handle are released when the array (and every view of it) is disposed or collected — the same lifecycle a normally-owned array frees on.
Exceptions
- NotSupportedException
The dtype cannot be zero-copy mapped: NumSharp byte-swaps big-endian files (a
'>U3'file included) and widens<c8on read, none of which a shared view allows. A native fixed-width string file maps like any other dtype.
ReadArray(Stream, bool, long)
Read an array — NumPy's read_array().
public static NDArray ReadArray(Stream stream, bool allowPickle = false, long maxHeaderSize = 10000)
Parameters
streamStreamA stream positioned at the magic string.
allowPickleboolWhether the caller trusts this file. A
'|O'file holds a pickle: without it NumPy's error, with it NumSharp reads NumPy's pickle of aStringDTypearray (and refuses any other pickle). Per NumPy it also liftsmaxHeaderSize.maxHeaderSizelongReject headers larger than this. Ignored when
allowPickle.
Returns
- NDArray
The array; for a pickled file the shape and layout the PICKLE states (NumPy ignores the header's for an object array).
Exceptions
- FormatException
A malformed file, or
Object arrays cannot be loaded when allow_pickle=False.- NotSupportedException
A dtype NumSharp cannot represent, or a pickle that is not NumPy's pickle of a
StringDTypearray.- UnicodeError
A pickled text that is not UTF-8: NumPy's
Unpickling a python object failed: UnicodeDecodeError('utf-8', b'\xff', 0, 1, 'invalid start byte')text (and its line aboutencoding=), with the UnicodeDecodeError as the inner exception.
ReadMagic(Stream)
Read and validate the magic string, returning the file's format version — NumPy's
read_magic(). Leaves the stream on the header-length field.
public static NpyFormat.FormatVersion ReadMagic(Stream stream)
Parameters
streamStream
Returns
Exceptions
- FormatException
The magic string is wrong or the stream ends inside it.
WriteArray(Stream, NDArray, FormatVersion?, bool)
Write an array with its header — NumPy's write_array().
public static void WriteArray(Stream stream, NDArray array, NpyFormat.FormatVersion? version = null, bool allowPickle = true)
Parameters
streamStreamAn open, writable stream. Written from its current position.
arrayNDArrayThe array to write. Any layout; non-contiguous is materialized C-order.
versionNpyFormat.FormatVersion?The format version, or null (default) for the oldest that fits.
allowPickleboolWhether a pickle may be written. Only a NEP 55
StringDTypearray needs one (NumPy pickles it behind a'|O'header); withfalsesuch an array raises NumPy's error AFTER its header is written, as NumPy'swrite_arraydoes. Every other dtype ignores it.
Exceptions
- NotSupportedException
The dtype has no NumPy equivalent (Decimal), or a
StringDTypemissing-data object has no pickle form NumSharp writes (see NumSharp.IO.NpyPickle).- ValueError
Object arrays cannot be saved when allow_pickle=False: aStringDTypearray withallowPicklefalse (the header is already instream).
WriteArrayHeader(Stream, Dictionary<string, object>, FormatVersion?)
Write an array header — NumPy's _write_array_header(). Leaves the stream exactly at
the (64-byte aligned) start of the data.
public static void WriteArrayHeader(Stream stream, Dictionary<string, object> d, NpyFormat.FormatVersion? version = null)
Parameters
streamStreamTarget stream.
dDictionary<string, object>Header dict:
descr,fortran_orderandshape.versionNpyFormat.FormatVersion?The version to write, or null to pick the oldest that fits.
Exceptions
- ArgumentException
dhas noshape, or one that is not along[]/int[].- FormatException
versionis not (1,0), (2,0) or (3,0).- ValueError
The header does not fit
version's length field (NumPy'sHeader length N too big for version=(1, 0)).- InvalidOperationException
The header is not latin-1 encodable and
versionis below (3, 0).