Table of Contents

Class NpzFile

Namespace
NumSharp.IO
Assembly
NumSharp.dll

A lazily-loaded .npz archive — NumPy's NpzFile.

public sealed class NpzFile : IReadOnlyDictionary<string, NDArray>, IReadOnlyCollection<KeyValuePair<string, NDArray>>, IEnumerable<KeyValuePair<string, NDArray>>, IEnumerable, IDisposable
Inheritance
NpzFile
Implements
Inherited Members

Remarks

A .npz is a ZIP archive of .npy members. Nothing is decoded until a key is accessed, and each array is cached from then on, so a huge archive costs only what is read out of it.

Keys work with or without the .npy suffix — npz["weights"] and npz["weights.npy"] are the same member — while files reports the stripped names, matching NumPy.

The members are NumPy's, names included (API-parity batch 13): files, f, zip, fid, allow_pickle, max_header_size, keys(), values(), items(), get(string, NDArray) and close(). The dictionary interface's Count / Keys / Values are explicit (NumPy has len(z), keys() and values()); ContainsKey(string) and TryGetValue(string, out NDArray) stay the C# spellings of NumPy's key in z, and foreach keeps the dictionary's name / array pairs.

A missing key raises NumPy's errors: the indexer a KeyError ('x is not a file in the archive') and dot access an AttributeError (x). Footgun: although the class is an IReadOnlyDictionary<TKey, TValue>, its indexer does NOT throw .NET's KeyNotFoundException; test with ContainsKey(string) or TryGetValue(string, out NDArray) first, or catch KeyError.

The archive holds an open file handle, so dispose it:

using var npz = np.load_npz("model.npz");
NDArray w = npz["weights"];
NDArray b = npz.f.biases;   // dot access, NumPy's BagObj

Any number of threads may read members of one archive at the same time (NumPy's NpzFile is not thread-safe; this one is). A member already loaded is returned without a lock. Everything that reads the archive itself (loading a member, GetRawBytes(string), the magic check of IsArray(string)) runs under one gate, because the archive reads a single underlying stream, so new members load one at a time. Two threads asking for the same member first get the same instance. Dispose() waits for a load in progress, and every access after it raises ObjectDisposedException. zip hands out the archive itself and bypasses the gate.

Constructors

NpzFile(Stream, bool, bool, long)

Open an archive over a stream - NumPy's NpzFile(fid, own_fid=False, allow_pickle=False, *, max_header_size=10000).

public NpzFile(Stream fid, bool own_fid = false, bool allow_pickle = false, long max_header_size = 10000)

Parameters

fid Stream

A readable, seekable stream holding the ZIP archive (NumPy's file object).

own_fid bool

When true, closing the archive also closes fid, and fid reports it (NumPy keeps self.fid only for an owned file).

allow_pickle bool

Whether members are trusted; see ReadArray(Stream, bool, long).

max_header_size long

Per-member header size cap.

Remarks

NumPy's names since API-parity batch 13 (they were stream, ownStream, allowPickle, maxHeaderSize); NumPy's pickle_kwargs has no NumSharp counterpart (its unpickler takes no Python keyword dictionary).

Exceptions

ArgumentNullException

fid is null.

InvalidDataException

The stream is not a ZIP archive (an owned stream is closed first).

NpzFile(string, bool, long)

Open an archive from a file path (NumPy's NpzFile(fid) with a path, which zipfile opens). The file handle is owned: fid reports it and close() closes it.

public NpzFile(string fid, bool allow_pickle = false, long max_header_size = 10000)

Parameters

fid string

The archive's path.

allow_pickle bool

Whether members are trusted; see ReadArray(Stream, bool, long).

max_header_size long

Per-member header size cap.

Exceptions

ArgumentNullException

fid is null.

FileNotFoundException

No such file.

InvalidDataException

The file is not a ZIP archive (the file is closed first).

Properties

this[string]

The array stored under key, with or without the .npy suffix. Loaded on first access and cached; every later access, from any thread, returns the same instance.

public NDArray this[string key] { get; }

Parameters

key string

The member name, with or without the .npy suffix.

Property Value

NDArray

The member's array, owned by the caller that first read it (the archive never disposes it).

Remarks

A cached member is returned without a lock. A member that is not cached yet is loaded under the archive's gate, so a thread loading one member waits for a thread loading another.

Exceptions

ObjectDisposedException

The archive was disposed.

KeyError

No such member: NumPy's KeyError, whose text is Python's str() of it ('x is not a file in the archive', quoted).

FormatException

The member is not a .npy file — use GetRawBytes(string) — or its header is malformed.

NotSupportedException

The member holds a dtype NumSharp cannot load (an object array, say).

InvalidDataException

A compressed member's data is corrupt.

allow_pickle

Whether members are loaded as trusted input (NumPy's allow_pickle).

public bool allow_pickle { get; }

Property Value

bool

f

Dot-notation access to members — NumPy's npz.f.weights (its BagObj). Requires a dynamic receiver: NDArray w = npz.f.weights;. Null once close() ran, as NumPy's self.f = None.

public dynamic f { get; }

Property Value

dynamic

fid

The file NumPy's fid names: the stream this archive OWNS (a path's file, or a stream given with own_fid: true), null for a borrowed stream (NumPy keeps self.fid only when it owns it) and after close().

public Stream fid { get; }

Property Value

Stream

files

The array names in the archive, with .npy stripped — NumPy's .files (a repeated name appears twice, as NumPy's list built from the zip's name list).

public IReadOnlyList<string> files { get; }

Property Value

IReadOnlyList<string>

max_header_size

The per-member header size cap (NumPy's max_header_size).

public long max_header_size { get; }

Property Value

long

zip

The underlying archive (NumPy's zip), for callers that need entry metadata; null once close() ran, as NumPy's. Reading entries through it bypasses the gate that serializes this class's own reads, so do not read it while other threads load members.

public ZipArchive zip { get; }

Property Value

ZipArchive

The open archive, or null after close().

Methods

ContainsKey(string)

Whether the archive has this member (with or without the .npy suffix).

public bool ContainsKey(string key)

Parameters

key string

Returns

bool

Dispose()

Close the archive and release an owned file handle (close(), for using).

public void Dispose()

Remarks

Takes the archive's gate, so a member another thread is loading finishes first (a large member can make this wait), and that thread's next access raises ObjectDisposedException. The arrays already handed out stay valid: they belong to the callers that read them. Calling it again does nothing.

GetEnumerator()

Enumerate every member as a name/array pair, loading each in turn.

public IEnumerator<KeyValuePair<string, NDArray>> GetEnumerator()

Returns

IEnumerator<KeyValuePair<string, NDArray>>

GetRawBytes(string)

A member's raw bytes, whatever it holds. NumPy returns these from its indexer for non-.npy members; for a .npy member this is the encoded file itself.

public byte[] GetRawBytes(string key)

Parameters

key string

The member name, with or without the .npy suffix.

Returns

byte[]

A new array holding the member's bytes.

Remarks

Capped at ~2 GB by Array itself. A member larger than that can still be read as an array through the indexer, which streams instead of buffering. Reads the archive under its gate, like a load.

Exceptions

ObjectDisposedException

The archive was disposed.

KeyError

No such member (the indexer's NumPy KeyError and text).

InvalidDataException

A compressed member's data is corrupt.

IsArray(string)

Whether key names a member that holds a .npy array.

public bool IsArray(string key)

Parameters

key string

The member name, with or without the .npy suffix.

Returns

bool

True when the member is cached or starts with the .npy magic string; false when there is no such member or it is not a .npy file.

Remarks

A cached member answers without a lock; otherwise the magic check reads the archive under its gate.

Exceptions

ObjectDisposedException

The archive was disposed.

ToString()

Formatted as NumPy's repr: NpzFile 'model.npz' with keys: a, b, c.

public override string ToString()

Returns

string

TryGetValue(string, out NDArray)

The array under key, or false if there is no such member.

public bool TryGetValue(string key, out NDArray value)

Parameters

key string
value NDArray

Returns

bool

close()

Close the archive and release an owned file handle — NumPy's close() (zip, fid and f become null).

public void close()

get(string, NDArray)

The array under key, or default when there is no such member — NumPy's get(key, default=None) (the Mapping method: no KeyError).

public NDArray get(string key, NDArray @default = null)

Parameters

key string

The member name, with or without the .npy suffix.

default NDArray

The answer for a missing member; null (NumPy's None) by default. A C# number converts.

Returns

NDArray

The member's array (the indexer's instance), or default.

Exceptions

ObjectDisposedException

The archive was disposed.

FormatException

The member is not a .npy file.

items()

Every member as a name / array pair — NumPy's items(). Enumerating it loads and caches every member.

public IEnumerable<(string key, NDArray value)> items()

Returns

IEnumerable<(string key, NDArray value)>

The pairs, in the archive's order (foreach (var (name, array) in z.items())).

Exceptions

ObjectDisposedException

The archive was disposed (checked as each member is read).

FormatException

A member is not a .npy file.

keys()

The member names — NumPy's keys(), the same names as files.

public IEnumerable<string> keys()

Returns

IEnumerable<string>

The names, in the archive's order.

Exceptions

ObjectDisposedException

The archive was disposed.

values()

Every member's array — NumPy's values(). Enumerating it loads and caches every member.

public IEnumerable<NDArray> values()

Returns

IEnumerable<NDArray>

The arrays, in the archive's order (each the instance the indexer returns).

Exceptions

ObjectDisposedException

The archive was disposed (checked as each member is read).

FormatException

A member is not a .npy file.