Strings, bytes and character arrays
NumSharp has NumPy's fixed-width Unicode (U), fixed-width bytes (S) and variable-width StringDType (T) dtypes. String arrays use the same indexing, views, broadcasting and file formats as other NDArrays. np.strings provides elementwise string operations, from concatenation and search to case conversion and encoding, without a Python runtime.
np.@char is the C# spelling of numpy.char, and NDCharArray is its chararray subclass. Neither is the same thing as NumSharp's existing Char dtype, which stores individual .NET UTF-16 code units. Use the table below to choose deliberately.
Choosing a text representation
| What you need | Use | Important consequence |
|---|---|---|
| A fixed number of Unicode code points per element | np.array(new[] { "cat", "bird" }, "U8") |
32 bytes per element; longer writes truncate to 8 code points |
| Fixed-width byte strings / binary fields | np.dtype("S8") |
8 bytes per element; item() returns byte[]; conversion from Unicode uses ASCII |
| Text of varying lengths, optionally with missing values | np.dtypes.StringDType() or np.dtype("T") |
No fixed text width; each element has a 16-byte cell plus any separately stored payload |
| NumPy's legacy whitespace-stripping character-array behavior | np.@char.array(...) → NDCharArray |
S / U storage with different element-access and comparison rules |
| A .NET array of UTF-16 code units | np.array(new[] { 'a', 'b' }) → Char |
2 bytes per element, unsigned-integer behavior; not U1, S1 or StringDType |
using NumSharp;
using var scope = NDScope.Open(); // Reclaim arrays created by this example when it ends.
var labels = np.array(new[] { "pear", "fig", "apple", "fig" }); // U5
var sorted = np.sort(labels); // ['apple' 'fig' 'fig' 'pear']
var (values, counts) = np.unique(labels, return_counts: true); // ['apple' 'fig' 'pear'], [1 2 1]
var repeated = np.strings.multiply(labels, 2); // ['pearpear' 'figfig' 'appleapple' 'figfig']
var nullable = np.dtypes.StringDType(na_object: null);
var notes = np.array(new object[] { "short", null }, nullable);
notes[0] = "This text can grow without a fixed-width truncation.";
var missing = notes.item(1); // null, not the text "None"
Examples below show focused calls; use using or an NDScope around array work in an application. NDListArray split results and np.nditer iterators are disposable holders: dispose those explicitly, since an NDScope reclaims arrays, not holders.
Use np.strings for elementwise text operations and the usual np.* functions for sorting, searching, unique values, indexing and reshaping. np.@char provides NumPy's legacy string API and the NDCharArray class; its split functions return NDListArray, a container of per-element lists of pieces.
Fixed-width string dtypes
| NumPy dtype | Descriptor | Bytes per character | Example | itemsize |
|---|---|---|---|---|
str_ |
U, <U5 |
4 (one UCS-4 code point, native byte order) | np.dtype("U5") |
20 |
bytes_ |
S, \|S5 |
1 | np.dtype("S5") |
5 |
The width is part of the dtype. Every element of a <U5 array holds up to five characters:
- A shorter value is padded with NUL characters.
- Reading an element removes the trailing NULs. NULs inside the value stay.
- A value that is too long is cut to the width when you write it, as NumPy does.
var names = np.array(new[] { "cat", "dog", "bird" });
names.dtype; // dtype('<U4') - the longest value sets the width
names.dtype.itemsize; // 16 - 4 characters x 4 bytes
np.dtype("S5").str; // "|S5"
np.dtype("U5").name; // "str160" - NumPy's name counts bits
The data buffer holds the characters directly, as in NumPy. A U character is a 4-byte code point. An S character is one byte:
np.array(new[] { "ab" }).tobytes(); // 61 00 00 00 62 00 00 00
np.array(new[] { "ab" }).astype("S3").tobytes(); // 61 62 00
A character outside the Basic Multilingual Plane (an emoji) is ONE U code point. C# stores it as two UTF-16 chars, but np.array(new[] { "😀" }) is <U1.
Creating string arrays
np.array(new[] { "cat", "dog" }); // ['cat', 'dog'], <U3
np.array(new[] { "cat", "dog" }, np.dtype("S5")); // [b'cat', b'dog'], |S5
np.zeros(new Shape(2), np.dtype("U3")); // ['', ''], <U3
np.full(new Shape(2), "ab"); // ['ab', 'ab'], <U2
NDArray.Scalar("abc"); // 'abc', a 0-d <U3 array
np.frombuffer(Encoding.ASCII.GetBytes("abcdef"), "S3"); // [b'abc', b'def'], no copy
A C# string is one text value. A byte[] passed directly to np.array is a numeric uint8 array, not a byte-string scalar. Build an S array from byte-string values with an object[], or use NDArray.Scalar((object)bytes) for one S scalar:
np.array(new object[] { Encoding.ASCII.GetBytes("ab"), Encoding.ASCII.GetBytes("cde") }); // [b'ab', b'cde'], |S3
Footgun:
np.array(new[] { bytes1, bytes2 })with abyte[][]is a jagged NUMERIC array in C#. It becomes a 2-Duint8array, not anSarray.
NumPy's character dtype np.dtype("c") is an |S1 that keeps the type character 'c'. Creating an array from values with it splits every text into one byte per character, so equal-length rows make a 2-D byte map:
np.array(new[] { "SFFF", "FHFH" }, np.dtype("c")); // [[b'S' b'F' b'F' b'F'] [b'F' b'H' b'F' b'H']], (2, 4) |S1
np.array("abc", np.dtype("c")); // [b'a' b'b' b'c'], (3,)
np.array(new[] { "ab", "abc" }, np.dtype("c")); // ValueError: ... inhomogeneous shape after 1 dimensions ...
A cast never splits: astype("c") and np.asarray(existingArray, "c") keep each value's first byte, as S1 does. Pass the C# string[] to np.array, not np.asarray, which converts it to a <U array before it reads the dtype.
Reading and writing elements
names.item(1); // "dog" - a C# string
names.astype("S4").item(1); // byte[] { 100, 111, 103 } - b'dog'
NDArray.AsStringArray(names); // { "cat", "dog", "bird" }
names[1]; // array('dog', dtype='<U3') - NumPy's scalar np.str_('dog'), a copy as wide as its text
names[1] = "zebras"; // the array is now ['cat', 'zebr', 'bird']: cut to 4 characters
names[2] = null; // NumPy's None is stored as its str(): ['cat', 'zebr', 'None']
GetString, SetString, GetStringAt and SetStringAt belong to the Char dtype: they read a vector of UTF-16 chars as one string. They refuse a U or S array with an InvalidOperationException.
Dtype integration
String descriptors take part in promotion and in the casting rules exactly as in NumPy:
np.promote_types(np.dtype("S3"), np.dtype("U2")); // dtype('<U3')
np.result_type(new[] { "U3", "S7" }); // dtype('<U7')
np.can_cast(np.dtype("S3"), np.dtype("U3"), "safe"); // true - ASCII bytes fit in Unicode
np.can_cast(np.dtype("U3"), np.dtype("S3"), "same_kind"); // false
np.can_cast(np.dtype("U5"), np.dtype("U3"), "safe"); // false - the cast would cut values
Moving data
Views, copies and element movement work on string arrays as on numeric ones: slicing, transposes, reshapes, copy, concatenate, take, boolean masks, where, pad, copyto, put and the other indexing routines.
np.pad pads text in every mode that moves elements: constant (with text, or a number written as text), edge, reflect, symmetric, wrap and empty. maximum and minimum pad a StringDType array with its extremes; NumPy has no such loop for S / U and raises. mean, median and linear_ramp compute with numbers and raise NumPy's errors for text.
Casts between string dtypes follow NumPy:
- A narrower width cuts values.
UtoSencodes ASCII.StoUdecodes ASCII.
np.concatenate(new[] { np.array(new[] { "a" }), np.array(new[] { "long" }) }); // ['a', 'long'], <U4
names.astype("U2"); // ['ca', 'do', 'bi']
np.array(new[] { "é" }).astype("S1"); // UnicodeEncodeError
The last call raises UnicodeEncodeError with Python's message: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128). Its encoding, @object, start, end and reason are Python's: ascii, é, 0, 1, ordinal not in range(128).
Converting between text and numbers
Numbers to text
A number becomes the text of its NumPy scalar's str(). An unsized "U" or "S" request takes the width NumPy reserves for the number type: 5 for bool, 21 for int64, 32 for the floats, 64 for complex. A sized request cuts the text to its width:
np.arange(3).astype("U"); // ['0' '1' '2'], <U21
np.array(new[] { 1.5, 1e16, 1e-5 }).astype("U"); // ['1.5' '1e+16' '1e-05'], <U32
np.array(new[] { true, false }).astype("U"); // ['True' 'False'], <U5
np.array(new[] { 0.1f }).astype("U"); // ['0.1'] - float32 prints its own shortest digits
np.array(new[] { 3.14159 }).astype("U3"); // ['3.1']
The same text is written wherever a number goes into a text array: an element, fill, np.full, np.ones, np.pad, np.copyto, and a masked array's fill value:
var u = np.array(new[] { "ab", "cd" });
u[0] = 5; // ['5' 'cd']
u[1] = 2.5; // ['5' '2.'] - cut to the width
np.full(new Shape(2), 7, "U"); // ['7' '7'], <U1
np.ones(new Shape(2), "U"); // ['1' '1'], <U1
np.pad(np.array(new[] { "ab", "c" }), 1); // ['0' 'ab' 'c' '0']
Text to numbers
A cast to a number parses each element. int() takes decimal digits with an optional sign, underscores between digits and surrounding whitespace. float() also takes inf, nan and exponents. complex() takes 1+2j forms, with or without parentheses. An element that does not parse raises NumPy's ValueError, and an integer that does not fit raises NumPy's OverflowError:
np.array(new[] { "1", " 22 ", "-3", "1_000" }).astype(np.int64); // [ 1 22 -3 1000]
np.array(new[] { "1.5", "inf", "-nan", "1e3" }).astype(np.float64); // [ 1.5 inf nan 1000. ]
np.array(new[] { "1+2j", "3", "(4-5j)" }).astype(np.complex128); // [1.+2.j 3.+0.j 4.-5.j]
np.array(new[] { "1", "x" }).astype(np.int64); // ValueError: invalid literal for int() with base 10: np.str_('x')
np.array(new[] { "1", "a" }).astype(np.float64); // ValueError: could not convert string to float: np.str_('a')
np.array(new[] { "300" }).astype(np.int8); // OverflowError: Python integer 300 out of bounds for int8
A bytes (S) element parses the same way. The error names the first element that fails, in NumPy's order.
A C# string written into a numeric array is NumPy's Python str. It parses the same way, and its error quotes the string itself:
var a = np.zeros(new Shape(3), np.int32);
a[0] = "12"; // [12 0 0]
a[0] = "hello"; // ValueError: invalid literal for int() with base 10: 'hello'
Conversions inside other calls
NumPy converts between text and numbers inside other calls too, and NumSharp follows:
np.add(np.array(new[] { 1.5, 2.0 }), np.array(new long[] { 1, 2 }), dtype: "U"); // ['1.51' '2.02'], <U53
np.equal(names, np.array(new[] { "cat", "cow", "bird" }), np.empty(new Shape(3), "U5")); // ['True' 'False' 'True']
np.add(np.array(new[] { 1.0, 2.5 }), np.array(new[] { 1.0, 3.0 }), np.zeros(new Shape(2), "U5")); // ['2.0' '5.5']
np.prod(np.array(new[] { "1.5", "2", "4" }), (int?)null, np.float64); // 12.0 - the text is parsed first
np.addwith a stringdtype=writes each number as text, then concatenates.- A ufunc with a text
out=writes each result as text, cut to the width ofout. - A reduction with a numeric
dtype=parses the text first. - A reduction or accumulation into a text
out=resolves its loop as NumPy does.np.cumsumofUtext into aStringDTypeout concatenates (['c' 'ca' 'cab']for['c', 'a', 'b']). Integer counts reduced byprodinto a text out repeat the first count's text. A numeric loop parses a text input and writes its running values into a text out.np.medianandnp.percentileinto a text out leave NumPy's partial result there before NumPy's refusal.
With where=, NumPy also parses the CURRENT text of out into the loop's number type before it computes, so a masked-off 'x' in a float loop raises ValueError.
Breaking change: a C# string converts to a 0-d
Uarray, asnp.array(str)does in NumPy.NDArray x = "[1, 2, 3]"isarray('[1, 2, 3]', dtype='<U9'). It no longer parses a numeric list.astypeparses an individual numeric text (np.array("1.5").astype(np.float64)), not list syntax. For delimited numbers usenp.fromstring("1, 2, 3", sep: ",")without the brackets.
Printing
ToString() is NumPy's str(), and ToString(true) is NumPy's repr(), including Python's quoting rules:
names.ToString(); // ['cat' 'dog' 'bird']
names.ToString(true); // array(['cat', 'dog', 'bird'], dtype='<U4')
np.array2string(np.array(new[] { "a'b", "c\"d" })); // ["a'b" 'c"d']
Computing on text
Comparisons
==, !=, <, <=, >, >= and the ufuncs np.equal … np.greater_equal compare two text arrays of one class (U with U, S with S), of any widths, as NumPy does. Strings compare code unit by code unit as unsigned values. Trailing NULs never count ('a' == 'a\0'), embedded NULs do. Broadcasting, every memory layout, out=, where= and dtype=bool work as for numbers. A C# string against a text array is NumPy's str scalar:
names == names; // [ True True True]
names < np.array(new[] { "dog" }); // [ True False True]
names > "cat"; // [False True False]
The names np.strings.equal, not_equal, less, less_equal, greater and greater_equal are the same comparison ufuncs; they do not strip whitespace.
A bytes array against a str array has no comparison loop in NumPy: == is all False, != all True, and < raises NumPy's TypeError.
Truth values
A string is True when it is not empty, that is when any code unit is not NUL. A string of spaces and the text 'False' are True. astype(bool), np.any, np.all, np.nonzero, np.count_nonzero, np.where, np.compress, np.logical_and / np.logical_or / np.logical_xor, np.place and np.putmask masks, and np.trim_zeros read it. np.logical_not has no text loop, as in NumPy; invert an explicitly computed Boolean array instead:
np.array(new[] { "", "a", " " }).astype(np.@bool); // [False True True]
np.count_nonzero(names); // 3
Concatenation and repetition
np.add and the + operator concatenate two text arrays of one class, element by element. The result is as wide as the two widths together. Trailing NULs never count, embedded NULs do. A C# string is NumPy's str scalar. A string on the left stays the first operand. np.strings.add is np.add:
names + "s"; // ['cats' 'dogs' 'birds'], <U5
"a " + names; // ['a cat' 'a dog' 'a bird'], <U6
np.add(names, np.array(new[] { "!", "?", "" })); // ['cat!' 'dog?' 'bird'], <U5
np.strings.multiply repeats each string. The result is as wide as the longest repetition. A count less than 1 gives an empty string:
np.strings.multiply(names, 2); // ['catcat' 'dogdog' 'birdbird'], <U8
np.strings.multiply(names, np.array(new long[] { 1, 0, -1 })); // ['cat' '' ''], <U3
The np.multiply ufunc cannot size its result, so it needs out=. It writes each repetition cut to the width of out. Without out=, np.multiply(names, 2) and names * 2 raise NumPy's TypeError, as in NumPy:
np.multiply(names, 2, np.zeros(new Shape(3L), "U5")); // ['catca' 'dogdo' 'birdb']
Broadcasting, every memory layout, out= (a narrower or wider text array receives the result cut or padded) and where= work as for numbers. A bytes array with a str array has no loop: np.add raises NumPy's UFuncBinaryResolutionError (a TypeError). A repetition longer than 2^63 - 1 characters raises NumPy's OverflowError.
Length and substring search
np.strings applies the familiar string operations element by element. String arguments and bounds broadcast against the input, so each element can have its own search text, start and end:
var words = np.array(new[] { "cat", "dog" });
np.strings.str_len(words); // [3 3]
np.strings.find(words, "a"); // [1 -1]
np.strings.startswith(words, "c"); // [True False]
np.strings.count(np.array(new[] { "banana", "pear" }), "a"); // [3 1]
| Function | Result |
|---|---|
str_len(a) |
Length of each element, as int64 |
find(a, sub, start, end) / rfind |
First / last matching position, or -1 |
index(a, sub, start, end) / rindex |
Matching position; raises ValueError("substring not found") if absent |
count(a, sub, start, end) |
Number of non-overlapping matches |
startswith(a, prefix, start, end) / endswith |
Boolean array |
Positions count bytes for S, Unicode code points for U and StringDType. For example, str_len of "é中😀" is 3 for either Unicode dtype, not its UTF-16 or UTF-8 byte count. Fixed-width trailing NUL padding does not contribute to the length. An S search takes an S argument; a C# string is U, so use a byte-string scalar rather than mixing the two classes.
Stripping, padding, replacing and slicing
| Function | Behavior |
|---|---|
strip, lstrip, rstrip |
Remove whitespace or a supplied set of characters from both ends / the left / the right |
replace(a, old, new, count) |
Replace occurrences, optionally limiting their number |
center, ljust, rjust |
Pad to a width with fillchar |
zfill(a, width) |
Pad with zeros after a leading sign |
expandtabs(a, tabsize) |
Expand tabs at tab stops |
slice(a, start, stop, step) |
Slice the characters inside each element, not the array's axes |
np.strings.strip(np.array(new[] { " cat ", "dog\t" })); // ['cat' 'dog']
np.strings.replace(words, "a", "oo"); // ['coot' 'dog']
np.strings.center(words, 7, fillchar: "-"); // ['--cat--' '--dog--']
np.strings.zfill(np.array(new[] { "-7", "12" }), 4); // ['-007' '0012']
np.strings.slice(words, 0, 2); // ['ca' 'do']
The output-width rule belongs to the function. Stripping and character slicing keep a fixed-width input's dtype; padding and replacement size their output from the resulting text. StringDType results need no fixed width. An array slice such as words["0:1"] selects elements and can share memory; np.strings.slice transforms the text and produces a new result.
strip uses ASCII whitespace for S and Python's Unicode whitespace for U and StringDType. This differs from the legacy np.@char comparisons, which strip only ASCII whitespace even for a U operand.
Partitioning each string
np.strings.partition(a, sep) returns three arrays: text before the first separator, the separator itself, and text after it. rpartition uses the last separator. When the separator is absent, partition puts the original text in the first part; rpartition puts it in the last.
var fields = np.array(new[] { "a-b", "c" });
var parts = np.strings.partition(fields, "-"); // ['a' 'c'], ['-' ''], ['b' '']
np.@char.partition(fields, "-"); // [['a' '-' 'b'], ['c' '' '']]
The np.@char forms stack the three parts on a new last axis, so their shape is the broadcast input shape followed by 3. NDCharArray.partition(sep) follows this string operation, not the sorting operation np.partition(a, kth).
For S / U, NumSharp returns three arrays where NumPy returns field views of one structured array; values match, but ownership and strides differ. A StringDType input requires a StringDType separator too: create one with np.array("-", np.dtype("T")). A plain C# string is a U separator and has no matching T partition loop in NumPy.
Unicode properties and case conversion
isalpha, isalnum, isdigit, isdecimal, isnumeric, isspace, islower, isupper and istitle return Boolean arrays. Byte strings use ASCII rules; isdecimal and isnumeric accept U and StringDType, not S. Unicode properties and full case mappings follow CPython 3.12's Unicode 15.0.0 database, rather than the machine's culture or the installed .NET Unicode version.
var numbers = np.array(new[] { "12", "²", "½" });
np.strings.isdecimal(numbers); // [True False False]
np.strings.isdigit(numbers); // [True True False]
np.strings.isnumeric(numbers); // [True True True]
upper, lower, swapcase, capitalize and title perform Python's case conversion. A mapping may expand one code point into several. Fixed-width results keep the input dtype, so an expansion can be cut; use StringDType when it must fit in full:
np.strings.upper(np.array(new[] { "ßa" })); // ['SS'], U2
np.strings.upper(np.array(new[] { "ßa" }, np.dtype("T"))); // ['SSA'], StringDType()
translate applies a translation table and also keeps the input dtype. A Unicode table maps code points to replacements or deletion; the bytes form uses a 256-byte table and optional deletechars. deletechars is ignored for Unicode, as in NumPy.
Formatting, encoding and decoding
mod(a, values) applies Python's % formatting to each element; the values broadcast against the format strings. encode converts Unicode to byte strings, and decode converts byte strings to Unicode:
np.strings.mod("item-%02d", np.array(new long[] { 1, 12 })); // ['item-01' 'item-12']
var encoded = np.strings.encode(np.array(new[] { "café" }), "utf-8"); // S5: 63 61 66 C3 A9
np.strings.decode(encoded, "utf-8"); // ['café'], U4
Unlike astype("S"), which encodes ASCII, encode takes an explicit codec. Supported codecs include UTF-8, UTF-16, UTF-32, ASCII, Latin-1 and the single-byte code pages listed under Text encodings. The default is UTF-8 with errors: "strict". Python's ignore, replace, backslashreplace, xmlcharrefreplace, namereplace, surrogateescape and surrogatepass handlers are available where the codec and direction allow them. An unknown codec raises LookupError; an unsupported codec raises NotSupportedException.
Formatting accepts scalar or array values, not an object array of per-element tuples or dictionaries. A tuple of values is not automatically one formatting tuple per element. Fixed-width results of mod, encode and decode are sized to the longest result, rather than the input width.
Sorting
np.sort, np.argsort, ndarray.sort, np.partition and np.argpartition sort S and U text as NumPy does: by code point (U) or byte value (S), compared unsigned, a value before every longer value that starts with it. Every memory layout and every axis works, and axis: null sorts the flattened array:
var words = np.array(new[] { "pear", "fig", "apple", "fig" });
np.sort(words); // ['apple' 'fig' 'fig' 'pear']
np.argsort(words); // [2 1 3 0]
np.sort(np.array(new[] { "b", "A", "a", "é", "Z" })); // ['A' 'Z' 'a' 'b' 'é'] - code-point order
words.sort(); // in place
NumPy's default sort is an introsort, which is not stable: the order of equal strings depends on the whole sort. NumSharp reproduces that order, so np.argsort returns NumPy's indices for every kind. kind: "stable" (or stable: true) keeps equal strings in their input order:
var grades = np.array("babacabcabacbacbabac".Select(c => c.ToString()).ToArray());
np.argsort(grades); // [ 1 3 16 5 13 8 18 10 0 17 15 12 9 6 2 7 14 4 11 19]
np.argsort(grades, kind: "stable"); // [ 1 3 5 8 10 13 16 18 0 2 6 9 12 15 17 4 7 11 14 19]
descending: true sorts the largest text first (NumPy 2.5). The default kind runs NumPy's introsort in reverse, whose order of equal strings NumSharp reproduces too; the stable kind keeps equal strings in their input order:
np.sort(words, descending: true); // ['pear' 'fig' 'fig' 'apple']
np.argsort(grades, descending: true); // [19 14 4 11 7 17 15 12 0 6 2 9 18 10 5 13 3 16 1 8]
np.argsort(grades, stable: true, descending: true); // [ 4 7 11 14 19 0 2 6 9 12 15 17 1 3 5 8 10 13 16 18]
The keywords follow NumPy's rules:
kindis read by its first letter, in either case:quicksort (the default),heapsort (the default kind: NumPy clears its bit before it looks the sort up),mergesort andstable. Any other value is NumPy'sValueError.kindcannot be given together withstableordescending,falseincluded.orderis refused, because no NumSharp dtype has fields.- NumPy has no partition for text and sorts it instead, so
np.partitionreturns the sorted text andnp.argpartitiontheargsortindices.
np.sort(words, kind: "timsort"); // ValueError: sort kind must be one of 'quick', 'heap', or 'stable' (got 'timsort')
np.sort(words, kind: "stable", stable: true); // ValueError: `kind` and keyword parameters can't be provided at the same time. ...
Measured against NumPy 2.4.2 on the same machine (pinned to two cores): over fresh inputs of 1,000 strings every np.sort, np.argsort, np.partition and np.argpartition measured ran at least 1.58 times as fast as NumPy (3 times in the geometric mean), and over 100,000 and 1,000,000 strings they ran 1.7 to 19 times as fast. The slowest cases are argsorts of text outside Latin-1 (1.7 to 2 times NumPy's speed), whose characters take more room in the sort key, and argsorts of text with many repeated values (about twice NumPy's speed): NumSharp reproduces NumPy's order of the equal strings by replaying its introsort.
Searching
np.argmax, np.argmin, np.nanargmax, np.nanargmin (and the ndarray methods), np.searchsorted (and ndarray.searchsorted) and np.digitize search S and U text with NumPy's results. They compare as the sorts do: by code point (U) or byte value (S), unsigned.
var words = np.array(new[] { "pear", "fig", "apple", "fig" });
np.argmax(words); // 0 ('pear')
np.argmin(words); // 2 ('apple')
np.argmax(np.array(new[,] { { "b", "a" }, { "a", "c" } }), 1); // [0 1]
var sorted = np.sort(words); // ['apple' 'fig' 'fig' 'pear']
np.searchsorted(sorted, np.array(new[] { "fig", "kiwi" })); // [1 3]
np.searchsorted(sorted, np.array(new[] { "fig", "kiwi" }), "right"); // [3 3]
The arg-reductions follow NumPy's rules:
- Of several equal extremes, the first wins.
keepdimskeeps the reduced axis.out=takes an integer array NumPy accepts:boolthroughuint32and the signed types.uint64and floats raise itsTypeError.np.nanargmaxandnp.nanargminof text arenp.argmaxandnp.argmin: text has no NaN.
np.searchsorted first converts the keys and the haystack to their common string dtype, as NumPy does. A number is written as text, so np.searchsorted(np.array(new[] { "1", "10", "2", "5" }), 3) searches for "3" and returns 3. Bytes beside str are decoded.
Over an unsorted haystack, NumPy's answer comes from its exact sequence of comparisons, and NumSharp follows that sequence. np.digitize of text searches the bins converted to text, so the bins follow text order, not number order:
np.digitize(np.array(new[] { "10", "3", "25" }), np.array(new[] { 5, 20 })); // [0 2 2]
Measured against NumPy 2.4.2 on the same machine (pinned to two cores, fresh inputs every call): the arg-reductions ran 3.4 times as fast as NumPy in the geometric mean, and np.searchsorted 4.6 times as fast (6 to 10 times over sorted haystacks of 100,000 strings, 1.4 to 4 times over unsorted ones, whose searches follow NumPy's exact sequence of comparisons). Two kinds of call stay close to NumPy's speed, because both libraries spend their time waiting on memory rather than comparing:
- An arg-reduction over 100,000 fresh strings (1.1 to 1.5 times NumPy's speed). It reads every cache line of the array, at the memory's bandwidth.
- A few keys searched in a long haystack (100 keys in 100,000 strings, 0.7 to 1.4 times NumPy's speed). Each step of a binary search reads an element whose address depends on the previous comparison, so the search waits on memory at every step.
Unique values, set routines and statistics
np.unique (every return_* flag, axis, equal_nan and sorted), np.unique_values, np.unique_counts, np.unique_inverse, np.unique_all, np.isin, np.intersect1d, np.union1d, np.setdiff1d, np.setxor1d, np.lexsort, np.sort_complex, np.histogram, np.histogram_bin_edges and the selection methods of np.quantile and np.percentile work on S and U text with NumPy's results. They order text as the sorts do.
var fruit = np.array(new[] { "pear", "fig", "apple", "fig", "pear" });
var (values, counts) = np.unique(fruit, return_counts: true); // ['apple' 'fig' 'pear'], [1 2 2]
np.isin(fruit, np.array(new[] { "fig", "kiwi" })); // [False True False True False]
np.intersect1d(fruit, np.array(new[] { "pear", "plum" })); // ['pear']
np.setdiff1d(fruit, np.array(new[] { "pear" })); // ['apple' 'fig']
np.quantile(fruit, 0.5, method: "lower"); // 'fig'
np.quantile(fruit, 0.5); // TypeError: unsupported operand type(s) for -: 'numpy.str_' and 'numpy.str_'
Some results follow from how NumPy computes them:
Numbers and bytes beside text.
np.isinand the set routines join their two operands into one array, as NumPy does. Numbers are written as text, and bytes are decoded as ASCII (a byte past ASCII raises NumPy'sUnicodeDecodeError). With few test values,np.isincompares the elements instead, and text never equals a number there. NumPy switches between the two at10 * len(ar1) ** 0.145test values:np.isin(np.array(new[] { "1", "2" }), np.arange(20)); // [ True True]: 20 values are joined np.isin(np.array(new[] { "1", "2" }), np.array(new long[] { 1 })); // [False False]: 1 value is comparedThe order of
np.unique_values. NumPy returns the distinct strings in the order of its hash table, which depends on the C++ library NumPy was built with. NumSharp returns them sorted; the values are the same.np.unique(..., sorted: false)without areturn_*flag differs the same way.np.sort_complexsorts the text first and parses it afterwards, so the result follows the text's order:np.sort_complex(np.array(new[] { "10", "9", "2+1j" }))is[10.+0.j 2.+1.j 9.+0.j].np.histogramof text needs explicit edges, as text or as numbers. Number edges are compared as text, so a count can come out negative, as in NumPy:np.histogram(np.array(new[] { "1", "5", "10", "30", "2" }), np.array(new long[] { 0, 5, 10 }))counts[ 4 -2]("10"sorts before"5"as text).np.quantileandnp.percentilecompute the methods that pick an element:lower,higher,nearest,inverted_cdf,closest_observation, andlinearwith an integerq(np.quantile(fruit, 1)is'pear'). The other methods interpolate, which subtracts text, so NumPy raises aTypeErrorand NumSharp raises the same one.np.percentiledividesqby 100 first, so itslinearmethod always interpolates.Weighted quantiles pick an element without interpolation, using only
method: "inverted_cdf". Weights must be non-negative and broadcast according to NumPy's reduced-axis rules. The selected value keeps the data's dtype:var fruit = np.array(new[] { "pear", "fig", "apple" }); np.quantile(fruit, 0.5, method: "inverted_cdf", weights: np.array(new[] { 1.0, 3.0, 1.0 })); // 'fig'
Measured against NumPy 2.4.2 on the same machine (pinned to two cores, fresh inputs every call, 1,000 and 100,000 strings): every call measured ran at least 1.5 times as fast as NumPy, 4.2 times in the geometric mean. np.unique of text with few distinct values hashes the strings and sorts only the distinct ones, as NumPy does; past 1 in 16 distinct strings it sorts every string, which is faster for mostly distinct text (11 times NumPy's speed on 100,000 random strings). The slowest cases, 1.5 to 2 times NumPy's speed, are weighted histograms (each block's weights are summed in NumPy's tie order) and np.unique along an axis of small arrays.
The numpy.char module
np.@char is NumPy's numpy.char module. char is a C# keyword, so NumPy's np.char.equal(a, b) is np.@char.equal(a, b). It exposes the string operations above, except slice, with these additional conventions:
equal,not_equal,less,less_equal,greaterandgreater_equalcompare two text arrays of one class after they remove the trailing NULs and ASCII whitespace (' ','\t','\n','\v','\f','\r') of each string. Other whitespace ('\xa0', U+3000) and leading spaces stay.compare_chararrays(a1, a2, cmp, rstrip)runs the comparisoncmp("==","!=","<","<=",">"or">=", as astringor abyte[]), with the strip whenrstripistrue. Any othercmpraises NumPy'sTypeError(a bytes-like object is required, not 'int').addisnp.add.multiplyisnp.strings.multiply, but a count that is not an integer gives NumPy'sValueError("Can only multiply by integers").partitionandrpartitionreturn one array with a last axis of length 3, rather than three arrays.array,asarrayandchararrayconstructNDCharArray.split,rsplit,splitlinesandjoincompute over text elements; see Splitting and joining.
var padded = np.array(new[] { "cat ", "dog\t", "bird" });
np.@char.equal(padded, "cat"); // [ True False False]
np.equal(padded, "cat"); // [False False False]
np.@char.compare_chararrays(padded, "dog", "<", true); // [ True False True]
np.@char.add(padded, "!"); // ['cat !' 'dog\t!' 'bird!'], <U6
np.@char.multiply(np.array(new[] { "ab", "c" }), 3); // ['ababab' 'ccc'], <U6
np.@char.multiply(padded, 2.5); // ValueError: Can only multiply by integers
The array arguments are NumPy's array_likes: an NDArray, a C# string (NumPy's str), a byte[] (NumPy's bytes in this module), a string[] or a byte[][] (a list of bytes). Two strings give a 0-d result. For the stripping comparisons, an S array against a U array is NumPy's NotImplemented, returned as null, and StringDType is refused with TypeError: comparison of non-string arrays. That restriction is not module-wide: add, multiply and the split/join family also accept StringDType.
np.@char also converts to the Char dtype, its meaning before the module existed, so astype(np.@char) and a.dtype == np.@char still work. Read a descriptor member through a cast: ((DType)np.@char).itemsize.
Measured against NumPy 2.4.2 (pinned to two cores, 1,000 to 1,000,000 strings): the comparisons ran 2.6 to 10 times as fast, np.@char.add 4.9 to 6.9 times, np.@char.multiply 1.5 to 5 times, np.@char.partition and rpartition 1.9 to 4.1 times. The slowest multiply case is a fresh 96 MB result, where both libraries pay the same page faults.
The chararray class
np.@char.chararray, np.@char.array and np.@char.asarray make NumPy's chararray: an NDCharArray, which is an NDArray of S or U text. A chararray differs from a plain string array in three ways:
- Element access removes the trailing whitespace of the element, as Python's
str.rstrip()does forUtext (29 whitespace characters: the ASCII ones,'\x1c'to'\x1f','\x85','\xa0', U+3000 and the other Unicode spaces) andbytes.rstrip()does forStext (the 6 ASCII whitespace bytes).item(),tolist(),take(i)and the flat iterator keep the whitespace. - Views and copies of a chararray are chararrays:
T,reshape,ravel,copy,astype,take,repeatand the other members it shares withNDArray. - A chararray holds text. A member that would make a chararray of numbers raises
ValueError: Can only create a chararray from string data.NumPy's rule also allowsint8data and NumSharp'sChardtype.
var c = np.@char.array(new[] { "cat ", "dog\t", "bird" });
c[0].item(); // "cat": a read-only 0-d chararray stands for Python's str
c.item(0); // "cat "
foreach (var s in c) // "cat", "dog", "bird"
Console.WriteLine(s);
c.ToString(true); // chararray(['cat', 'dog', 'bird'], dtype='<U5')
c.T; // a chararray view
c.astype(np.float64); // ValueError: Can only create a chararray from string data.
np.@char.array takes NumPy's arguments:
- A
stringfills a chararray oflen // itemsizeelements. Abyte[]is a read-only buffer ofStext. - A list, or an
S/Uarray, becomes a chararray view, a cast to another width or class, or a copy. - A number array, a
StringDTypearray and a C# number become text the way NumPy'stolist()route writes them:np.@char.array(np.array(new[] { 0.1f }))is[b'0.10000000149011612'].
np.@char.asarray is array with copy: false, and returns an S chararray itself. a.view(typeof(NDCharArray)) makes a chararray view of an S / U array, as NumPy's a.view(np.char.chararray) does.
One difference from NumPy: with order: and text data, NumPy's np.char.array returns a plain array, because its np.asarray call drops the class. NumSharp returns a chararray with the same data and layout.
String methods on a chararray
A chararray has the corresponding methods, so c.upper(), c.strip() and c.find("a") operate on its stored text. These methods do not first strip the input through c[i]. Their return types follow NumPy:
| Methods | Result |
|---|---|
capitalize, center, expandtabs, ljust, lower, partition, rjust, rpartition, swapcase, title, translate, upper, zfill |
NDCharArray; U results are copied by NumPy's chararray conversion, S results can be views of the computed array |
lstrip, rstrip, strip, replace |
A chararray view of the text result |
encode, decode, join |
A plain text NDArray |
count, find, rfind, index, rindex, startswith, endswith, and the character predicates |
A plain int64 or Boolean NDArray |
split, rsplit, splitlines |
NDListArray |
The % operator applies np.@char.mod and returns a chararray. As with other NDArray operators, C# compound assignment rebinds the variable rather than performing NumPy's in-place update.
Operators and functions with a chararray
The comparison operators of a chararray remove the trailing whitespace before they compare, as np.@char.equal does. They do this also when a plain array is on the left, because Python asks the chararray first:
var c = np.@char.array(new[] { "ab ", "b", "ab", "a " });
var u = np.array(new[] { "ab", "b", "ab ", "a" });
c == "ab"; // [ True False True False]
u == c; // [ True True True True]
c + "x"; // a chararray view, <U5: 'ab x', 'bx', 'abx', 'a x'
c * 2; // a chararray copy, <U8: 'ab ab ', 'bb', 'abab', 'a a '
c * 2.5; // ValueError: Can only multiply by integers
An S chararray against U text is NumPy's NotImplemented. Then a plain array on the other side compares without the strip, or Python's fallback answers: == is False, != is True, and < raises TypeError: '<' not supported between instances of 'chararray' and 'bytes'.
The members:
sort()sorts in place.argsort()andsearchsorted()return plain arrays.argmax()andargmin()raiseValueError: Can only create a chararray from string data., because NumPy creates their index result as a chararray; giving them a plainout:allows the result.argpartition()has the same finalize error but nooutparameter.partition(sep)is the string partition ofnp.@char.partition, not the ordering operationnp.partition(a, kth). Consequently, calls that pass an orderingaxiskeyword to the chararray method raise NumPy'sTypeError: chararray.partition() got an unexpected keyword argument 'axis'(np.partition(c, 1)and unweighted quantile/median paths can reach it).
The np.* functions treat a chararray the way NumPy does:
- The result is a chararray when NumPy calls a method of the array or views it with
np.asanyarray:np.reshape,np.ravel,np.transpose,np.squeeze,np.sort,np.repeat,np.tile,np.flip,np.take, the split functions,np.zeros_likeand the other*_likefunctions,np.copy(c, subok: true). - The result is a chararray view when it is the text result of a ufunc (
np.add,np.strings.multiply) or ofnp.insert,np.delete,np.uniqueand the other functions that wrap their result. The view does not own its data. - The result is a plain array when NumPy converts with
np.asarray:np.concatenate,np.stack,np.where,np.append,np.asarray,np.intersect1d,np.union1dandnp.setxor1d.np.setdiff1dinstead wraps its text result as a chararray view. - The call raises the
ValueErrorwhen NumPy creates a non-text result as a chararray:np.argmax/np.argminwithoutout:,np.argpartition, andnp.zeros_like(c, np.float64). A ufunc's preliminary cast can also trigger it: an unmasked logical ufunc copies a 0-d or short 1-d chararray into its bool loop dtype as a chararray (up tonp.getbufsize(), normally 8,192 elements). This is not a general size limit on chararray operations.
np.sort(c); // a chararray
np.concatenate(new NDArray[] { c, c }); // a plain array
np.argmax(c); // ValueError: Can only create a chararray from string data.
np.unique(c, return_counts: true); // values 'a ', 'ab', 'b' (a chararray), counts [1 2 1]
np.unique(c); // 'a ', 'ab', 'ab ', 'b': without a flag, no strip
The C# type of the variable decides which operators and members run, as for NDMatrix. A chararray in a variable of type NDArray uses the operators and members of NDArray: == does not strip, + returns a plain array, and argmax() returns the index. The np.* functions still find the chararray: np.reshape(plainVariable, ...) returns a chararray.
Other differences from NumPy:
c == nullandc != nullare reference tests that return a 0-dFalseandTrue. NumPy'sc == NoneraisesTypeError: comparison of non-string arrays.np.squeezeof a chararray with no axis of length 1 returns a new chararray view. NumPy returns the same object.- A view of an element (
np.atleast_2d(c[0])) is read-only, because the element is a read-only 0-d chararray. NumPy makes a writeable array from the Pythonstr. c + twith aStringDTypearraytof two or more dimensions raisesNotSupportedException, because NumPy's result is a plainStringDTypearray and the+ofNDCharArrayreturns a chararray. Callnp.add(c, t)for NumPy's result.- A masked array of a chararray (
np.ma.array(c)) holds the data as a plain array.
Measured against NumPy 2.4.2 (pinned to two cores, 1,000 to 1,000,000 strings): the comparison operators ran 2.8 to 6.1 times as fast, + and * 2.4 to 14 times, the np.* functions 1.3 to 9 times. The slowest cases are searchsorted (1.0 to 1.3 times NumPy's speed, the same as for a plain array: both libraries run one binary search per key), argmax with out: and a 1,000-string np.concatenate.
Splitting and joining: split, rsplit, splitlines and join
np.@char.split, rsplit and splitlines call Python's str or bytes method on every element, as NumPy does. The result is NumPy's object array of lists. NumSharp has no object dtype, so the result is an NDListArray: it has the shape of the input, and each element is the list of pieces of one string.
var words = np.array(new[] { "a b c", " lead trail ", "" });
using NDListArray w = np.@char.split(words); // [list(['a', 'b', 'c']) list(['lead', 'trail']) list([])]
np.@char.split(words, " "); // empty pieces where two separators touch
np.@char.rsplit(words, maxsplit: 1); // ['a b', 'c'], [' lead', 'trail'], []
np.@char.splitlines(np.array(new[] { "a\nb\r\nc" }), true); // [list(['a\n', 'b\r\n', 'c'])]
w.item(0); // a List<object>: "a", "b", "c"
w.tolist(); // the nested lists, every piece exact
w[0]; // the list of element 0: a 0-d list array, printed ['a', 'b', 'c']
w["1:"]; // an NDListArray view of the last two elements
w.ToString(true); // array([list(['a', 'b', 'c']), ...], dtype=object)
The rules are Python's:
- Without a separator, the pieces are the runs of characters that are not whitespace: 29 whitespace characters for
UandStringDTypetext, the 6 ASCII whitespace bytes forStext. With a separator, every occurrence splits, so empty pieces can occur.maxsplitlimits the number of splits, from the end forrsplit. splitlinesbreaksUtext at\n,\r,\r\n,\v,\f,\x1cto\x1e,\x85, U+2028 and U+2029, andStext at\n,\rand\r\nonly.keependsfollows Python's truth rule:0.5and"x"are true,0and""are false.- The separator,
maxsplitandkeependscan be arrays that broadcast against the text, one value per element. bytes.splitreads its separator as raw bytes, so a number works as a separator ofStext:np.@char.split(s, 5)splits on the 8 bytes of theint64value 5. A separator ofUtext must be astrorNone.- When the separator is longer than a
Ustring, the list holds the element itself, which NumPy prints asnp.str_('...').
np.@char.join(sep, seq) joins the CHARACTERS of each string of seq with its separator, as Python's sep.join(seq) does for a string:
np.@char.join("-", "osd"); // 'o-s-d'
np.@char.join(new[] { "-", "." }, new[] { "ghc", "osd" }); // ['g-h-c', 'o.s.d'], <U5
np.@char.join("-", np.@char.split(words)); // TypeError, as in NumPy 2.5.4 (see below)
The result has the class of seq: U or S as wide as the longest result, or StringDType(). An S sequence joins only when its strings are empty, because Python iterates bytes as numbers.
NumPy 2.5.4 cannot join the lists that split returns: it joins every list and then raises TypeError: currently only the no-argument instantiation is supported; use np.dtype instead. NumSharp raises the same error. Over an empty NDListArray NumPy returns an empty object array, which has no NDArray form: NumSharp raises NotSupportedException.
The errors are NumPy's, in NumPy's order. The first element that fails stops the call:
np.@char.split(words, ""); // ValueError: empty separator
np.@char.split(words, " ", 1.5); // TypeError: 'numpy.float64' object cannot be interpreted as an integer
np.@char.split(words, new byte[] { 32 }); // TypeError: must be str or None, not numpy.bytes_
np.@char.split(np.array(new long[] { 1 })); // TypeError: string operation on non-string array
In these four functions a C# null is Python's None: np.@char.split(null) raises NumPy's TypeError, and new object[] { " ", null } as a separator splits the second element at whitespace. The chararray members c.split(), c.rsplit(), c.splitlines() and c.join(seq) call the same functions on the chararray's elements with their trailing whitespace.
An NDListArray has the ndarray members of NumPy's object array, each with NumPy's parameters: the structural ones (T, reshape, ravel, copy, view, take, repeat, diagonal, squeeze, ...) and the ones that compare, sort, reduce, compute and assign lists with Python's list rules (==, <, sort, argsort, searchsorted, max, sum, cumsum, mean, var, clip, +, *, put, fill, ...).
w.sum(); // ['a', 'b', 'c', 'lead', 'trail']: the lists concatenated
w.cumsum(@out: np.@char.split(np.array(new[] { "x", "y", "z" }))); // into a list array of the same shape
w.reshape(new Shape(1, 3), 'C', copy: false); // a view, or NumPy's ValueError when none exists
w.mean(dtype: np.@bool, where: np.array(new[] { true, false, true })); // True: the masked truths over the count
A dtype= whose NumPy result is not a list raises NotSupportedException: w.sum(dtype: np.@bool) is NumPy's np.True_, which np.sum(w.astype(np.@bool), np.@bool) computes.
Differences from NumPy:
- An element access (
w[0],w[i, j]) returns a 0-d list array that stands for the list, where NumPy returns the Python list itself.w.item(i, j)returns the list as aList<object>. - Iterating a 1-D list array (
foreach (NDArray pieces in w)) yields the pieces of each element as a newUorSarray, where NumPy yields the lists.
Measured against NumPy 2.4.2 (pinned to two cores, 1,000 to 1,000,000 strings): 4.6 to 48 times as fast, 12.6 times on average.
Operations that do not accept text
String support does not give every numerical operation a string meaning. NumSharp follows NumPy's dtype-specific refusals:
np.sum(names); // TypeError: the resolved dtypes are not compatible with add.reduce. Resolved (dtype('<U4'), dtype('<U4'), dtype('<U8'))
np.sqrt(names); // TypeError: ufunc 'sqrt' not supported for the input types, ...
Those reductions differ for StringDType, which has concatenation and lexical-selection loops. The NumSharp expression engine, np.evaluate, computes on text as NumPy does: S, U and StringDType operands (a C# string is NumPy's str) take every node, anywhere in an expression, and each node runs as NumPy's own function — the six comparisons, the truth value (LogicalAnd / LogicalOr / LogicalXor, Where's condition, Any / All / CountNonzero), IsNaN, Cast to and from text, the np.strings functions as NDExpr.StrLen, NDExpr.IsAlpha … NDExpr.IsNumeric, NDExpr.Find / RFind / Count / Index / RIndex and NDExpr.StartsWith / EndsWith, concatenation (+), StringDType repetition (*), Maximum / Minimum over StringDType, Where with text branches (NumPy's promotion) and the reductions (ArgMax / ArgMin over every class, Sum / Max / Min over StringDType). Its result, text or numbers, feeds the rest of the expression in the same call, and the call's out=, where=, dtype= and order= apply to a text result too. Where NumPy has no loop (the square root of text, bytes plus str), np.evaluate raises NumPy's own error; only NDExpr.Call refuses text, because a delegate's parameters are numbers (StringComputationNotSupportedException):
var t = names.astype(np.dtypes.StringDType());
t == t; // [ True True True]
np.unique(t); // ['bird' 'cat' 'dog']
np.evaluate(NDExpr.Equal(names, "cat") & ((NDExpr)np.array(new[] { 4, 2, 7 }) > 3)); // [ True False False]
np.evaluate(NDExpr.CountNonzero(NDExpr.StrLen(names) > 3)); // 1
np.evaluate(NDExpr.Where(NDExpr.StartsWith(names, "b"), 1.0, 0.5)); // [0.5 0.5 1. ]
np.evaluate((NDExpr)names + names); // ['catcat' 'dogdog' 'birdbird']
np.evaluate(NDExpr.Where(NDExpr.StrLen(names) > 3, names, "-")); // ['-' '-' 'bird']
np.evaluate(NDExpr.ArgMax(names)); // 1
np.evaluate(NDExpr.Sqrt(names)); // TypeError: ufunc 'sqrt' not supported for the input types, ...
A comparison of text with numbers is "never equal" in NumPy, and in NumSharp: names == np.array(new[] { 1, 2, 3 }) is [False False False].
numpy.polynomial takes text operands the way NumPy does, up to the first statement that computes with the strings themselves. Its checks and the statements before that point raise or answer as in NumPy: as_series' common-type refusal, polyline's stacking (polyline(np.array("x"), 1) is ['x' '1']), the derivative's and integral's argument checks, the first statement of the *val family and the x + 0.0 of the Vandermonde and fit functions. NumPy then repeats and concatenates the Python strings (polyder(['ab', 'c', 'de']) is ['c' 'dede']) or raises CPython's TypeError from that arithmetic. NumSharp raises NotSupportedException there, because its polynomial package does not model Python's string arithmetic.
NumSharp also reports the problems of a call in NumPy's order, before it answers:
- The
dtype=request. - A read-only
out=, then a non-boolwhere=. - The axis.
- The operation's dtype error.
np.sum(names, 9); // AxisError: axis 9 is out of bounds for array of dimension 1
An unsized string dtype= request ("U", "S") gets NumPy's answer too. np.sqrt(x, dtype: "U") names no loop. np.add(x, x, dtype: "U") writes the numbers as text and concatenates them.
When the shape decides the answer
Some calls never read a character, because the shape or dtypes decide the result:
- An empty array. Truth tests, counts, index searches, sorts,
np.unique,np.isin, the set routines and the binary ufuncs return NumPy's empty or identity result. - Lines of one element. A sort, a partition or an arg-reduction over lines of one element compares nothing.
- A pair with no comparison loop. Text against numbers is never equal, so
np.isinandnp.array_equalreturnFalse.
var empty = np.zeros(new Shape(0L, 3L), "U2");
np.any(empty); // False
np.argmax(np.array(new[,] { { "b" }, { "a" } }), 1); // [0 0]
np.isin(names, np.array(new[] { 1.5, 2.0 })); // [False False False]
np.array_equal(names, np.zeros(new Shape(3L))); // False
np.place and np.putmask follow NumPy's two value forms. A C# array is a Python list: NumSharp converts it straight into the target's dtype. An NDArray must cast to the target's dtype under the 'safe' rule, as in NumPy:
var a = np.arange(5).astype(np.int32);
var mask = np.ones(new Shape(5L), np.@bool);
np.place(a, mask, new[] { 1.9, 2.9 }); // a list: [1 2 1 2 1]
np.place(a, mask, np.array(new[] { 1.9, 2.9 })); // InvalidCastException (NumPy's TypeError):
// Cannot cast array data from dtype('float64') to dtype('int32') according to the rule 'safe'
Masked text arrays
np.ma works on text arrays as numpy.ma does. The default fill value is 'N/A', stored at its own width (<U3 or |S3). filled() cuts it to the width of the array:
var m = np.ma.array(np.array(new[] { "ab", "c", "de" }), mask: new[] { true, false, false });
m.fill_value; // "N/A"
m.filled(); // ['N/' 'c' 'de']
m.filled("zz"); // ['zz' 'c' 'de']
np.ma.count(m); // 2
The rules for computing are the same as for plain arrays. A masked reduction refuses with NumPy's message. A number used as a fill value is written as its text:
np.ma.sum(m); // TypeError: the resolved dtypes are not compatible with add.reduce. Resolved (dtype('<U2'), dtype('<U2'), dtype('<U4'))
m.filled(5); // ['5' 'c' 'de']
For application-specific processing in .NET, NDArray.AsStringArray and tolist() provide managed values. For NumPy's text semantics, use np.strings or np.@char; their Unicode and output-width rules differ from a culture-dependent LINQ transformation.
Variable-width text: StringDType
NumPy 2's StringDType (T, NEP 55) has no fixed text width. Each element is a 16-byte cell: up to 15 UTF-8 bytes sit inside it, longer text lives in an arena owned by the buffer, and a longer rewrite may move the text to the heap. NumSharp follows NumPy's packed layout, missing-data and coercion rules. itemsize is 16 and nbytes counts cells only, not all text memory. tobytes() is not a portable serialization: cells can contain offsets or process-local pointers. Use np.save or the supported text readers and writers to persist values; never reconstruct a T array by copying those cells into another buffer.
StringDType uses the same np.strings operations as fixed-width text, with lengths and search positions measured in Unicode code points. It also supports comparisons, truth values, concatenation, repetition, lexical reductions, sorting, searching, unique values and set routines. Missing values have operation-specific behavior, described below.
Creating StringDType arrays
var t = np.array(new[] { "a", "a much longer text" }, np.dtype("T"));
t.dtype; // StringDType()
t.dtype.itemsize; // 16: the cell, whatever the length of the text
np.zeros(new Shape(2), np.dtypes.StringDType()); // ['', '']
np.ones(new Shape(2), np.dtypes.StringDType()); // ['1', '1']
Missing data: the NA object
np.dtypes.StringDType(na_object: …) names the object that stands for a missing element: null (NumPy's None), double.NaN, a string, or another object. Without an NA object, None is just the text 'None':
var withNA = np.dtypes.StringDType(na_object: null); // NumPy's StringDType(na_object=None)
var a = np.array(new object[] { "x", null }, withNA);
a.item(1); // null: the NA object
a.ToString(true); // array(['x', None], dtype=StringDType(na_object=None))
np.array(new object[] { "x", null }, np.dtype("T")).item(1); // "None"
Writing elements
A value written into a StringDType element is its str(). With coerce: false, only text is accepted:
var t = np.array(new[] { "a", "b" }, np.dtypes.StringDType());
t[0] = 3.25; // '3.25'
var strict = np.array(new[] { "a" }, np.dtypes.StringDType(coerce: false));
strict[0] = 5; // ValueError: StringDType only allows string data when string coercion is disabled.
Moving and casting
Views share the buffer and its arena. A copy, a gather (np.take, fancy indexing, a boolean mask) and a scatter (np.put, np.putmask, np.place, np.choose) duplicate every string into the destination's own buffer. The casts follow NumPy: text, bool and numbers cast to StringDType (an S text only when its bytes are valid UTF-8, otherwise NumPy's TypeError: Invalid UTF-8 bytes found, cannot convert to StringDType); StringDType casts to a sized U or S (the text is cut to the width) and to numbers (each text is parsed by Python's int(), float() or complex(); a missing element raises):
var t = np.array(new[] { "12", "-3" }, np.dtypes.StringDType());
t.astype(np.int64); // [12, -3]
t.astype("U1"); // ['1', '-']
np.array(new[] { 1.5, 2.0 }).astype(np.dtypes.StringDType()); // ['1.5', '2.0']
A copy moves a short text's cell as one block and reserves the destination's arena once for the longer texts. Measured against NumPy 2.4.2 on the same machine, copies, gathers (np.take, a boolean mask), np.concatenate, np.where, np.repeat, np.array from a list and the casts between U and StringDType run 1.8 to 31 times as fast as NumPy, for 1,000 to 1,000,000 elements.
To an embedded NumPy and back, a StringDType array crosses as an exact copy through NumPy's own C API, with its NA values, na_object and coerce; see Crossing to Python.
Iterating with np.nditer
np.nditer iterates a StringDType operand when you give it refs_ok, as NumPy requires. Buffering and casting work as they do for any other dtype: each buffer has its own string allocator, a buffer fill duplicates every string or runs the cast of the pair (StringDType to U, S, bool or numbers and back; an NA value follows the cast's NA rules), and a written operand gets its strings back when the iterator moves on or closes. it[i] = value stores like a[...] = value: null is the NA object, or 'None' without one.
var t = np.array(new[] { "a", "a long string over fifteen" }, np.dtypes.StringDType());
using (var it = np.nditer(t, new[] { "refs_ok", "buffered" }, op_dtypes: new DType[] { "U5" }, casting: "unsafe"))
foreach (var x in it)
Console.WriteLine(x[0].item()); // a, then a lon (cut to U5)
using (var it = np.nditer(t, new[] { "refs_ok", "buffered" }, new[] { "readwrite" }))
foreach (var x in it)
x[0][Slice.Ellipsis] = (string)x[0].item() + "!"; // t is ['a!', 'a long string over fifteen!']
Computing on StringDType text
The six comparisons, the truth values (astype(bool), np.any, np.nonzero, logical_and / logical_or / logical_xor, np.where), np.isnan, add, multiply, maximum, minimum and the reductions np.sum, np.cumsum, np.max, np.min, np.argmax, np.argmin and np.trace compute over StringDType text with NumPy's results, for every memory layout, with out=, where=, axis= and keepdims= where NumPy takes them. A U array or a C# string (NumPy's str) may meet a StringDType array in a comparison and in add; maximum and minimum take two StringDType arrays only, as in NumPy. Missing data follows the NA object:
- A NaN-like NA (
na_object: double.NaN) makes every comparison with itFalse,!=included. Concatenation and repetition propagate it. Lexical selection treats it as greater than every text:maximum/maxselect NA, whileminimum/minselect a text when one is available.argmaxreturns the last NaN-like NA, not the first. - A string NA (
na_object: "NA") computes as its text. - A
None-like NA (na_object: null) is equal to another NA and unequal to any text; an ordering or arithmetic with it raises NumPy'sValueError.
var nan = np.dtypes.StringDType(na_object: double.NaN);
var t = np.array(new object[] { "pear", "fig", double.NaN }, nan);
var u = np.array(new object[] { "kiwi", "apple", "plum" }, nan);
t < "grape"; // [False True False]
t != u; // [ True True False]: even != is False with the NA
t + "!"; // ['pear!' 'fig!' nan]
t * 2; // ['pearpear' 'figfig' nan]
np.maximum(t, u); // ['pear' 'fig' nan]
np.minimum(t, u); // ['kiwi' 'apple' 'plum']
np.sum(t["0:2"]).item(); // "pearfig"
np.cumsum(t); // ['pear' 'pearfig' nan]
np.argmax(t); // 2: the NA orders last
np.isnan(t); // [False False True]
np.maximum(t, "x"); // TypeError: ufunc 'maximum' did not contain a loop with signature matching types (<class 'numpy.dtypes.StringDType'>, <class 'numpy.dtypes.StrDType'>) -> None
Texts compare as NumPy's NpyString_cmp compares them: memcmp over the shorter length, then the sizes. The first byte that differs decides, a NUL included: two texts are equal only when their bytes are ('a\0c' != 'a\0b', and 'a\0b' < 'a\0c'), a NUL is smaller than any byte it meets ('a\0' < 'ab'), and a text orders before every longer text that starts with it. Fixed-width U text compares every code point too. np.array_equal and np.array_equiv are (a1 == a2).all() with these rules: a NaN-like NA is never equal unless equal_nan: true matches the NA positions (through np.isnan), and a pair that has no comparison loop (S, numbers) is NumPy's all-False array, so only an empty one is equal:
var t2 = np.dtypes.StringDType();
np.array(new object[] { "a\0c" }, t2) == np.array(new object[] { "a\0b" }, t2); // [False]
np.array_equal(np.array(new object[] { "a\0c", "x" }, t2), np.array(new object[] { "a\0b", "x" }, t2)); // False
np.array_equal(t, t); // False: the NA is not equal to itself
np.array_equal(t, t, equal_nan: true); // True
A result takes the operands' common NA object (two different NA objects have no loop) and the first operand's coerce. out= follows NumPy's descriptor resolution: a StringDType out receives the loop's result as it is, and an out of another dtype receives the result of the default StringDType() through NumPy's output cast, so an NA becomes '' in a U or S out:
np.add(t, "!", np.empty(new Shape(3), np.dtype("U8"))); // ['pear!' 'fig!' '']
Two short texts compare inside their 16-byte cells, a text in the arena is read only as far as its first difference, and a new result reserves its arena once at the exact size its texts take. Measured against NumPy 2.4.2 on the same machine (pinned to two cores, 1,000 and 100,000 elements, texts of 6 to 24 bytes: ASCII, non-ASCII, with a long common prefix, with missing elements): all 200 cells run 1.5 to 47 times as fast as NumPy, 3.3 times in the geometric mean. Comparisons run 1.6 to 4.5 times as fast, add / multiply 2.0 to 3.7, maximum / minimum 1.9 to 3.6, truth values and np.isnan 1.7 to 6.3, np.max / np.min 1.5 to 4.4, np.cumsum 1.8 to 5.9, np.argmax 4.9 to 15 and np.sum 2.7 to 47.
String operations with missing values
A missing value is not automatically an empty string. Its behavior depends on the operation and the descriptor's na_object:
- A string sentinel is processed as that string.
- A NaN-like sentinel propagates through transformations such as stripping, padding and replacement. Character predicates return
Falsefor it. Length and substring-search functions raise because the missing value has no text to measure or search. - A None-like sentinel is refused by operations that require text, with NumPy's operation-specific error.
- Case conversion, formatting, translation and codec functions use the scalar object that an element represents. A non-string missing object can therefore raise a
TypeErrorinstead of propagating; for example,uppercannot apply a string method to a NaN orNone.
Choose the sentinel for the operations you need, or replace missing values explicitly before transforming text.
Ordering StringDType text
np.sort, np.argsort, ndarray.sort, np.partition, np.argpartition, np.lexsort, np.searchsorted, np.unique and the Array-API unique family, np.isin, the set routines, np.median, np.quantile, np.percentile, their nan forms and numpy.ma's sort / argsort / median / unique order StringDType text with NumPy's results, for every memory layout and axis. Texts compare by their UTF-8 bytes, a text before every longer text that starts with it (NumPy's NpyString_cmp, see Computing on StringDType text), so equal texts are identical texts and every route of np.unique keeps one entry per text. Missing data follows the NA object, as NumPy's sort compares it:
- A NaN-like NA sorts after every text. The stable kinds preserve the NAs' input order; the default introsort gives NumPy's algorithm-dependent permutation.
- A string NA reads as its text (the descriptor's default string), and
np.uniquekeeps it and that text as one entry. - A
None-like NA raises NumPy'sValueErrorat the first comparison that meets it, so in any line of two or more elements. A line of one element compares nothing. np.uniquekeeps one entry for all the NAs of a descriptor whose NA object is not NaN-like, whateverequal_nansays; NaN-like NAs are one entry underequal_nan: true(the default) and one each without it.
var nan = np.dtypes.StringDType(na_object: double.NaN);
var t = np.array(new object[] { "pear", double.NaN, "fig", "apple", "fig" }, nan);
np.sort(t); // ['apple' 'fig' 'fig' 'pear' nan]
np.argsort(t, kind: "stable"); // [3 2 4 0 1]
np.unique(t); // ['apple' 'fig' 'pear' nan]
np.searchsorted(np.sort(t), np.array(new object[] { "grape" }, nan)); // [3]
np.isin(t, np.array(new object[] { "fig" }, nan)); // [False False True False True]
var none = np.dtypes.StringDType(na_object: null);
np.sort(np.array(new object[] { "b", null }, none)); // ValueError: Cannot compare null that is not a nan-like value
np.sort(t, descending: true); // RuntimeError: unsupported sort kind
NumPy's default kind is an introsort, and NumSharp reproduces its order of equal texts, as for S and U. kind: "heapsort" sorts as the default kind (NumPy clears the heapsort bit before it looks the sort up). descending: true raises NumPy's RuntimeError, because StringDType registers no descending sort; S and U text sorts descending.
The order statistics compute what NumPy computes, and refuse where NumPy refuses:
np.mediansorts, then takes the mean of the middle elements. Their sum is text, whose division NumPy refuses with aTypeError, so a median exists only when the middle elements are NaN-like NAs (NaN).np.quantileandnp.percentilepick an element with the selection methods, as forSandU. An interpolating method reads the two neighbours and subtracts them, which raises Python'sTypeError(unsupported operand type(s) for -: 'str' and 'str').- The nan forms drop the NaN-like NAs first. An all-NaN slice gives the NA itself.
var s = np.array(new[] { "b", "a", "c" }, np.dtypes.StringDType());
np.quantile(s, 0.5, method: "lower"); // 'b'
np.quantile(s, 0.5); // TypeError: unsupported operand type(s) for -: 'str' and 'str'
np.median(s); // TypeError: ufunc 'divide' not supported for the input types, ...
np.unique_values (and np.unique with sorted: false and no return_* flag) hashes the texts in NumPy and returns them in its hash table's order; NumSharp returns them sorted, the same values. An axis over two or more dimensions raises NumPy's TypeError, because its structured path cannot hold StringDType fields.
Measured against NumPy 2.4.2 on the same machine (pinned to two cores, fresh inputs every call, 1,000 and 100,000 texts of five kinds: random 8-letter words, 8 repeated words, 24-byte URLs that live in the arena, 6-character non-Latin words, and words with missing elements): all 168 calls measured ran 1.8 to 32 times as fast as NumPy, 7.8 times in the geometric mean. The sorts and partitions ran 2.2 to 32 times as fast, np.lexsort 6.5 to 22, np.searchsorted 3.4 to 5.8, np.unique 2.2 to 13, np.unique_values 1.8 to 4.6, np.isin 9.6 to 24, the set routines 2.9 to 12 and the quantiles 9 to 21. NumPy's sorts call a comparison function for every pair they compare; NumSharp sorts 8-byte prefix keys and replays NumPy's order of equal texts only where it can differ. np.unique hashes the texts when they repeat and sorts them when they are mostly distinct, judged from the repeats among its first texts (an eighth of the array, 256 to 4,096). (np.intersect1d was measured with assume_unique: true: on these inputs NumPy's default path crashes in the sort of the joined unique values, the heapsort-fallback bug listed under "What StringDType refuses".)
Text files: loadtxt, savetxt and tofile
np.loadtxt reads every field of a StringDType column whole, whatever its length. A field equal to a string NA object is a missing element, and a converter's result is stored as a[...] = value stores it. So with na_object: null, a converter that returns null makes the element missing:
var na = np.dtypes.StringDType(na_object: null);
var t = np.loadtxt(new[] { "a b", "- c" }, na, converters: (Func<string, object>)(s => s == "-" ? null : s));
t.item(1, 0); // null: the NA object
np.loadtxt(new[] { "a b" }, np.dtypes.StringDType(), delimiter: ",").item(); // "a b": the whole line, spaces kept
np.savetxt formats each element as the Python object NumPy's getitem returns: a str, or the descriptor's NA object for a missing element. %s of a missing element is therefore str(na_object), and a numeric conversion takes the NA object's own type:
var w = new StringWriter();
np.savetxt(w, np.array(new object[] { "x", null }, na), "%s");
w.ToString(); // "x\nNone\n"
var nanNA = np.dtypes.StringDType(na_object: double.NaN);
np.savetxt(new StringWriter(), np.array(new object[] { double.NaN }, nanNA), "%d");
// ValueError: cannot convert float NaN to integer
np.savetxt(new StringWriter(), np.array(new[] { "x" }, na), "%d");
// TypeError: Mismatch between array dtype ('StringDType(na_object=None)') and format specifier ('%d')
ndarray.tofile(path, sep: ",") writes each element's str(), or format % element. Its binary mode (sep: ""), np.fromfile and np.fromstring refuse a StringDType, as NumPy does:
t.tofile(path, sep: ","); // a,b,None,c
np.fromfile(path, np.dtypes.StringDType(), sep: ","); // ValueError: Cannot read into object array
All three follow NumPy's text encodings (see Reading and writing text files): a file name is read and written as UTF-8 by default, a byte Stream as Latin-1, and encoding: accepts every name Python's codecs.lookup accepts for UTF-8, UTF-16, UTF-32, Latin-1, ASCII and the single-byte code pages.
Measured against NumPy 2.4.2 on the same machine (pinned to two cores, 1,000 to 1,000,000 rows of two StringDType columns): np.loadtxt from a list of lines or a file runs 2.9 to 3.8 times as fast as NumPy, np.savetxt to a writer or a byte stream 5.3 to 8.1 times, and np.save / np.load of the result 2.5 to 7.7 times.
What StringDType refuses
| Call | NumSharp |
|---|---|
view(another dtype), memoryview (.data), np.frombuffer(..., "T") |
NumPy's errors: a cell is not raw text |
view(dtype) / getfield(dtype) with an equal but separate StringDType() instance |
NumPy's TypeError (Cannot change data-type for array of references., Cannot get/set field of an object array): the cells point into the array's own allocator, so only the array's own dtype object views them |
ndarray.byteswap(), in place or not |
NumPy's TypeError: dtype StringDType() does not implement copyswap: a cell has no byte order (the descriptor's byteorder is '|') |
np.ndarray(shape, "T", buffer: ...) over raw bytes or a non-StringDType array, another dtype over a StringDType array's cells, or an offset that splits a cell |
NotSupportedException. NumPy reads the bytes as cells; a cell holds an arena offset or a heap pointer, so NumSharp refuses. Over another StringDType array the call is a view that shares its cells (offsets and strides in 16-byte cells), and without a buffer the array starts with empty strings |
np.fromfile, np.fromstring and tofile's binary mode with StringDType |
NumPy's errors (Cannot read into object array, OSError for the binary write) |
np.unique(..., axis: k) over two or more dimensions |
NumPy's TypeError: its structured path cannot hold StringDType fields |
A np.nanquantile / np.nanmedian row whose result is a None NA |
NotSupportedException. NumPy builds an object array, a dtype NumSharp does not have |
NumPy 2.5.4 has bugs on StringDType that NumSharp does not reproduce; NumSharp gives the correct answer:
np.lexsortwith aStringDTypekey that is not contiguous along the axis (a transposed 2-D key, a stepped or reversed view) crashes the process (an access violation).np.argsort(view, axis=None)of aStringDTypeview that is not C-contiguous (broadcast, reversed, transposed, stepped) misorders texts that live in the arena (a 17-byte text lands before its 16-byte prefix). The same view copied,np.sort, an explicit axis, the partitions andndarray.sortsort right.- The default-kind
np.sort/ndarray.sortnever returns when its introsort spends its depth budget: the heapsort fallback reads its method context as an array (NumPy 2.4.2 crashed the process there;np.argsortof the same input answers). NumSharp's heapsort answers, in the ordernp.argpartitiongives. - A short string copied onto itself (
a[...] = a[:1]) becomes''. np.loadtxtkeeps theStringDTypeinstance it is given as its result's descriptor, so a second call with the same instance returns an array NumPy cannot read (MemoryError: Failed to load string in StringDType getitem). NumSharp gives every result its own allocator.np.maximum/np.minimumwith aNone-like NA into anSout whose texts are not all ASCII raiseSystemError: <class 'UnicodeEncodeError'> returned a result with an exception set: the output cast runs after the loop has already failed. NumSharp raises the loop'sValueError: Cannot compare null that is not a nan-like value.
.npy files
np.save and np.load read and write U and S arrays byte for byte, as NumPy writes them. A big-endian >U3 file is byte-swapped when it is read.
Breaking change: a
<U1file now loads as a<U1string array, not asChar. AChararray is still written as<U1, because NumPy has no 2-byte character dtype.
var stream = new MemoryStream();
np.save(stream, names);
stream.Position = 0;
np.load_npy(stream); // array(['cat', 'dog', 'bird'], dtype='<U4')
A StringDType array has no fixed-size bytes to write, so NumPy writes a pickle: the header names |O, and the data is pickle.dump(array, protocol=4). np.save writes the same bytes as numpy.save, and np.load(allow_pickle: true) reads them, NA values, na_object and coerce included. As in NumPy, loading without allow_pickle refuses. .npz archives hold StringDType members the same way.
var words = np.array(new object[] { "cat", null }, np.dtypes.StringDType(na_object: null));
var stream = new MemoryStream();
np.save(stream, words); // the bytes numpy.save writes
stream.Position = 0;
np.load_npy(stream, allow_pickle: true); // array(['cat', None], dtype=StringDType(na_object=None))
stream.Position = 0;
np.load_npy(stream); // FormatException: Object arrays cannot be loaded when allow_pickle=False
NumSharp reads only the pickle NumPy writes for a StringDType array. It has no general unpickler: a pickle of anything else (an object array, a Python dict, a call to a Python function) is refused, and nothing in the file runs. np.save(..., allow_pickle: false) raises NumPy's ValueError: Object arrays cannot be saved when allow_pickle=False, and mmap_mode raises ValueError: Array can't be memory-mapped: Python objects in dtype.
Crossing to Python
With NumSharp.Interop.pythonnet, text crosses to an embedded NumPy the way numbers do, and a StringDType array as an exact copy:
| Array | NumSharp → NumPy | NumPy → NumSharp |
|---|---|---|
S<n>, U<n> |
ToNumpy() shares the bytes (\|S<n>, <U<n>) |
AsNDArray() shares numpy's bytes |
StringDType |
ToNumpyCopy() copies, NA values and descriptor included |
ToNDArray() copies, NA values and descriptor included |
NumPy object array of str |
- | ToNDArray() copies as StringDType() |
var words = np.array(new object[] { "cat", null }, np.dtypes.StringDType(na_object: null));
using (Py.GIL())
{
using PyObject u = names.ToNumpy(); // array(['cat', 'dog', 'bird'], dtype='<U4'), the same memory
using PyObject t = words.ToNumpyCopy(); // array(['cat', None], dtype=StringDType(na_object=None))
}
A StringDType array has no shared form: NumPy keeps each element's text in an allocator its descriptor owns, so the bridge copies it element by element through NumPy's own NpyString C API. The missing-data object crosses when C# can hold it (None, bool, int, float, complex, str, bytes, np.float16); any other object, such as pd.NA, is refused.
Breaking change: NumPy's
<U1(and a 4-bytewchar_tbuffer) now imports as aU1view, not as aCharcopy. AU1element is a 4-byte code point and holds an astral character that acharcannot.
The Char dtype
NPTypeCode.Char wraps .NET's System.Char, a 2-byte UTF-16 code unit. It stores characters as numbers, not text:
var letters = np.array(new[] { 'a', 'b', 'c' });
letters.typecode; // NPTypeCode.Char
InfoOf<char>.Size; // 2 (actual memory footprint)
Char is not NumPy's <U1 or S1. NumPy's <U1 is a 4-byte code point, and S1 is one byte. NumSharp's Char is a 2-byte UTF-16 unit that behaves like a 2-byte integer in arithmetic. For text, use a U array. The descriptor reports itemsize == 2 and kind == 'u'. See Data types → Char.
Byte streams → byte[] + np.frombuffer
For raw bytes, use byte[] and reinterpret it with np.frombuffer when you want to view the bytes as numbers:
byte[] raw = File.ReadAllBytes("data.bin");
var asBytes = np.frombuffer(raw, np.uint8); // NDArray<byte>, zero-copy view
var asFloats = np.frombuffer(raw, np.float32); // reinterpret the same bytes as float32
This is NumPy's np.frombuffer on a bytestring, and the general interop bridge. See Any library via np.frombuffer.
Porting cheat-sheet
| NumPy | NumSharp |
|---|---|
np.array(["hello", "world"]) (→ <U5) |
np.array(new[] { "hello", "world" }) (→ <U5) |
np.array([b"hi"], dtype="S2") |
np.array(new object[] { Encoding.ASCII.GetBytes("hi") }, np.dtype("S2")) |
arr.astype("U10") |
arr.astype("U10") |
np.arange(3).astype(str) (→ <U21) |
np.arange(3).astype("U") (→ <U21) |
np.array(["1", "2"]).astype(float) |
np.array(new[] { "1", "2" }).astype(np.float64) |
arr[0] (a numpy.str_) |
arr.item(0) (a C# string) |
arr.tolist() |
arr.tolist() or NDArray.AsStringArray(arr) |
a + b / np.char.add(a, b) / np.strings.add(a, b) |
a + b / np.@char.add(a, b) / np.strings.add(a, b) |
np.char.multiply(a, 3) / np.strings.multiply(a, 3) |
np.@char.multiply(a, 3) / np.strings.multiply(a, 3) |
np.char.equal(a, b) (strips trailing whitespace) |
np.@char.equal(a, b) |
np.char.compare_chararrays(a, b, '<', True) |
np.@char.compare_chararrays(a, b, "<", true) |
c = np.char.array(["a ", "b"]); c[0] (the stripped str) |
var c = np.@char.array(new[] { "a ", "b" }); c[0].item() |
a.view(np.char.chararray) |
a.view(typeof(NDCharArray)) |
c == "ab", u == c (a chararray strips on either side) |
the same, with c of type NDCharArray (in an NDArray variable, == does not strip) |
c.argmax(out=o) |
c.argmax(null, o) (without out, ValueError as in NumPy) |
np.char.upper(arr) / np.strings.upper(arr) |
np.@char.upper(arr) / np.strings.upper(arr) |
np.strings.find(arr, "ab") / np.strings.strip(arr) |
the same calls |
np.strings.encode(arr, "utf-8") |
np.strings.encode(arr, "utf-8") |
np.strings.partition(arr, "-") |
three result arrays; np.@char.partition(arr, "-") stacks them on the last axis |
np.array(["a", "bb"], dtype=StringDType()) |
np.array(new[] { "a", "bb" }, np.dtypes.StringDType()) |
StringDType(na_object=None) |
np.dtypes.StringDType(na_object: null) |
np.array(['a','b','c']) (single characters, <U1) |
np.array(new[] { "a", "b", "c" }) (<U1); np.array(new[] { 'a', 'b', 'c' }) is Char |
Troubleshooting
"Why does np.evaluate raise a TypeError on my string array?"
np.evaluate computes on text exactly where NumPy does, so it raises exactly where NumPy raises, with NumPy's text: a node NumPy has no loop for (NDExpr.Sqrt(names), NDExpr.Subtract(names, names), NDExpr.LogicalNot(names)), operands NumPy will not mix (bytes with str, text with a number in NDExpr.Add, a Python number beside fixed-width text in NDExpr.Where), or a reduction NumPy refuses for that class (NDExpr.Sum of U text, a flat NDExpr.Sum of StringDType over several axes). The error comes when the call runs, in the order NumPy's statements would raise it. Only an NDExpr.Call delegate refuses text in its own words (StringComputationNotSupportedException): its parameters are numbers, so measure or compare the text in the expression (NDExpr.StrLen, NDExpr.Equal, ...) and pass that number in.
"ValueError: could not convert string to float"
A cast from text to a number parses every element, as NumPy does. Clean the text first, or keep it as text.
"My NDArray x = "1.5" is text, not a number"
A C# string is NumPy's str: a 0-d U array. Write the number itself (NDArray x = 1.5), or parse the text with astype(np.float64).
"My byte[][] became a uint8 matrix"
A jagged byte[][] is numeric data in C#. Pass an object[] of byte[] values to get an S array.
"My value was cut"
An S / U array has a fixed width. Writing a longer value cuts it, as NumPy does. Use a wider descriptor (arr.astype("U10")) or convert to np.dtypes.StringDType() when future writes must grow.
"UnicodeEncodeError when I cast to S"
A U to S cast encodes ASCII only, as NumPy does. Keep non-ASCII text in a U array.
"GetString threw InvalidOperationException"
GetString reads Char vectors. Read a string element with item(i) or NDArray.AsStringArray.
API reference
| Item | Behavior |
|---|---|
np.dtype("U5") / np.dtype("S5") |
The fixed-width descriptors: itemsize 20 / 5, kind 'U' / 'S' |
np.array(string[]) |
A U array as wide as the longest value |
np.array(object[] of byte[]) |
An S array as wide as the longest value |
NDArray.Scalar(string) / NDArray.Scalar(byte[]) |
A 0-d U / S array |
arr.item(i) |
A C# string (U) or byte[] (S), trailing NULs removed |
NDArray.AsStringArray(arr) |
Every element of a U array as a string[], in C order |
arr[i] = "text" |
Writes one element, cut to the width |
arr.astype("S3") / arr.astype("U8") |
Casts between string dtypes, as NumPy does |
arr.astype("U") of numbers |
Each number as its scalar's str(), at NumPy's width for the number type (21 for int64, 32 for float64) |
arr.astype(np.float64) of text |
Each text parsed by float() (int() / complex() for those targets), with NumPy's errors |
arr[i] = 5 / arr[i] = "12" |
A number stored as text / a C# string parsed into a numeric array |
Comparisons (==, <, np.equal, ...) and truth values (astype(bool), np.any, np.nonzero, ...) |
NumPy's results |
np.add(a, b) / a + b / np.strings.add(a, b) over two text arrays of one class |
The concatenation, as wide as the two widths together |
np.strings.multiply(a, i) / np.multiply(a, i, out) |
The repetition, as wide as the longest one / cut to the width of out |
np.strings.str_len, find, rfind, index, rindex, count, startswith, endswith |
Elementwise lengths, substring positions, counts and prefix/suffix tests |
np.strings.strip, lstrip, rstrip, replace, expandtabs, center, ljust, rjust, zfill, slice |
Elementwise text transformations with each function's output-width rule |
np.strings.partition / rpartition |
Three arrays containing the before / separator / after parts |
np.strings.isalpha, isalnum, isdigit, isdecimal, isnumeric, isspace, islower, isupper, istitle |
Boolean character-property tests; isdecimal / isnumeric require Unicode text |
np.strings.upper, lower, swapcase, capitalize, title, translate |
Unicode/ASCII transformations; fixed-width results keep their input dtype |
np.strings.mod, encode, decode |
Python-style formatting and codec conversion, sized to the resulting text |
np.@char string functions and NDCharArray string methods |
The legacy interface to the same text operations, with chararray return rules and stacked partition results |
np.sort / np.argsort / arr.sort() / np.partition / np.argpartition |
NumPy's results, NumPy's order of equal strings for every kind, NumPy's argument errors |
np.@char.equal … np.@char.greater_equal, np.@char.compare_chararrays |
Comparisons after the trailing NULs and ASCII whitespace are removed; null (NumPy's NotImplemented) for S against U |
np.@char.add / np.@char.multiply |
np.add / np.strings.multiply; a count that is not an integer is ValueError("Can only multiply by integers") |
np.@char.chararray / np.@char.array / np.@char.asarray |
NumPy's chararray (NDCharArray): element access and iteration remove trailing whitespace, views and copies stay chararrays, text data only; the comparison operators strip (also with a plain array on the left), + and * return chararrays, and the np.* functions keep, wrap or drop the class as NumPy's do |
np.@char.split / rsplit / splitlines |
Python's str / bytes methods per element; the result is an NDListArray, NumPy's object array of lists |
np.@char.join |
The characters of each string joined by its separator; the result has the class of seq; over an NDListArray, NumPy 2.5.4's TypeError (over an empty one, NotSupportedException: NumPy's empty object array) |
np.argmax / np.argmin / np.searchsorted / np.digitize |
NumPy's results: the first extreme element, NumPy's exact search over unsorted haystacks |
np.unique / np.unique_values / np.unique_counts / np.unique_inverse / np.unique_all |
NumPy's results for every flag and axis; unique_values (and sorted: false) sorted where NumPy keeps its hash order |
np.isin / np.intersect1d / np.union1d / np.setdiff1d / np.setxor1d / np.lexsort / np.sort_complex |
NumPy's results: numbers beside text written as text, bytes decoded as ASCII where NumPy joins the operands |
np.histogram / np.histogram_bin_edges |
NumPy's results with text or number edges (compared as text), weights and density |
np.quantile / np.percentile |
NumPy's results for the methods that pick an element; the interpolating methods raise NumPy's TypeError |
Operations without a text loop (np.sqrt; np.sum of S / U) |
NumPy's exception and message |
np.dtypes.StringDType(...) / np.dtype("T") |
NumPy's variable-width StringDType: storage, element access, copies, gathers, scatters, casts and printing as NumPy's; comparisons, truth values, add / multiply / maximum / minimum, the reductions, the sorts, searches, unique values, set routines and order statistics compute with NumPy's results and its ordering of missing values |
np.dtype("V7") / "O" |
NotSupportedException |
NPTypeCode.Char |
A 2-byte UTF-16 code unit dtype, written to .npy as <U1 |
np.frombuffer(byte[], dtype) |
Reinterprets raw bytes as an array, string dtypes included |
Related reading
- Data types - the full dtype set and the
Charquirks. - Structured arrays - a NumPy dtype family NumSharp does not implement.
- Any library via np.frombuffer - the byte-buffer bridge.
- Python & numpy - text arrays crossing to an embedded NumPy.
- NumPy strings-and-bytes guide - the upstream article.