Getting started
Gym.NET's API is Gymnasium's, spelled in C#: gymnasium.make("CartPole-v1") is Gymnasium.Make("CartPole-v1"),
env.reset(seed=42) is env.Reset(seed: 42), and an env built with the same seed and fed the same actions returns
the same numbers as Gymnasium's. Every sample on this page compiles and runs as shown, warning-free. The built-in ids
are also constants, so the compiler checks them and the IDE lists them: Gymnasium.Make(Gymnasium.Envs.CartPoleV1)
(DR-88).
Install
dotnet add package Gym.NET.Environments # the environments and Gym.NET's core
dotnet add package Gym.NET.Rendering.Avalonia # only for render_mode "human"
The package registers its environment ids before your Main runs (a C# module initializer the package adds to your
project), so Gymnasium.Make works with no set-up call. F# and Visual Basic projects call
Gym.Environments.GymEnvironments.Register() once instead.
Gym.NET needs the JIT: NumSharp compiles its array kernels at run time, so Native AOT and trimmed publishing are not
supported. A .NET 10 file-based app (dotnet run app.cs) publishes AOT by default, and its first env throws
PlatformNotSupportedException until the app opts out with #:property PublishAot=false.
Make an env and run an episode
using System;
using Gym;
using Gym.Envs;
using NumSharp;
// make builds the registered env inside Gymnasium's wrappers: TimeLimit(500), OrderEnforcing, PassiveEnvChecker.
using Env<NDArray, NDArray> env = Gymnasium.Make<NDArray, NDArray>("CartPole-v1");
env.ActionSpace.Seed(42);
NDArray observation = env.Reset(seed: 42).Observation;
double episodeReturn = 0;
while (true) {
using NDArray action = env.ActionSpace.Sample(); // 0 (push left) or 1 (push right)
var (next, reward, terminated, truncated, info) = env.Step(action);
observation.Dispose(); // the arrays an env returns belong to the caller
observation = next;
episodeReturn += reward;
if (terminated || truncated)
break;
}
observation.Dispose();
Console.WriteLine($"Episode return: {episodeReturn}");
The later samples also use System.Collections.Generic, Gym.Observations, Gym.Spaces, Gym.Wrappers, Gym.Vector,
Gym.Environments.Envs.Classic and Gym.Rendering.Avalonia.
Reset(seed, options)returns aResetResult<TObs>,(Observation, Info).Step(action)returns aStepResult<TObs>,(Observation, Reward, Terminated, Truncated, Info).Terminatedmeans the task reached a terminal state (the pole fell);Truncatedmeans something outside the task cut the episode short (TimeLimit's step limit). Reset when either is set.- Rewards are
doubleeverywhere, also where Gymnasium returns anintor a float32. Infois a Python dict: insertion-ordered, keyed by strings, printed as Python prints it.info.Get<double>("x")reads a value as a .NET type (a NumPy scalar arrives as a 0-d array, which a plain cast refuses),info.PyEquals(other)is Python's==, and==compares references, as for any .NET collection.- The built-in envs that send info keys have a typed record of their info, in their env class's namespace
(
Gym.Environments.Envs.MuJoCofor HalfCheetah):HalfCheetahInfoV5 info = step.Info;theninfo.XPosition(adouble),info.XVelocity(adouble?,nullon a reset's info, which lacks it); the record keeps the dict's own values behind its indexer,info["reward_ctrl"]. CartPole and the other envs whose info is always{}have none.
Releasing what an env returns
The arrays an env returns belong to the caller, and they hold native memory. The way to release them is NumSharp's
[NDScoped]: every array a scoped method builds, or receives from an env, and does not return is released when the
method returns. NumSharp.Build, a build-time package, weaves the attribute into your assembly:
<PackageReference Include="NumSharp.Build" Version="0.81.0" PrivateAssets="all" />
using Env<NDArray, NDArray> env = Gymnasium.Make<NDArray, NDArray>("CartPole-v1"); // built outside the scoped method
for (int episode = 0; episode < 3; episode++)
Console.WriteLine(RunEpisode(env, seed: episode));
[NDScoped] // every array this call builds and does not return is released when it returns
static double RunEpisode(Env<NDArray, NDArray> env, int seed) {
env.Reset(seed: seed);
double episodeReturn = 0;
while (true) {
var (_, reward, terminated, truncated, _) = env.Step(env.ActionSpace.Sample());
episodeReturn += reward;
if (terminated || truncated)
return episodeReturn;
}
}
Two rules come with it:
- Build envs and spaces outside the scoped method, or dispose them inside it. An env built inside the scope and kept
after it would find its arrays released when the scope closes; NumSharp.Build refuses an
[NDScoped]method that returns one (NDW003), but cannot see one stored in a field. - Scope an episode, not a training run: a scope releases when its method returns, so a scoped method running a million
steps holds every step's arrays until it ends. An
asyncmethod takes[NDScopedAsync].
Every built-in env, wrapper and vector env keeps working when you step it inside a scope: what they keep from one step to the next is detached from your scope (DR-96).
using is the alternative, one result at a time. Both results are IDisposable: disposing one releases its observation
and the arrays in its info:
using Env<NDArray, NDArray> env = Gymnasium.Make<NDArray, NDArray>("CartPole-v1");
using ResetResult<NDArray> reset = env.Reset(seed: 0);
using NDArray action = env.ActionSpace.Sample();
using StepResult<NDArray> step = env.Step(action);
Console.WriteLine($"{step.Observation.Shape} {step.Reward}"); // (4) 1
A copy of a result shares its arrays: dispose one of them, after the last use, and .copy() an observation you keep
longer. The vector results (VectorStepResult, VectorResetResult) work the same way, and so does [NDScoped].
Typed and untyped
Gymnasium.Make<TObs, TAct> returns an Env<TObs, TAct> and raises TypeError when the env has other types.
Gymnasium.Make returns an IEnv, whose observations are objects and whose Step(object) converts the action:
using IEnv env = Gymnasium.Make("CartPole-v1"); // untyped: observations are objects
var (observation, info) = env.Reset(seed: 0);
var step = env.Step(1); // the env converts the action (here to a 0-d int64 array)
Console.WriteLine($"{observation} {step.Reward}");
(observation as IDisposable)?.Dispose();
(step.Observation as IDisposable)?.Dispose();
| Ids | Env | Observation | Action |
|---|---|---|---|
| CartPole-v0, CartPole-v1 | CartPoleEnv |
NDArray float32 (4,) |
NDArray 0-d int64, Discrete(2) |
| MountainCar-v0 | MountainCarEnv |
NDArray float32 (2,) |
NDArray 0-d int64, Discrete(3) |
| MountainCarContinuous-v0 | MountainCarEnv(continuous: true) |
NDArray float32 (2,) |
NDArray float32 (1,), Box(-1, 1) |
| Pendulum-v1 | PendulumEnv |
NDArray float32 (3,) |
NDArray float32 (1,), Box(-2, 2) |
| Acrobot-v1 | AcrobotEnv |
NDArray float32 (6,) |
NDArray 0-d int64, Discrete(3) |
| FrozenLake-v1, FrozenLake8x8-v1 | FrozenLakeEnv |
NDArray 0-d int64 (the cell) |
NDArray 0-d int64, Discrete(4) |
| CliffWalking-v1, CliffWalkingSlippery-v1 | CliffWalkingEnv |
NDArray 0-d int64 |
NDArray 0-d int64, Discrete(4) |
| Taxi-v4 | TaxiEnv |
NDArray 0-d int64 (the encoded state) |
NDArray 0-d int64, Discrete(6) |
| Blackjack-v1 | BlackjackEnv |
object?[], Python's tuple (player sum, dealer card, usable ace); BlackjackObservation reads it by name |
NDArray 0-d int64, Discrete(2) |
A discrete action or observation is a 0-d int64 NDArray, the shape NumPy gives Discrete.sample(). Gymnasium.PprintRegistry()
prints every id; Gymnasium.Spec(id) returns its EnvSpec:
Gymnasium.PprintRegistry(); // every registered id, grouped by namespace
var spec = Gymnasium.Spec("CartPole-v1");
Console.WriteLine($"{spec.MaxEpisodeSteps} {spec.RewardThreshold}"); // 500 475
Constructor arguments
make's keyword arguments are the env's constructor arguments by their Python names. Pass them as an anonymous object,
a dictionary, or the options record generated for the env class:
using IEnv a = Gymnasium.Make("CartPole-v1", new { render_mode = "rgb_array", sutton_barto_reward = true });
using IEnv b = Gymnasium.Make("CartPole-v1", new CartPoleEnvOptions { RenderMode = "rgb_array", SuttonBartoReward = true });
using IEnv c = Gymnasium.Make("CartPole-v1", new Dictionary<string, object?> { ["render_mode"] = "rgb_array" });
using IEnv d = Gymnasium.Make("CartPole-v1", maxEpisodeSteps: 1000); // instead of the spec's 500
Console.WriteLine(a); // <TimeLimit<OrderEnforcing<PassiveEnvChecker<CartPoleEnv<CartPole-v1>>>>>
CartPoleEnvOptions lives next to CartPoleEnv in Gym.Environments.Envs.Classic; it passes only the properties you
set, so the registered defaults still apply. An unknown name raises Gymnasium's TypeError. Building an env class
directly works too (new CartPoleEnv(renderMode: "rgb_array")), without make's wrappers and spec.
Seeding
Reset(seed: n) seeds the env's generator, NpRandom, with Gymnasium's Generator(PCG64(SeedSequence(n))), the
generator NumPy builds for np.random.default_rng(n); NumSharp implements it bit for bit. So Reset(seed: 42) in
Gym.NET and env.reset(seed=42) in Gymnasium draw the same starting state. Reset() without a seed continues the
current stream. Spaces have their own generators: env.ActionSpace.Seed(42) makes Sample() repeat
env.action_space.seed(42) and sample(). Gym.Utils.Seeding.NpRandom(seed) returns such a generator, with its seed,
for your own draws (gymnasium.utils.seeding.np_random).
Spaces
var box = new Box(-1.0, 1.0, new Shape(3), dtype: np.float32, seed: 0);
using NDArray point = box.Sample(); // float32 (3,), the values NumPy's generator draws for seed 0
bool inside = box.Contains(point); // true
var discrete = new Discrete(4, seed: 0);
using NDArray choice = discrete.Sample(); // a 0-d int64 array
var tuple = new Gym.Spaces.Tuple(new Space[] { discrete, box }, seed: 0);
object?[] sample = tuple.Sample(); // Python's tuple: object?[] { NDArray, NDArray }
Console.WriteLine($"{box} {inside} {discrete} {tuple}"); // Box(-1.0, 1.0, (3,), float32) True Discrete(4) Tuple(Discrete(4), Box(...))
All ten of Gymnasium's spaces are in Gym.Spaces: Box, Discrete, MultiBinary, MultiDiscrete, Tuple (samples
are object?[]), Dict (samples are Info), Text (strings), Sequence, Graph (GraphInstance) and OneOf.
SpaceUtils holds flatdim, flatten_space, flatten and unflatten. Spaces own unmanaged memory (their bounds and
generators): dispose the ones you build yourself; an env disposes its own.
The static class Spaces is the gymnasium.spaces module: with using Gym.Spaces;, Spaces.Discrete(4, seed: 0)
reads as spaces.Discrete(4, seed=0) and builds what new Discrete(4, seed: 0) builds, and Spaces.Flatdim,
Flatten, FlattenSpace and Unflatten are the module's functions (DR-79, DR-86). Inside a Gym.* namespace the
simple name Spaces is the namespace, so write new Discrete(4) there. The masks are typed where Gymnasium's shape is
fixed (DR-97): discrete.Sample(mask) and discrete.Sample(probability: p) take NDArrays, box.Sample() takes
nothing, and multiDiscrete.Sample(masks) takes one int8 array per entry, Gymnasium's tuple as a list (object?[]
tuples keep working, nested ones included). Seed returns a simple space's seed as a BigInteger, and Clone() is
DeepCopy() (DR-81).
Rendering
The render mode is a constructor argument, fixed for the env's life, as in Gymnasium. Pass it as the RenderMode
enum or as Gymnasium's string: Gymnasium.Make("CartPole-v1", RenderMode.RgbArray), Gymnasium.Make("CartPole-v1", "rgb_array") and Gymnasium.Make("CartPole-v1", new { render_mode = "rgb_array" }) build the same env, as do
new CartPoleEnv(RenderMode.RgbArray) and new CartPoleEnv(renderMode: "rgb_array"). The env's RenderMode is
Gymnasium's string, since Gymnasium accepts modes no enum lists (a custom env's own); Gym.Envs.RenderModes names the
strings as constants (RenderModes.RgbArray is "rgb_array") and converts (RenderModes.TryParse(env.RenderMode) is
RenderMode.RgbArray, or null for a mode the enum does not list).
rgb_array:Render()returnsRenderResult.Rgb, whoseFrameis a(height, width, 3)uint8 array with the pixels Gymnasium's env draws.(NDArray)resultreads that frame directly (the result's own array: it lives until the result is disposed), and(Image)resultgives a new ImageSharp image to save or show.using Env<NDArray, NDArray> env = Gymnasium.Make<NDArray, NDArray>("CartPole-v1", RenderMode.RgbArray); using NDArray observation = env.Reset(seed: 0).Observation; using RenderResult result = env.Render(); NDArray frame = (NDArray)result; // the result's own frame; disposed with the result Console.WriteLine(frame.Shape); // (400, 600, 3), uint8, the pixels Gymnasium's CartPole drawsansi(toy_text):Render()returnsRenderResult.Ansiwith the text Gymnasium prints, escape codes included.using Env<NDArray, NDArray> env = Gymnasium.Make<NDArray, NDArray>("FrozenLake-v1", new { render_mode = "ansi" }); using NDArray start = env.Reset(seed: 0).Observation; // a 0-d int64 array: the cell, 0 to 15 using RenderResult text = env.Render(); Console.WriteLine(((RenderResult.Ansi)text).Text); // the 4x4 map, the agent's cell highlightedhuman: everyResetandStepdraws into a window, paced atMetadata["render_fps"], andRender()returnsRenderResult.None. Set the window type once, at start-up, or pass it toMake; without either, the first frame raisesDependencyNotInstalled, as Gymnasium does without pygame.Gymnasium.HumanViewerFactory = AvaloniaEnvViewer.Factory; // once, at start-up (WinFormEnvViewer.Factory on Windows) using Env<NDArray, NDArray> env = Gymnasium.Make<NDArray, NDArray>("CartPole-v1", RenderMode.Human); using NDArray observation = env.Reset(seed: 0).Observation; // the window opens and shows the first frame // Every Step draws the next frame, paced at metadata["render_fps"] (50 for CartPole); Render() returns None.Makealso takes the window for one env:Gymnasium.Make("CartPole-v1", RenderMode.Human, WinFormEnvViewer.Factory). The factory reaches the env (or theHumanRenderingwrappermakeputs around an env that only renders images), and the spec does not record it.
An env class also takes its own viewer: new CartPoleEnv(RenderMode.Human, viewerFactory: WinFormEnvViewer.Factory).
A viewer of your own implements IEnvViewer: Render(NDArray frame) receives the (H, W, 3) uint8 frame, which stays
the caller's (copy what you keep), and Close() closes the window; its factory is an EnvViewerFactory that returns the
viewer once its window is up. A viewer that can also render without blocking implements IAsyncEnvViewer
(RenderAsync, CloseAsync) as well.
A MuJoCo env's human window is Gymnasium's MuJoCo viewer, with its controls (DR-70): drag with the left button to orbit
the camera and with the right one to pan it (Shift: horizontally), scroll to zoom; Space pauses (each Step then waits
in the window until Space or → is pressed), → advances one step, Tab cycles the model's cameras, C shows contact
forces, E the bodies' frames, R makes the bodies transparent, 0 to 4 switch geom groups, T saves the frame as a PNG
(/tmp/frame_0000000.png and on, on the current drive), H hides the menu, S and F slow down and speed up, D switches
between drawing every step and real time, Esc closes the window. Its frames come from Gym.NET's software renderer, a
few to a few dozen a second.
Wrappers
A wrapper is an env around an env: Wrapper<TObs, TAct> and the ObservationWrapper, ActionWrapper and
RewardWrapper bases, as in Gymnasium. 1.0 has TimeLimit, OrderEnforcing, PassiveEnvChecker, Autoreset,
RecordEpisodeStatistics, RenderCollection, NormalizeObservation, NormalizeReward, TimeAwareObservation and
FlattenObservation (Gym.Wrappers):
using var env = new RecordEpisodeStatistics<NDArray, NDArray>(
new NormalizeObservation<NDArray, NDArray>(Gymnasium.Make<NDArray, NDArray>("CartPole-v1")));
NDArray observation = env.Reset(seed: 0).Observation;
while (true) {
using NDArray action = env.ActionSpace.Sample();
var (next, _, terminated, truncated, info) = env.Step(action);
observation.Dispose();
observation = next;
if (terminated || truncated) {
var episode = (Info)info["episode"]!; // {"r": return, "l": length, "t": seconds}
Console.WriteLine($"return {episode["r"]}, length {episode["l"]}");
break;
}
}
observation.Dispose();
A wrapper owns the env it wraps: disposing the outermost one closes the stack. env.Unwrapped is the innermost env, and
Python attributes are reachable by their Python names through a stack (env.GetWrapperAttr("gravity")).
Vector envs
A vector env steps several copies of an env at once and returns batched arrays:
// 8 carts in one env: CartPole's own vector env (its vector entry point), or "sync" for 8 copies of make's stack.
using VectorEnv<NDArray, NDArray> envs = Gymnasium.MakeVec<NDArray, NDArray>("CartPole-v1", numEnvs: 8);
envs.ActionSpace.Seed(42);
using NDArray observations = envs.Reset(seed: 42).Observations; // (8, 4) float32
using NDArray actions = envs.ActionSpace.Sample(); // (8,) int64
var (next, rewards, terminations, truncations, infos) = envs.Step(actions);
using (next)
using (rewards)
using (terminations)
using (truncations)
Console.WriteLine($"{next.Shape} {rewards.dtype.name} {terminations.dtype.name}"); // (8, 4) float64 bool
// A sub-env whose episode ended is reset by the next Step (AutoresetMode.NextStep, Gymnasium's default).
Gymnasium.MakeVec(id, n)uses the env's own vectorized class when it has one (CartPole'sCartPoleVectorEnv), elseSyncVectorEnv;vectorizationMode: "sync"asks forSyncVectorEnv(n copies ofmake's stack, stepped in turn).MakeVec(id, n, vectorizationMode: "sync", vectorKwargs: new { autoreset_mode = "SameStep" })picks the autoreset mode of aSyncVectorEnv:NextStep(the default) resets an ended sub-env at the following step, whose reward is 0;SameStepresets it at once and returns the last observation and info underfinal_obs/final_info;Disabledleaves it toReset(options: new Dictionary<string, object?> { ["reset_mask"] = mask }), withmaska bool array of one flag per sub-env. A vectorized env class such asCartPoleVectorEnvtakes novectorKwargs(Gymnasium'sError) and has its own mode,NextStepfor CartPole.- Infos are merged as Gymnasium merges them: one array per key, and a boolean
_keyarray saying which sub-envs set it. vectorizationMode: "async"buildsAsyncVectorEnv, which steps the sub-envs in parallel on worker threads and returns whatSyncVectorEnvreturns, bit for bit.vectorKwargschoose where the calls run (execution_mode:AsyncExecutionMode.ThreadPool, the default,DedicatedThreadswiththread_count, orThreadPerEnv), cap how many sub-envs run at once (max_concurrency) and keep Gymnasium'sshared_memoryandcopy. Each sub-env runs on another thread, so an env function must build its own spaces and buffers rather than share one object among sub-envs.
Arrays belong to the caller
Observations, rewards, actions and frames are NumSharp arrays in unmanaged memory. What an env or a space returns is
yours: dispose it when done (using, or Dispose() when the next one replaces it), and dispose the arrays you create.
Arrays you drop are reclaimed by the garbage collector eventually, and NumSharp's analyzer, which comes with the
packages, warns about them (NDW012). The samples above dispose everything and build without a warning.
The pre-0.26 API
Code written for Gym.NET 0.2 (Reset() returning only the observation, the 4-tuple Step with Done, Seed(s),
Render("rgb_array")) can run against 1.0 through Gym.Legacy.LegacyEnvAdapter, obsolete and removed in 2.0. The
migration guide maps every 0.2 call to its replacement.