Table of Contents

Class CliffWalkingEnv

Namespace
Gym.Environments.Envs.ToyText
Assembly
Gym.Environments.dll

CliffWalking-v1 and CliffWalkingSlippery-v1: a 4x12 grid walked from the bottom-left corner to the bottom-right one along a cliff; stepping off it costs -100 and returns the agent to the start, every other step costs -1. Slippery moves go perpendicular with probability 2/3. Byte for byte what Gymnasium's CliffWalking computes: the transition table P, the generator draws, observations, rewards, {"prob": p} infos, ansi text and rgb_array pixels.

public class CliffWalkingEnv : Env<NDArray, NDArray>, IEnv, IDisposable, IDynamicAttributes
Inheritance
CliffWalkingEnv
Implements
Inherited Members
Extension Methods

Remarks

Like Gymnasium's, the env checks no action against its space: self.P[self.s][a] is a dict lookup, so any number equal to 0 (up), 1 (right), 2 (down) or 3 (left) is an action, anything else raises KeyError. Each step draws one float64 from the env's generator, even on a certain move. The goal is not absorbing: a step from it moves on, as in Gymnasium. Observations are 0-d int64 arrays (DR-08); rewards are Gymnasium's -1 and -100 as float64.

Rendering is Gymnasium's pixel art at 64-pixel cells (its art scale is 1, so the images keep their sizes); ansi marks the agent x, the goal T, the cliff C. Any render mode other than human and ansi draws and returns the frame, as Gymnasium's else branches do. Human mode renders into the window from viewer_factory or Gymnasium.HumanViewerFactory (DR-20) at 4 frames per second. Not thread-safe.

Reference: gymnasium/envs/toy_text/cliffwalking.py.

Its info reads typed (DR-91): CliffWalkingInfo info = step.Info; (CliffWalkingInfo), with the dict's own values still behind the indexer.

Constructors

CliffWalkingEnv(CliffWalkingEnvOptions)

Builds the instance from its options record: the properties set, the constructor's defaults for the rest.

public CliffWalkingEnv(CliffWalkingEnvOptions options)

Parameters

options CliffWalkingEnvOptions

The options.

Exceptions

ArgumentNullException

Thrown when options is null.

CliffWalkingEnv(RenderMode, bool, EnvViewerFactory?)

Builds the env with the render mode given as the RenderMode enum (DR-84, DR-93), as CliffWalkingEnv(string?, bool, EnvViewerFactory?) builds it from the mode's string: new CliffWalkingEnv(RenderMode.RgbArray) builds what new CliffWalkingEnv(renderMode: "rgb_array") builds, the member reaching the env as Gymnasium's string (Name(RenderMode)), which its RenderMode property then holds.

public CliffWalkingEnv(RenderMode renderMode, bool isSlippery = false, EnvViewerFactory? viewerFactory = null)

Parameters

renderMode RenderMode

The render mode, which the env must list in its metadata["render_modes"] as for the string; it behaves exactly as its Gymnasium string does in the other constructor.

isSlippery bool

Gymnasium's is_slippery: each move goes as intended or to either side, with probability 1/3 each.

viewerFactory EnvViewerFactory

Gym.NET's viewer_factory (DR-20) for human mode; null uses Gymnasium.HumanViewerFactory.

Remarks

The mode is required and comes first, so new CliffWalkingEnv() keeps meaning the other constructor, without a mode; leave the mode out there for no rendering. Every other parameter means what it means there.

Exceptions

ArgumentOutOfRangeException

Thrown for a value RenderMode does not declare (a cast integer such as (RenderMode)99).

CliffWalkingEnv(string?, bool, EnvViewerFactory?)

Creates the env.

public CliffWalkingEnv(string? renderMode = null, bool isSlippery = false, EnvViewerFactory? viewerFactory = null)

Parameters

renderMode string

Gymnasium's render_mode: null, "human", "rgb_array" or "ansi".

isSlippery bool

Gymnasium's is_slippery: each move goes as intended or to either side, with probability 1/3 each.

viewerFactory EnvViewerFactory

Gym.NET's viewer_factory (DR-20) for human mode; null uses Gymnasium.HumanViewerFactory.

Fields

Down

Gymnasium's DOWN, action 2.

public const long Down = 2

Field Value

long

Left

Gymnasium's LEFT, action 3.

public const long Left = 3

Field Value

long

Right

Gymnasium's RIGHT, action 1.

public const long Right = 1

Field Value

long

Up

Gymnasium's UP, action 0.

public const long Up = 0

Field Value

long

Properties

AgentImg

Gymnasium's agent_img.

public Surface? AgentImg { get; }

Property Value

Surface

CellSize

Gymnasium's cell_size: (64, 64) pixels.

public (int Width, int Height) CellSize { get; }

Property Value

(int Width, int Height)

ClassMetadata

Gymnasium's class attribute metadata: {"render_modes": ["human", "rgb_array", "ansi"], "render_fps": 4}. Every env starts with a copy as its instance metadata.

public static Metadata ClassMetadata { get; }

Property Value

Metadata

Remarks

A new dict on every read, so editing one env's metadata never reaches another (Python's is one shared dict).

CliffImgs

Gymnasium's cliff_imgs: the cliff tiles by name (left, mid_a, mid_b, right).

public IReadOnlyDictionary<string, Surface>? CliffImgs { get; }

Property Value

IReadOnlyDictionary<string, Surface>

Clock

Gymnasium's clock: paces human rendering, null until the first image render.

public Clock? Clock { get; }

Property Value

Clock

GoalFlagImg

Gymnasium's goal_flag_img.

public Surface? GoalFlagImg { get; }

Property Value

Surface

GoalImg

Gymnasium's goal_img, the letter G.

public Surface? GoalImg { get; }

Property Value

Surface

GroundImg

Gymnasium's ground_img; null until the first image render.

public Surface? GroundImg { get; }

Property Value

Surface

InitialStateDistrib

Gymnasium's initial_state_distrib: 1 at StartStateIndex, a new array on every read.

public double[] InitialStateDistrib { get; }

Property Value

double[]

IsSlippery

Gymnasium's is_slippery, fixed at construction.

public bool IsSlippery { get; }

Property Value

bool

LastAction

Gymnasium's lastaction: the action key of the last step, or null after a reset (Python's None).

[PyName("lastaction")]
public long? LastAction { get; }

Property Value

long?

NA

Gymnasium's nA: 4 actions.

[PyName("nA")]
public int NA { get; }

Property Value

int

NS

Gymnasium's nS: 48 states.

[PyName("nS")]
public int NS { get; }

Property Value

int

P

Gymnasium's P: for each state and action, the possible outcomes (probability, next state, reward, terminated) in Gymnasium's order (a slippery move: counter-clockwise, as intended, clockwise).

[PyName("P")]
public IReadOnlyDictionary<long, IReadOnlyDictionary<long, IReadOnlyList<ToyTextTransition>>> P { get; }

Property Value

IReadOnlyDictionary<long, IReadOnlyDictionary<long, IReadOnlyList<ToyTextTransition>>>

S

Gymnasium's s: the agent's state, row * 12 + col; null before the first reset. Assigning one injects a state (one outside the grid makes the next step's lookup fail with KeyError).

[PyName("s")]
public long? S { get; set; }

Property Value

long?

Shape

Gymnasium's shape: (4, 12).

public (int Rows, int Columns) Shape { get; }

Property Value

(int Width, int Height)

StartImg

Gymnasium's start_img, the letter S.

public Surface? StartImg { get; }

Property Value

Surface

StartStateIndex

Gymnasium's start_state_index: 36, the cell (3, 0).

public long StartStateIndex { get; }

Property Value

long

WindowSize

Gymnasium's window_size: (768, 256) pixels, twelve cells by four.

public (int Width, int Height) WindowSize { get; }

Property Value

(int Width, int Height)

WindowSurface

Gymnasium's window_surface: the frame every image render draws on, null until the first one.

public Surface? WindowSurface { get; }

Property Value

Surface

Methods

Close()

Once a window surface exists, shuts the human-mode window down (pygame's display.quit()); the surface is kept.

public override void Close()

Remarks

ConvertAction(object?)

The untyped step's action conversion, with Python's order kept: the state's existence first, then the dict lookup on the action as given, so the errors name the Python counterpart of the value given.

protected override NDArray ConvertAction(object? action)

Parameters

action object

The action as given to the untyped step.

Returns

NDArray

The action as an array: the one given, or a new one the step releases.

Exceptions

AttributeError

Thrown before the first reset.

KeyError

Thrown for an action no key equals.

TypeError

Thrown for an array of one or more dimensions.

Render()

Renders by the render mode: ansi returns the grid as text; any other mode draws Gymnasium's pixel art (and returns the frame unless the mode is human).

public override RenderResult Render()

Returns

RenderResult

ansi: the text. human: nothing. Any other mode: the (256, 768, 3) uint8 frame, a new array the caller owns.

Remarks

Exceptions

AssertionError

Thrown without a render mode when the env has no spec (assert self.spec is not None).

AttributeError

Thrown before the first reset (no s; for images after the first cell's ground was drawn).

KeyError

Thrown in human mode when metadata["render_fps"] was removed.

DependencyNotInstalled

Thrown on the first human render when no viewer factory is set (DR-20).

Reset(BigInteger?, IReadOnlyDictionary<string, object?>?)

Starts an episode: reseeds the generator when a seed is given, then draws the start state (one draw, always state 36) and clears the last action. Renders in human mode.

public override ResetResult<NDArray> Reset(BigInteger? seed = null, IReadOnlyDictionary<string, object?>? options = null)

Parameters

seed BigInteger?

The seed of a fresh Generator(PCG64(SeedSequence(seed))), or null to keep the generator.

options IReadOnlyDictionary<string, object>

Ignored, as in Gymnasium.

Returns

ResetResult<NDArray>

The state as a 0-d int64 array, and {"prob": 1} (a Python int).

Remarks

Exceptions

Error

Thrown for a negative seed, with Gymnasium's message.

DependencyNotInstalled

Thrown in human mode by the first render when no viewer factory is set (DR-20).

Step(NDArray)

Moves by the action: looks up P[s][a], draws one outcome (one float64 from the generator) and moves there. Renders in human mode.

public override StepResult<NDArray> Step(NDArray action)

Parameters

action NDArray

A number equal to 0 (up), 1 (right), 2 (down) or 3 (left): a 0-d array, or through the untyped step any .NET number.

Returns

StepResult<NDArray>

The new state (a 0-d int64 array), the reward (-1, or -100 off the cliff), whether it is the goal, truncated=False, and {"prob": p}.

Remarks

Exceptions

AttributeError

Thrown before the first reset ('CliffWalkingEnv' object has no attribute 's'), before the action is looked at.

KeyError

Thrown for an action no key equals, or an injected state outside the grid, with Python's message.

TypeError

Thrown for an array action of one or more dimensions (unhashable type: 'numpy.ndarray').

DependencyNotInstalled

Thrown in human mode by the first render when no viewer factory is set (DR-20).