Table of Contents

Class NormalizeRewardVector<TObs, TAct>

Namespace
Gym.Wrappers.Vector
Assembly
Gym.dll

Scales the batched rewards by the running standard deviation of each sub-env's discounted return, as Gymnasium's vector NormalizeReward does: reward / sqrt(var + epsilon), where the returns accumulate per sub-env with discount gamma.

[PyName("NormalizeReward")]
public class NormalizeRewardVector<TObs, TAct> : VectorWrapper<TObs, TAct>, IVectorEnv, IDisposable, IRecordConstructorArgs

Type Parameters

TObs

The batched observation type, passed through.

TAct

The batched action type, passed through.

Inheritance
VectorEnv<TObs, TAct>
VectorWrapper<TObs, TAct, TObs, TAct>
VectorWrapper<TObs, TAct>
NormalizeRewardVector<TObs, TAct>
Implements
Inherited Members
Extension Methods

Remarks

The accumulated returns are float32 (Gymnasium's array), so each update is computed as NumPy computes it (the discounted return in float64 through the int 1 - terminated, stored back as float32). With NEXT_STEP the autoreset step after an episode's end is skipped; with SAME_STEP every step counts and an ended return restarts at once. Not thread-safe.

Reference: gymnasium/wrappers/vector/stateful_reward.py.

Constructors

NormalizeRewardVector(VectorEnv<TObs, TAct>, double, double)

Wraps env.

public NormalizeRewardVector(VectorEnv<TObs, TAct> env, double gamma = 0.99, double epsilon = 1E-08)

Parameters

env VectorEnv<TObs, TAct>

The vector env.

gamma double

The discount, in [0, 1].

epsilon double

Added to the variance under the square root, strictly positive.

Exceptions

InvalidBound

Thrown for a gamma outside [0, 1] or a non-positive epsilon, with Gymnasium's messages.

TypeError

Thrown when env is null.

Properties

AccumulatedReward

Each sub-env's discounted return, float32 (accumulated_reward); owned by this wrapper.

public NDArray AccumulatedReward { get; }

Property Value

NDArray

Epsilon

The variance's offset under the square root (epsilon).

public double Epsilon { get; set; }

Property Value

double

Gamma

The discount (gamma).

public double Gamma { get; set; }

Property Value

double

ReturnRms

The running statistics of the discounted returns (return_rms); owned by this wrapper.

public RunningMeanStd ReturnRms { get; }

Property Value

RunningMeanStd

SavedKwargs

The recorded constructor arguments (Python's _saved_kwargs).

[PyName("_saved_kwargs")]
public Info SavedKwargs { get; }

Property Value

Info

UpdateRunningMean

Whether returns update the statistics (update_running_mean).

public bool UpdateRunningMean { get; set; }

Property Value

bool

Methods

Dispose(bool)

Releases the statistics and the per-sub-env arrays, then disposes as every vector wrapper does.

protected override void Dispose(bool disposing)

Parameters

disposing bool

true from Dispose().

Reset(IReadOnlyList<BigInteger?>, IReadOnlyDictionary<string, object?>?)

Clears the returns and the done flags, then resets the env with one seed per sub-env.

public override VectorResetResult<TObs> Reset(IReadOnlyList<BigInteger?> seeds, IReadOnlyDictionary<string, object?>? options = null)

Parameters

seeds IReadOnlyList<BigInteger?>

The seeds, passed on.

options IReadOnlyDictionary<string, object>

The options, passed on.

Returns

VectorResetResult<TObs>

The env's reset.

Reset(BigInteger?, IReadOnlyDictionary<string, object?>?)

Clears the returns and the done flags, then resets the env.

public override VectorResetResult<TObs> Reset(BigInteger? seed = null, IReadOnlyDictionary<string, object?>? options = null)

Parameters

seed BigInteger?

The seed, passed on.

options IReadOnlyDictionary<string, object>

The options, passed on.

Returns

VectorResetResult<TObs>

The env's reset.

Step(TAct)

Steps the env, accumulates the returns of the active sub-envs, updates the statistics, and scales the rewards.

public override VectorStepResult<TObs> Step(TAct actions)

Parameters

actions TAct

The actions, passed on.

Returns

VectorStepResult<TObs>

The transition with the rewards divided by sqrt(var + epsilon) (a new float64 array).