Table of Contents

Class NormalizeReward<TObs, TAct>

Namespace
Gym.Wrappers
Assembly
Gym.dll

Scales rewards so their exponential moving average has roughly unit variance: every reward is divided by the square root of the running variance (plus Epsilon) of a discounted return, as Gymnasium's NormalizeReward does, byte for byte.

public class NormalizeReward<TObs, TAct> : Wrapper<TObs, TAct>, IDynamicAttributes, IWrapper, IEnv, IDisposable, IRecordConstructorArgs

Type Parameters

TObs

The observation type, passed through.

TAct

The action type, passed through.

Inheritance
Env<TObs, TAct>
Wrapper<TObs, TAct, TObs, TAct>
Wrapper<TObs, TAct>
NormalizeReward<TObs, TAct>
Implements
Inherited Members
Extension Methods

Remarks

The statistics live in this wrapper and survive resets: the discounted return is multiplied by gamma * (1 - terminated) before each reward is added (truncation does not clear it), as Gymnasium keeps it for backward compatibility. The variance starts at 1, so the first rewards are scaled by about 1. Freeze the statistics for evaluation with UpdateRunningMean. Not thread-safe.

Reference: gymnasium/wrappers/stateful_reward.py.

Constructors

NormalizeReward(Env<TObs, TAct>, NormalizeRewardOptions)

Builds the instance from its options record: the properties set, the constructor's defaults for the rest.

public NormalizeReward(Env<TObs, TAct> env, NormalizeRewardOptions options)

Parameters

env Env<TObs, TAct>

The env to wrap.

options NormalizeRewardOptions

The options.

Exceptions

ArgumentNullException

Thrown when options is null.

NormalizeReward(Env<TObs, TAct>, double, double)

Wraps env.

public NormalizeReward(Env<TObs, TAct> env, double gamma = 0.99, double epsilon = 1E-08)

Parameters

env Env<TObs, TAct>

The env whose rewards to scale.

gamma double

The discount of the tracked return, in [0, 1]: a value above 1 would make the average diverge.

epsilon double

Added to the variance under the square root, strictly positive, so the division never reaches zero.

Exceptions

InvalidBound

Thrown for a gamma outside [0, 1] (gamma should be in the interval [0, 1]. Received 1.5) or a non-positive epsilon, before the env is looked at, as in Gymnasium.

TypeError

Thrown when env is null.

Properties

DiscountedReward

The tracked discounted return, a (1,) float64 array (discounted_reward), replaced at every step.

public NDArray DiscountedReward { get; set; }

Property Value

NDArray

Epsilon

The variance's offset under the square root (epsilon).

public double Epsilon { get; set; }

Property Value

double

Gamma

The discount of the tracked return (gamma).

public double Gamma { get; set; }

Property Value

double

ReturnRms

The running statistics of the discounted return (return_rms); owned by this wrapper.

public RunningMeanStd ReturnRms { get; }

Property Value

RunningMeanStd

SavedKwargs

The recorded constructor arguments, {'gamma': 0.99, 'epsilon': 1e-08} (Python's _saved_kwargs).

[PyName("_saved_kwargs")]
public Info SavedKwargs { get; }

Property Value

Info

UpdateRunningMean

Whether steps update the statistics (update_running_mean); turn off to evaluate with frozen statistics.

public bool UpdateRunningMean { get; set; }

Property Value

bool

Methods

Dispose(bool)

Disposes the statistics and the tracked return with the stack.

protected override void Dispose(bool disposing)

Parameters

disposing bool

true from Dispose().

Step(TAct)

Steps the env, updates the discounted return and its statistics, and scales the reward.

public override StepResult<TObs> Step(TAct action)

Parameters

action TAct

The action, passed on.

Returns

StepResult<TObs>

The env's transition with the reward divided by sqrt(var + epsilon).