Class NormalizeReward<TObs, TAct>
Scales rewards so their exponential moving average has roughly unit variance: every reward is divided by the square
root of the running variance (plus Epsilon) of a discounted return, as Gymnasium's
NormalizeReward does, byte for byte.
public class NormalizeReward<TObs, TAct> : Wrapper<TObs, TAct>, IDynamicAttributes, IWrapper, IEnv, IDisposable, IRecordConstructorArgs
Type Parameters
TObsThe observation type, passed through.
TActThe action type, passed through.
- Inheritance
-
Env<TObs, TAct>Wrapper<TObs, TAct, TObs, TAct>Wrapper<TObs, TAct>NormalizeReward<TObs, TAct>
- Implements
- Inherited Members
- Extension Methods
Remarks
The statistics live in this wrapper and survive resets: the discounted return is multiplied by
gamma * (1 - terminated) before each reward is added (truncation does not clear it), as Gymnasium keeps it for
backward compatibility. The variance starts at 1, so the first rewards are scaled by about 1. Freeze the statistics
for evaluation with UpdateRunningMean. Not thread-safe.
Reference: gymnasium/wrappers/stateful_reward.py.
Constructors
NormalizeReward(Env<TObs, TAct>, NormalizeRewardOptions)
Builds the instance from its options record: the properties set, the constructor's defaults for the rest.
public NormalizeReward(Env<TObs, TAct> env, NormalizeRewardOptions options)
Parameters
envEnv<TObs, TAct>The env to wrap.
optionsNormalizeRewardOptionsThe options.
Exceptions
- ArgumentNullException
Thrown when
optionsis null.
NormalizeReward(Env<TObs, TAct>, double, double)
Wraps env.
public NormalizeReward(Env<TObs, TAct> env, double gamma = 0.99, double epsilon = 1E-08)
Parameters
envEnv<TObs, TAct>The env whose rewards to scale.
gammadoubleThe discount of the tracked return, in [0, 1]: a value above 1 would make the average diverge.
epsilondoubleAdded to the variance under the square root, strictly positive, so the division never reaches zero.
Exceptions
- InvalidBound
Thrown for a
gammaoutside [0, 1] () or a non-positivegammashould be in the interval [0, 1]. Received 1.5epsilon, before the env is looked at, as in Gymnasium.- TypeError
Thrown when
envis null.
Properties
DiscountedReward
The tracked discounted return, a (1,) float64 array (discounted_reward), replaced at every step.
public NDArray DiscountedReward { get; set; }
Property Value
Epsilon
The variance's offset under the square root (epsilon).
public double Epsilon { get; set; }
Property Value
Gamma
The discount of the tracked return (gamma).
public double Gamma { get; set; }
Property Value
ReturnRms
The running statistics of the discounted return (return_rms); owned by this wrapper.
public RunningMeanStd ReturnRms { get; }
Property Value
SavedKwargs
The recorded constructor arguments, {'gamma': 0.99, 'epsilon': 1e-08} (Python's _saved_kwargs).
[PyName("_saved_kwargs")]
public Info SavedKwargs { get; }
Property Value
UpdateRunningMean
Whether steps update the statistics (update_running_mean); turn off to evaluate with frozen statistics.
public bool UpdateRunningMean { get; set; }
Property Value
Methods
Dispose(bool)
Disposes the statistics and the tracked return with the stack.
protected override void Dispose(bool disposing)
Parameters
Step(TAct)
Steps the env, updates the discounted return and its statistics, and scales the reward.
public override StepResult<TObs> Step(TAct action)
Parameters
actionTActThe action, passed on.
Returns
- StepResult<TObs>
The env's transition with the reward divided by
sqrt(var + epsilon).