Class NormalizeRewardVector<TObs, TAct>
Scales the batched rewards by the running standard deviation of each sub-env's discounted return, as Gymnasium's
vector NormalizeReward does: reward / sqrt(var + epsilon), where the returns accumulate per sub-env
with discount gamma.
[PyName("NormalizeReward")]
public class NormalizeRewardVector<TObs, TAct> : VectorWrapper<TObs, TAct>, IVectorEnv, IDisposable, IRecordConstructorArgs
Type Parameters
TObsThe batched observation type, passed through.
TActThe batched action type, passed through.
- Inheritance
-
VectorEnv<TObs, TAct>VectorWrapper<TObs, TAct, TObs, TAct>VectorWrapper<TObs, TAct>NormalizeRewardVector<TObs, TAct>
- Implements
- Inherited Members
- Extension Methods
Remarks
The accumulated returns are float32 (Gymnasium's array), so each update is computed as NumPy computes it (the
discounted return in float64 through the int 1 - terminated, stored back as float32). With NEXT_STEP
the autoreset step after an episode's end is skipped; with SAME_STEP every step counts and an ended return
restarts at once. Not thread-safe.
Reference: gymnasium/wrappers/vector/stateful_reward.py.
Constructors
NormalizeRewardVector(VectorEnv<TObs, TAct>, double, double)
Wraps env.
public NormalizeRewardVector(VectorEnv<TObs, TAct> env, double gamma = 0.99, double epsilon = 1E-08)
Parameters
envVectorEnv<TObs, TAct>The vector env.
gammadoubleThe discount, in [0, 1].
epsilondoubleAdded to the variance under the square root, strictly positive.
Exceptions
- InvalidBound
Thrown for a gamma outside [0, 1] or a non-positive epsilon, with Gymnasium's messages.
- TypeError
Thrown when
envis null.
Properties
AccumulatedReward
Each sub-env's discounted return, float32 (accumulated_reward); owned by this wrapper.
public NDArray AccumulatedReward { get; }
Property Value
Epsilon
The variance's offset under the square root (epsilon).
public double Epsilon { get; set; }
Property Value
Gamma
The discount (gamma).
public double Gamma { get; set; }
Property Value
ReturnRms
The running statistics of the discounted returns (return_rms); owned by this wrapper.
public RunningMeanStd ReturnRms { get; }
Property Value
SavedKwargs
The recorded constructor arguments (Python's _saved_kwargs).
[PyName("_saved_kwargs")]
public Info SavedKwargs { get; }
Property Value
UpdateRunningMean
Whether returns update the statistics (update_running_mean).
public bool UpdateRunningMean { get; set; }
Property Value
Methods
Dispose(bool)
Releases the statistics and the per-sub-env arrays, then disposes as every vector wrapper does.
protected override void Dispose(bool disposing)
Parameters
Reset(IReadOnlyList<BigInteger?>, IReadOnlyDictionary<string, object?>?)
Clears the returns and the done flags, then resets the env with one seed per sub-env.
public override VectorResetResult<TObs> Reset(IReadOnlyList<BigInteger?> seeds, IReadOnlyDictionary<string, object?>? options = null)
Parameters
seedsIReadOnlyList<BigInteger?>The seeds, passed on.
optionsIReadOnlyDictionary<string, object>The options, passed on.
Returns
- VectorResetResult<TObs>
The env's reset.
Reset(BigInteger?, IReadOnlyDictionary<string, object?>?)
Clears the returns and the done flags, then resets the env.
public override VectorResetResult<TObs> Reset(BigInteger? seed = null, IReadOnlyDictionary<string, object?>? options = null)
Parameters
seedBigInteger?The seed, passed on.
optionsIReadOnlyDictionary<string, object>The options, passed on.
Returns
- VectorResetResult<TObs>
The env's reset.
Step(TAct)
Steps the env, accumulates the returns of the active sub-envs, updates the statistics, and scales the rewards.
public override VectorStepResult<TObs> Step(TAct actions)
Parameters
actionsTActThe actions, passed on.
Returns
- VectorStepResult<TObs>
The transition with the rewards divided by
sqrt(var + epsilon)(a new float64 array).