CATS | Revisiting Temporal Regularization for Smooth Control

Unlike explicit spatial regularizers, CATS can induce spatial smoothness indirectly through shared next states in the temporal penalty.

Action oscillation in deep RL

Deep RL policies can produce nonsmooth action oscillations that hinder deployment on physical robots. Reducing these oscillations calls for two policy properties: temporal smoothness, low policy sensitivity to state inputs received over time, and spatial smoothness, low policy sensitivity to small input changes caused by observation noise.

Arrow sizes are adjusted to make direction changes easier to see.

Limitations of Spatial Regularization

Although action oscillation is fundamentally a change in policy outputs over time, many existing methods seek smoothness across the state space without considering the temporal ordering of inputs. While they can promote spatial smoothness, their broad constraints may also restrict action changes needed for high return.

Spatial regularization

Figure 2: temporal-only CAPS preserves more return than spatial-only CAPS and LipsNet++ under stronger smoothing on Hopper and Walker. — At stronger smoothing levels (from right to left), temporal regularization preserves more return than spatial regularization.

Revisiting temporal regularization

Temporal regularization

Temporal regularization instead constrains action differences only along observed transitions, better preserving return under strong smoothing. However, sensitivity to noise in other directions appears unaddressed. Prior methods have thus added spatial penalties, reintroducing broad constraints and unclear penalty weighting.

Induced spatial smoothness

During training, however, the next state is not fixed, nor does the policy repeatedly visit only the same current state. We hypothesize that nearby states encountered during training share possible next states, through which temporal penalties can indirectly align their actions and promote spatial smoothness.

vs. Spatial Regularization (Matched Smoothness)

PPO / Hopper

By avoiding direct spatial regularization, CATS preserves more return than the spatial regularizers shown here at comparable action smoothness.

Spatial smoothness without a spatial penalty

(Lower sensitivity indicates better spatial smoothness.)
Evaluated at σ = 0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0.

CATS achieves policy sensitivity comparable to several explicit spatial regularization methods without an additional spatial penalty, yet better preserves return (see Table I). These results 1) show that spatial smoothness is not exclusive to spatial regularization and 2) suggest that performance loss stems from directly restricting action changes across multiple state-space directions, not from spatial smoothness itself.

Under observation noise

Noise scale σ

The induced spatial smoothness allows CATS to remain substantially smoother across noise scales while maintaining comparable or higher return than PPO.

Smooth control after sim-to-real transfer

Main view

Additional view

On real-world Reach, PPO's abrupt movements prevented the end effector from remaining near the target, reducing task performance. In contrast, CATS better preserved task performance while maintaining smooth control. We recorded an additional camera view to show PPO's small but persistent oscillations more clearly.

Conclusion

Our findings show that temporal regularization can provide both temporal and spatial smoothness while better preserving return under strong smoothing. More broadly, these findings motivate designing smoothness regularization around transition directions relevant to control, avoiding unnecessary constraints on action changes needed for task performance. Linear ramp-up complements this direction by allowing the policy to learn rewarding behavior before smoothing its actions, rather than seeking rewarding behavior under smoothness constraints. CATS demonstrates the practical value of these findings through smooth control in simulation and on a physical robot with little additional complexity.

BibTeX

@misc{ahn2026cats,
  title         = {Revisiting Temporal Regularization for Smooth Control in Deep Reinforcement Learning},
  author        = {SungJae Ahn and Jeong Woon Lee and Kyoleen Kwak and Hyoseok Hwang},
  year          = {2026},
  eprint        = {2610.07910},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.07910}
}