Abstract
While reinforcement learning for legged robots has achieved high motor performance, it has been constrained by the limited exploration capability of actions confined to the joint space. To address this issue, this study proposes a new method, Wrench-Augmented Reinforcement Learning (WARL), which introduces a wrench (force and torque) into the action space. The proposed method combines wrench-guided exploration with a success rate-based curriculum mechanism to expand exploration capabilities in the early stages of learning, with the ultimate goal of acquiring behaviors based solely on joint control. Experiments using a quadruped robot demonstrated that WARL can learn robustly across diverse terrains and motor tasks without requiring terrain-specific reward adjustments or complex curriculum designs. Furthermore, an ablation study verified the effectiveness of the Switching Curriculum, which gradually eliminates the wrench.
Method 1: Wrench Augmentation
WARL reduces the mismatch between the action and task spaces by allowing the agent to exert hypothetical wrenches (forces and torques) on its body. This "Wrench-Augmentation" assists the robot in overcoming initial learning hurdles, enabling dynamic whole-body motions from the very beginning of the training process.
Accelerated Spatial Exploration
By including a wrench in the action space, it becomes possible to explore directly in the robot's position space with simple exploration noise. This dramatically improves the spatial exploration capability compared to conventional joint-space actions, enabling the agent to efficiently discover successful behaviors even in complex tasks.
Method 2: Switching Curriculum
A. Joint + Wrench Learning
In the first phase, the agent simultaneously learns a joint policy and a wrench policy. The wrench policy applies hypothetical external forces (wrenches) to assist the robot in overcoming initial exploration hurdles. The successful wrench outputs generated during this phase are stored in a buffer to be utilized later.
B. Joint-only Adaptation Learning
In the second phase, the agent learns exclusively with the joint policy. Instead of computing new wrenches dynamically, it reuses the previously stored wrench sequences as a decaying auxiliary input. This ensures that the joint policy learns to reproduce the successful behaviors without relying on active external assistance.
C. Switching Curriculum
To prevent training instability, the Switching Curriculum alternates between the Joint+Wrench Learning phase and the Joint-only Adaptation Learning phase. As training progresses, the magnitude of the applied wrench is scaled down using an exponentially decaying coefficient (α). Ultimately, the dependence on the wrench is completely eliminated, yielding a robust, joint-only control policy.
Results
Learning Result 1: Position Reaching Task
In this task, the robot learns to reach and maintain various target positions. WARL significantly improves exploration efficiency, allowing the policy to rapidly cover a wide range of state spaces and robustly achieve the desired poses without relying on complex, task-specific reward shaping.
Learning Result 2: Dynamic Behavior Task
For more complex, dynamic tasks such as jumping over hurdles, leaping across gaps, or performing acrobatics, standard reinforcement learning often fails due to the vast action space and sparse successes. By leveraging wrench-augmented exploration alongside the Switching Curriculum, WARL successfully and consistently acquires these highly dynamic whole-body motions.
Sim2Sim Transfer
To verify the robustness of the learned policies and ensure they are not exploiting simulator-specific dynamics, we perform Sim2Sim transfer. The policy trained in the primary environment (e.g., Isaac Gym) is directly deployed in a completely different physics simulator (MuJoCo). The robot successfully executes the learned dynamic behaviors, demonstrating the high reliability and true physical viability of the acquired joint-only control.
Detailed Video
BibTeX
@inproceedings{yoneda2026warl,
author = {Yoneda, Keita and Kawaharazuka, Kento and Okada, Kei},
title = {WARL: Wrench-Augmented Reinforcement Learning for Task-Agnostic Learning in Legged Robots},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2026}
}