Constraints as Rewards: Reinforcement Learning for Robots without Reward Functions

This repository provides the official pytorch implementation of QRSAC-Lagrangian algorithm presented in the paper: "Constraints as Rewards: Reinforcement Learning for Robots without Reward Functions".

   

Contact:

• Yu ISHIHARA: [email protected]

Prerequisites

• Install pytorch by following the instruction on the official website.
• Install gymnasium

$ pip install gymnasium

• Install car

$ pip install .

How to run the training script

Go to scripts and execute the train_pendulum.py.

$ cd scripts
# It runs with QRSAC algorithm with CaR by default
# The default constraint is episodic. Eq. (18) of the paper.
$ python train_pendulum.py

To run the algorithm with timestep constraint (Eq. (17)) do:

$ python train_pendulum.py --constraint-type timestep

To run QRSAC without CaR (Training with original rewards) do:

$ python train_pendulum.py --without-car

Citation

@article{ishihara2025car,
         title={Constraints as Rewards: Reinforcement Learning for Robots without Reward Functions}, 
         author={Yu Ishihara and Noriaki Takasugi and Kotaro Kawakami and Masaya Kinoshita and Kazumi Aoyama},
         journal={arXiv preprint arXiv:2501.04228},
         year={2025},
}

License

• MIT License.