Goal-Conditioned Supervised Learning (GCSL)

This repository provides an implementation of Goal-Conditioned Supervised Learning (GCSL), as proposed in Learning to Reach Goals via Iterated Supervised Learning The manuscript is available on arXiv

If you use this codebase, please cite

Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, Sergey Levine. Learning to Reach Goals via Iterated Supervised Learning

Bibtex source is provided at the bottom of the Readme.

Setup

Conda

conda env create -f environment/environment.yml

Docker

docker pull dibyaghosh/gcsl:0.1

Example script

python experiments/gcsl_example.py

If you have, and would like to use, a GPU, you will need to additionally install a GPU-compiled version of PyTorch. To do so, simply run

pip uninstall torch && pip install torch==1.1.0

Development Notes

The directory structure currently looks like this:

• gcsl (Contains all code)

  • envs (Contains all environment files and wrappers)
  • algo (Contains all GCSL code)
    • gcsl.py (implements high-level algorithm logic, e.g. data collection, policy update, evaluate, save data)
    • buffer.py (The replay buffer used to relabel and sample (s,g,a,h) tuples
    • networks.py (Implements neural network policies.)
    • variants.py (Contains relevant hyperparameters for GCSL)

• experiments (Contains all launcher files)

• doodad (We require this old version of doodad)
• dependencies (Contains other libraries like rlkit, rlutil, room_world, multiworld, etc.)
• data (Not synced by github, but this will contain all experiment logs)

Please file an issue if you have trouble running this code.

Bibtex

@article{Ghosh2019LearningTR,
title={Learning To Reach Goals Without Reinforcement Learning},
author={Dibya Ghosh and Abhishek Gupta and Justin Fu and Ashwin Reddy and Coline Devin and Benjamin Eysenbach and Sergey Levine},
journal={ArXiv},
year={2019},
volume={abs/1912.06088}
}

Issues

Why onlu support discrete envs in the source code?

opened on 2022-02-17 09:08:00 by poisonwine

hello, I've read the source code of the paper, and I found that this source code discretes all the envs after creating the env. What should I do in order to support continous envs?

Vanilla GCSL implementation

opened on 2021-10-06 18:56:39 by cheind

Hey,

I've added a barebone GCSL implementation to https://github.com/cheind/gcsl, featuring a new goal-reaching cartpole environment. I believe the code is quite easy to digest in its current form and might help users to get started more easily.

Error: TypeError: argument of type 'property' is not iterable

opened on 2021-07-07 16:35:45 by YSLIU627

When I run your code, I encounter such an error:

Traceback (most recent call last): File "/home/gcsl/scripts/run_experiment_lite_doodad.py", line 10, in fn() File "/home/gcsl/doodad/easy_sweep/hyper_sweep.py", line 70, in run_method_args run_method(**config) File "gcsl_example.py", line 10, in run import rlutil.torch as torch File "/home/gcsl/dependencies/rlutil/torch/init.py", line 31, in _replace_funcs(globals()) File "/home/gcsl/dependencies/rlutil/torch/init.py", line 24, in _replace_funcs if _value.doc and 'device (:class:torch.device, optional):' in _value.doc: TypeError: argument of type 'property' is not iterable

It seems that there may exist some error in the function _replace_funcs.

Source code for TD3-HER baseline?

opened on 2021-06-14 21:05:59 by DrJimFan

This is one of the best papers I've read this year. Thank you so much for open-sourcing it!

Do you still have the code for TD3-HER baseline? Would you mind open-sourcing that part as well? Thanks in advance for your help!

TimeSeries analysis

opened on 2021-05-20 03:44:07 by algopy

Please provide a example on how to use GCSL for TimeSeries analysis .

Handle dependencies in setup.py

opened on 2020-09-21 22:54:26 by scottemmons

I was interested in using the environments you used in this project, and I wanted to be able to include the environments as a package in another project I'm developing. In order to do so, I wanted to make it possible to install your code with pip. With the changes I've made in this pull request, running this command in the root directory:

pip install .

leads to what appears to be a successful installation of the environments. Specifically, the following code runs successfully:

``` from gcsl import envs

for env_name in envs.env_names: envs.create_env(env_name) print(f"Successfully created env: {env_name}") ```

It's probably worth testing if the rest of the code, e.g., the training script, also works after pip install .