Datasets and code to evaluate transferability of dynamics learning methods.

rr-learning, updated 🕥 2022-12-08 11:29:37

Transferable Dynamics Learning

Benchmarking of different dynamics learning methods under changes in the distribution over controls. In this repository we provide the evaluation code and links to the corresponding evaluation data sets.

Robotic setup

The data was recorded on a 3-DOF torque-controlled real-robotic system. Please check the Open Dynamic Robot Initiative for the details about the robotic platform.

Robotic dataset

All the dataset files can be downloaded from here or alternatively from here

We use the aforementioned robotic platform to record different datasets under substantially different conditions in order to assess the predictive performance of dynamics learning algorithms beyond the iid. setting. In this study such conditions are in turn governed by the particular controllers used to generate the control inputs feed into the system. We consider two types of controllers depending on whether there is a feedback control loop or not.

Closed-loop dataset - Sine waves PD control

This dataset was generated using a superposition of sine waves to generate trajectories that were subsequently tracked by the robot using PD position control (Please refer to the paper for details). Check out some sampled robot movements under the 4 different families of controllers considered in the closed-loop dataset:

The shown movements are arranged according to the following diagram where each of the datasets D account for a particular configuration of sine angular frenquencies (low, high) and reachable task space (left, full). The arrows denote the different transfer settings that are discussed in our paper.

Open-loop dataset - GP torque controls

We used sampled trajectories from a Gaussian process (GP) directly as torque inputs (controls) to record this dataset in an open loop fashion. Please refer to the paper for details.

Dataset structure and naming convention.

The released files of the closed-loop dataset are named according to the following convention: * Sines_full.npz * Sines_training.npz * Sines_validation.npz * Sines_test_iid.npz * Sines_test_transfer_%d.npz

with the different test transfer files being indexed starting from 1 (e.g., Sines_test_transfer_1.npz). Note that each _full.npz files includes all recorded rollouts of its corresponding category. The rest of the files can be obtained from it as follows:

python -m DL.utils.data_extractor --raw_data Sines_full.npz --training Sines_training.npz --testiid Sines_test_iid.npz --validation Sines_validation.npz --testtransfer Sines_test_transfer_{}.npz

The open-loop dataset files are equally named, except that the prefix GPs is used instead of Sines.

Each of the released npz files can be indexed by the following keywords which account for different recorded variables: * measured_angles * measured_velocities * measured_torques * constrained_torques * desired_torques

Each of these keys is associated with a numpy array of shape (S, T, D), where S denotes the number of sequences/rollouts, and T, D are the sequence length and number of degrees of freedom, respectively. We recorded at a frequency of 1 Hz, meaning that the time elapsed between consecutive observations is 0.001 seconds. We recorded using a 3-DOF finger robot (D=3), and all rollouts have a total duration of 14s (T=14000).

Simulated closed-loop dataset

We also provide a simulated version of the closed-loop dataset. We keep the naming convention consistent with the real datasets but use the prefix Sim_Sines instead.

Evaluation.

Our evaluation protocol consists of two stages. First, we compute the error vectors of a particular method over different datasets and save them in a single .npz file. After we compute the error files corresponding to all benchmarked methods we aggregate the results in different plots as shown in the paper, which we refer to for further details.

Computing error vectors

As an example the following command computes the aforementioned error file for a linear model trained with SGD:

python -m DL.evaluation.evaluation \ --method linear_model_sgd \ --training_data Sines_training.npz \ --validation_data Sines_validation.npz \ --iid_test_data Sines_test_iid.npz \ --transfer_test_data Sines_test_transfer_1.npz Sines_test_transfer_2.npz Sines_test_transfer_3.npz \ --prediction_horizon 1 \ --history_length 1 \ --no-averaging \ --streaming \ --verbose \ --output_errors errors.npz

Among the implemented methods we have Gaussian Processes (SVGPR), System Identification methods (system_id), Neural networks NN, etc.

Dependencies

All dependencies are tracked using Pipenv. In order to reproduce our Python environment with all dependencies type the following command in this project's directory (transferable_dynamics_dataset):

pipenv install

After this the virtual environment can be activated by typing:

pipenv shell

Paper & Reference

Preprint.

In order to cite us please do so according to: @conference{AgudeloEspanaetal20, title = {A Real-Robot Dataset for Assessing Transferability of Learned Dynamics Models }, author = {Agudelo-España, D. and Zadaianchuk, A. and Wenk, P. and Garg, A. and Akpo, J. and Grimminger, F. and Viereck, J. and Naveau, M. and Righetti, L. and Martius, G. and Krause, A. and Sch{\"o}lkopf, B. and Bauer, S. and W{\"u}thrich, M.}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, year = {2020} }

Issues

Bump certifi from 2020.6.20 to 2022.12.7

opened on 2022-12-08 11:29:37 by dependabot[bot]

Bumps certifi from 2020.6.20 to 2022.12.7.

Commits

9e9e840 2022.12.07
b81bdb2 2022.09.24
939a28f 2022.09.14
aca828a 2022.06.15.2
de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...
b8eb5e9 2022.06.15.1
47fb7ab Fix deprecation warning on Python 3.11 (#199)
b0b48e0 fixes #198 -- update link in license
9d514b4 2022.06.15
4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/rr-learning/transferable_dynamics_dataset/network/alerts).

Bump protobuf from 3.12.4 to 3.18.3

opened on 2022-09-23 22:55:59 by dependabot[bot]

Bumps protobuf from 3.12.4 to 3.18.3.

Release notes

Sourced from protobuf's releases.

Protocol Buffers v3.18.3

C++

Reduce memory consumption of MessageSet parsing

This release addresses a Security Advisory for C++ and Python users

Protocol Buffers v3.16.1

Java

Improve performance characteristics of UnknownFieldSet parsing (#9371)

Protocol Buffers v3.18.2

Java

Improve performance characteristics of UnknownFieldSet parsing (#9371)

Protocol Buffers v3.18.1

Python

Update setup.py to reflect that we now require at least Python 3.5 (#8989)

Performance fix for DynamicMessage: force GetRaw() to be inlined (#9023)

Ruby

Update ruby_generator.cc to allow proto2 imports in proto3 (#9003)

Protocol Buffers v3.18.0

C++

Fix warnings raised by clang 11 (#8664)

Make StringPiece constructible from std::string_view (#8707)

Add missing capability attributes for LLVM 12 (#8714)

Stop using std::iterator (deprecated in C++17). (#8741)

Move field_access_listener from libprotobuf-lite to libprotobuf (#8775)

Fix #7047 Safely handle setlocale (#8735)

Remove deprecated version of SetTotalBytesLimit() (#8794)

Support arena allocation of google::protobuf::AnyMetadata (#8758)

Fix undefined symbol error around SharedCtor() (#8827)

Fix default value of enum(int) in json_util with proto2 (#8835)

Better Smaller ByteSizeLong

Introduce event filters for inject_field_listener_events

Reduce memory usage of DescriptorPool

For lazy fields copy serialized form when allowed.

Re-introduce the InlinedStringField class

v2 access listener

Reduce padding in the proto's ExtensionRegistry map.

GetExtension performance optimizations

Make tracker a static variable rather than call static functions

Support extensions in field access listener

Annotate MergeFrom for field access listener

Fix incomplete types for field access listener

Add map_entry/new_map_entry to SpecificField in MessageDifferencer. They record the map items which are different in MessageDifferencer's reporter.

Reduce binary size due to fieldless proto messages

TextFormat: ParseInfoTree supports getting field end location in addition to start.

... (truncated)

Commits

a902b39 No-op whitespace change
ae62acd Updating version.json and repo version numbers to: 18.3
f43ac49 Merge pull request #10542 from deannagarcia/3.18.x
9efdf55 Add missing includes
d1635e1 Apply patch
5b37c91 Update version.json with "lts": true (#10534)
c39d622 Merge pull request #10529 from protocolbuffers/deannagarcia-patch-5
f77d3b6 Update version.json
8178b06 Merge pull request #10503 from deannagarcia/3.18.x
24ca839 Add version file
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump urllib3 from 1.25.10 to 1.26.5

opened on 2022-09-20 22:12:07 by dependabot[bot]

Bumps urllib3 from 1.25.10 to 1.26.5.

Release notes

Sourced from urllib3's releases.

1.26.5

:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

Fixed deprecation warnings emitted in Python 3.10.

Updated vendored six library to 1.16.0.

Improved performance of URL parser when splitting the authority component.

If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors

1.26.4

:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

Changed behavior of the default SSLContext when connecting to HTTPS proxy during HTTPS requests. The default SSLContext now sets check_hostname=True.

If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors

1.26.3

:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

Fixed bytes and string comparison issue with headers (Pull #2141)

Changed ProxySchemeUnknown error message to be more actionable if the user supplies a proxy URL without a scheme (Pull #2107)

If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors

1.26.2

:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

Fixed an issue where wrap_socket and CERT_REQUIRED wouldn't be imported properly on Python 2.7.8 and earlier (Pull #2052)

1.26.1

:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

Fixed an issue where two User-Agent headers would be sent if a User-Agent header key is passed as bytes (Pull #2047)

1.26.0

:warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

Added support for HTTPS proxies contacting HTTPS servers (Pull #1923, Pull #1806)

Deprecated negotiating TLSv1 and TLSv1.1 by default. Users that still wish to use TLS earlier than 1.2 without a deprecation warning should opt-in explicitly by setting ssl_version=ssl.PROTOCOL_TLSv1_1 (Pull #2002) Starting in urllib3 v2.0: Connections that receive a DeprecationWarning will fail

Deprecated Retry options Retry.DEFAULT_METHOD_WHITELIST, Retry.DEFAULT_REDIRECT_HEADERS_BLACKLIST and Retry(method_whitelist=...) in favor of Retry.DEFAULT_ALLOWED_METHODS, Retry.DEFAULT_REMOVE_HEADERS_ON_REDIRECT, and Retry(allowed_methods=...) (Pull #2000) Starting in urllib3 v2.0: Deprecated options will be removed

... (truncated)

Changelog

Sourced from urllib3's changelog.

1.26.5 (2021-05-26)

Fixed deprecation warnings emitted in Python 3.10.

Updated vendored six library to 1.16.0.

Improved performance of URL parser when splitting the authority component.

1.26.4 (2021-03-15)

Changed behavior of the default SSLContext when connecting to HTTPS proxy during HTTPS requests. The default SSLContext now sets check_hostname=True.

1.26.3 (2021-01-26)

Fixed bytes and string comparison issue with headers (Pull #2141)

Changed ProxySchemeUnknown error message to be more actionable if the user supplies a proxy URL without a scheme. (Pull #2107)

1.26.2 (2020-11-12)

Fixed an issue where wrap_socket and CERT_REQUIRED wouldn't be imported properly on Python 2.7.8 and earlier (Pull #2052)

1.26.1 (2020-11-11)

Fixed an issue where two User-Agent headers would be sent if a User-Agent header key is passed as bytes (Pull #2047)

1.26.0 (2020-11-10)

NOTE: urllib3 v2.0 will drop support for Python 2. Read more in the v2.0 Roadmap <https://urllib3.readthedocs.io/en/latest/v2-roadmap.html>_.

Added support for HTTPS proxies contacting HTTPS servers (Pull #1923, Pull #1806)

Deprecated negotiating TLSv1 and TLSv1.1 by default. Users that still wish to use TLS earlier than 1.2 without a deprecation warning

... (truncated)

Commits

d161647 Release 1.26.5
2d4a3fe Improve performance of sub-authority splitting in URL
2698537 Update vendored six to 1.16.0
07bed79 Fix deprecation warnings for Python 3.10 ssl module
d725a9b Add Python 3.10 to GitHub Actions
339ad34 Use pytest==6.2.4 on Python 3.10+
f271c9c Apply latest Black formatting
1884878 [1.26] Properly proxy EOF on the SSLTransport test suite
a891304 Release 1.26.4
8d65ea1 Merge pull request from GHSA-5phf-pp7p-vc2r
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump mistune from 0.8.4 to 2.0.3

opened on 2022-07-29 23:34:25 by dependabot[bot]

Bumps mistune from 0.8.4 to 2.0.3.

Release notes

Sourced from mistune's releases.

Version 2.0.2

Fix escape_url via lepture/mistune#295

Version 2.0.1

Fix XSS for image link syntax.

Version 2.0.0

First release of Mistune v2.

Version 2.0.0 RC1

In this release, we have a Security Fix for harmful links.

Version 2.0.0 Alpha 1

This is the first release of v2. An alpha version for users to have a preview of the new mistune.

Changelog

Sourced from mistune's changelog.

Changelog

Here is the full history of mistune v2.

Version 2.0.4
Released on Jul 15, 2022

Fix url plugin in &lt;a&gt; tag
Fix * formatting

Version 2.0.3
Released on Jun 27, 2022

Fix table plugin

Security fix for CVE-2022-34749

Version 2.0.2
Released on Jan 14, 2022
Fix escape_url
Version 2.0.1
Released on Dec 30, 2021

XSS fix for image link syntax.

Version 2.0.0
Released on Dec 5, 2021
This is the first non-alpha release of mistune v2.
Version 2.0.0rc1
Released on Feb 16, 2021

Version 2.0.0a6
</tr></table> 

... (truncated)

Commits

3f422f1 Version bump 2.0.3
a6d4321 Fix asteris emphasis regex CVE-2022-34749
5638e46 Merge pull request #307 from jieter/patch-1
0eba471 Fix typo in guide.rst
61e9337 Fix table plugin
76dec68 Add documentation for renderer heading when TOC enabled
799cd11 Version bump 2.0.2
babb0cf Merge pull request #295 from dairiki/bug.escape_url
fc2cd53 Make mistune.util.escape_url less aggressive
3e8d352 Version bump 2.0.1
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump ipython from 7.16.1 to 7.16.3

opened on 2022-01-21 20:49:13 by dependabot[bot]

Bumps ipython from 7.16.1 to 7.16.3.

Commits

d43c7c7 release 7.16.3
5fa1e40 Merge pull request from GHSA-pq7m-3gw7-gq5x
8df8971 back to dev
9f477b7 release 7.16.2
138f266 bring back release helper from master branch
5aa3634 Merge pull request #13341 from meeseeksmachine/auto-backport-of-pr-13335-on-7...
bcae8e0 Backport PR #13335: What's new 7.16.2
8fcdcd3 Pin Jedi to <0.17.2.
See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

Bump pillow from 7.2.0 to 8.3.2

opened on 2021-09-08 02:33:31 by dependabot[bot]

Bumps pillow from 7.2.0 to 8.3.2.

Release notes

Sourced from pillow's releases.

8.3.2

https://pillow.readthedocs.io/en/stable/releasenotes/8.3.2.html

Security

CVE-2021-23437 Raise ValueError if color specifier is too long [hugovk, radarhere]

Fix 6-byte OOB read in FliDecode [wiredfool]

Python 3.10 wheels

Add support for Python 3.10 #5569, #5570 [hugovk, radarhere]

Fixed regressions

Ensure TIFF RowsPerStrip is multiple of 8 for JPEG compression #5588 [kmilos, radarhere]

Updates for ImagePalette channel order #5599 [radarhere]

Hide FriBiDi shim symbols to avoid conflict with real FriBiDi library #5651 [nulano]

8.3.1

https://pillow.readthedocs.io/en/stable/releasenotes/8.3.1.html

Changes

Catch OSError when checking if fp is sys.stdout #5585 [@radarhere]

Handle removing orientation from alternate types of EXIF data #5584 [@radarhere]

Make Image.array take optional dtype argument #5572 [@t-vi]

8.3.0

https://pillow.readthedocs.io/en/stable/releasenotes/8.3.0.html

Changes

Use snprintf instead of sprintf #5567 [@radarhere]

Limit TIFF strip size when saving with LibTIFF #5514 [@kmilos]

Allow ICNS save on all operating systems #4526 [@newpanjing]

De-zigzag JPEG's DQT when loading; deprecate convert_dict_qtables #4989 [@gofr]

Do not use background or transparency index for new color #5564 [@radarhere]

Simplified code #5315 [@radarhere]

Replaced xml.etree.ElementTree #5565 [@radarhere]

... (truncated)

Changelog

Sourced from pillow's changelog.

8.3.2 (2021-09-02)

CVE-2021-23437 Raise ValueError if color specifier is too long [hugovk, radarhere]

Fix 6-byte OOB read in FliDecode [wiredfool]

Add support for Python 3.10 #5569, #5570 [hugovk, radarhere]

Ensure TIFF RowsPerStrip is multiple of 8 for JPEG compression #5588 [kmilos, radarhere]

Updates for ImagePalette channel order #5599 [radarhere]

Hide FriBiDi shim symbols to avoid conflict with real FriBiDi library #5651 [nulano]

8.3.1 (2021-07-06)

Catch OSError when checking if fp is sys.stdout #5585 [radarhere]

Handle removing orientation from alternate types of EXIF data #5584 [radarhere]

Make Image.array take optional dtype argument #5572 [t-vi, radarhere]

8.3.0 (2021-07-01)

Use snprintf instead of sprintf. CVE-2021-34552 #5567 [radarhere]

Limit TIFF strip size when saving with LibTIFF #5514 [kmilos]

Allow ICNS save on all operating systems #4526 [baletu, radarhere, newpanjing, hugovk]

De-zigzag JPEG's DQT when loading; deprecate convert_dict_qtables #4989 [gofr, radarhere]

Replaced xml.etree.ElementTree #5565 [radarhere]

... (truncated)

Commits

8013f13 8.3.2 version bump
23c7ca8 Update CHANGES.rst
8450366 Update release notes
a0afe89 Update test case
9e08eb8 Raise ValueError if color specifier is too long
bd5cf7d FLI tests for Oss-fuzz crash.
94a0cf1 Fix 6-byte OOB read in FliDecode
cece64f Add 8.3.2 (2021-09-02) [CI skip]
e422386 Add release notes for Pillow 8.3.2
08dcbb8 Pillow 8.3.2 supports Python 3.10 [ci skip]
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

rr-learning

GitHub Repository