rr-learning, updated 🕥 2022-01-22 20:42:25

MPI3D Disentanglement Datasets

MPI3D datasets have been introduced to benchmark representations learning algorithms across simulated and real-world environments. The first transfer learning results of unsupervised disentangled representations are presented in our NeurIPS 2019 paper.

UPDATE: We have now added real-world complex dataset. The dataset contains 3D printed replicas of real-world objects such coffee cup, tennis ball, croissant, beer cup. Find more details on the factors of variations below.

The dataset is also used in the NeurIPS Disentanglement Challenge. If you use this dataset in your work then kindly cite us. ``` @inproceedings{NEURIPS2019_d97d404b, author = {Gondal, Muhammad Waleed and Wuthrich, Manuel and Miladinovic, Djordje and Locatello, Francesco and Breidt, Martin and Volchkov, Valentin and Akpo, Joel and Bachem, Olivier and Sch\"{o}lkopf, Bernhard and Bauer, Stefan}, booktitle = {Advances in Neural Information Processing Systems}, editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett}, pages = {}, publisher = {Curran Associates, Inc.}, title = {On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset}, url = {https://proceedings.neurips.cc/paper/2019/file/d97d404b6119214e4a7018391195240a-Paper.pdf}, volume = {32}, year = {2019} }


Datasets Information

There are following four different datasets. The gifs are created using disentanglement_lib visualization tool.

1. Real world simple shapes (mpi3d_real).

2. Realistic rendered images (mpi3d_realistic).

3. Simplistic rendered images (mpi3d_toy).

4. Complex real world shapes (mpi3d_complex).

The first three datasets consists of 1,036,800 images, corresponding to all the possible combinations of the following factors of variation:

|Factors|Possible Values| |---|---| |object_color|white=0, green=1, red=2, blue=3, brown=4, olive=5| |object_shape|cone=0, cube=1, cylinder=2, hexagonal=3, pyramid=4, sphere=5| |object_size|small=0, large=1| |camera_height|top=0, center=1, bottom=2| |background_color|purple=0, sea green=1, salmon=2| |horizontal_axis|0,...,39| |vertical_axis|0,...,39|

The real-world complex dataset consists of 460,800 images, containing the combinations of the following factors of variations.

|Factors|Possible Values| |---|---| |object_color|yellow=0, green=1, olive=2, red=3| |object_shape|coffee-cup=0, tennis-ball=1, croissant=2, beer-cup=3| |object_size|small=0, large=1| |camera_height|top=0, center=1, bottom=2| |background_color|purple=0, sea green=1, salmon=2| |horizontal_axis|0,...,39| |vertical_axis|0,...,39|

So far we only provide the datasets in 64x64 resolution. Higher resolution versions will be made available in the near future.

Reading the Datasets

The datasets are provided in the form of numpy arrays. Once the data is loaded, you can use array.reshape([6,6,2,3,3,40,40,64,64,3]) to obtain an array where the first 7 dimensions corresponds to data generative factors as in the table above and the last three to the image dimensions. Note that for real-world complex dataset you need to use array.reshape([4,4,2,3,3,40,40,64,64,3]).

``` import numpy as np data = np.load('./mpi3d_real.npz')['images']

To visualize each factor of the data independently, you can reshape

the array as the following.

data = data.reshape([6,6,2,3,3,40,40,64,64,3])

For real-world complex dataset use:

data = data.reshape([4,4,2,3,3,40,40,64,64,3]) ```


Use the following links to download the datasets.

  1. mpi3d_toy (simplistic rendered): link
  2. mpi3d_realistic (realistic rendered): link
  3. mpi3d_real (real-world images): link
  4. mpi3d_real_complex (real-world complex shapes images) : link


Please send any feedback to [email protected]


This work is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/).


releasing plan for higher resolution dataset

opened on 2020-06-03 06:08:49 by XudongOliverShen


Great work! just wondering when will the higher resolution version be released?