The Household Activities from Virtual Environments Data Set consist of 240 recordings in virtual reality, done at the Automatica Trade Fair 2018 in Munich. Recordings are split among the three household scenarios: "setting the table", "washing the dishes" and "cleaning the living room". The data set is of exploratory nature and each scenario name serves as the sole instruction the participants received (e.g. "please set the table"), in order to collect a wide variety of styles. Therefore, the data set contains various execution styles for each task, e.g. setting the table for one or two persons, or throwing items out the window.

We provide multiple files per recording:

  • First-person video with floating object labels
  • Third-person video with floating object labels
  • CSV containing object and user trajectories
  • CSV containing grasp events


  • v1.0.0 Initial release


If you have requests or comments regarding the dataset, please refer to the issue tracker:


The HAVE Dataset and provided models are licensed under the Apache 2.0 license


A sample recording can be downloaded here (312 Mb): [31_set_the_table]

The full data set can be downloaded here:

[set_the_table] (19Gb)

[do_the_dishes] (20Gb)

[clean_the_living_room] (13Gb)

[environment_files] (1.5Mb .fbx format)


If you are using this dataset in your research, please cite the following paper:

C. Uhde, N. Berberich, K. Ramirez-Amaro, and G. Cheng, “The Robot as Scientist: Using Mental Simulation to Test Causal Hypotheses Extracted from Human Activities in Virtual Reality,” p. 6.

    title = {The {Robot} as {Scientist}: {Using} {Mental} {Simulation} to {Test} {Causal} {Hypotheses} {Extracted} from {Human} {Activities} in {Virtual} {Reality}},
    booktitle = {IROS 2020},
    author = {Uhde, Constantin and Berberich, Nicolas and Ramirez-Amaro, Karinne and Cheng, Gordon},
    year = {2020},
    address={Las Vegas, USA}

Related Papers:

Ramirez-Amaro, K., Uhde, C., Bates, T., & Cheng, G. (2019). A Benchmarking Dataset for Automatic Symbolic Grounding from Virtual Demonstrations. In 2nd International Workshop on Computational Models of Affordance in Robotics at the IEEE International Conference on Robotics and Automation (ICRA).


Constantin Uhde