Habitat Rearrangement Challenge 2022

Overview

For NeurIPS 2022, we are hosting the Object Rearrangement challenge in the Habitat 2.0 simulator 1. Object rearrangement focuses on mobile manipulation, low-level control, and task planning.

For details on how to participate, submit and train agents refer to github.com/facebookresearch/habitat-challenge/tree/rearrangement-challenge-2022 repository.

Task: Object Rearrangement

In the object rearrangement task, a Fetch robot is randomly spawned in an unseen environment and asked to rearrange a list of objects from initial to desired positions – picking/placing objects from receptacles (counter, sink, sofa, table), opening/closing containers (drawers, fridges) as necessary. The task is communicated to the robot using the GeometricGoal specification which provides the initial 3D center-of-mass position of each target object to be rearranged along with the desired 3D center-of-mass position for that object. An episode is successful if all target objects are within 15cm of their desired positions (without considering orientation).

The Fetch robot is equipped with an egocentric 256x256 90-degree FoV RGBD camera on the robot head. A camera with the same specifications is also mounted to the robot arm. The agent also has access to idealized base-egomotion giving the relative displacement and angle of the base since the start of the episode. Additionally, the robot has proprioceptive joint sensing giving the current angles of the robot joints.

There are two tracks in the Habitat Rearrangement Challenge.

  • Track 1: 1-rearrange: The agent must rearrange 1 object. Furthermore, all containers (such as the fridge, cabinets, and drawers) start open, meaning the agent never needs to open containers to access objects or goals. 1-rearrange therefore always requires the same sequence of navigation to the object, picking the object, navigating to the goal, and then placing the object at the goal, thus no task planning is needed. The maximum episode length is 1500 time steps.
  • Track 2: 2-rearrange: The agent must rearrange 2 objects. The order the agent rearranges the objects can matter. If the rearrangement goal for one object is stacked on top of another object, then the agent must first rearrange the object on the bottom. Furthermore, due to object potentially starting in closed receptacles, the agent may need to perform intermediate actions to access the object. For example, an apple may start in a closed fridge and have a goal position on the table. To rearrange the apple, the agent first needs to open the fridge before picking the apple. The agent is not provided task information about if these intermediate open actions need to be executed. It also is not provided information about if two objects need to be stacked. All of this information needs to be inferred from the egocentric observations and goal specification. The maximum episode length is 5000 time steps.

Dataset

The rearrangement challenge provides 50,000 rearrangement problem instances for training and 1,000 for validation for both of the tracks. These episodes will consist of scenes from the ReplicaCAD dataset. 21 of the 105 scenes in the ReplicaCAD dataset 1 are held out for the testing dataset of 1,000 episodes.

Evaluation

The primary evaluation criteria of the agents is the Overall Success. We also include additional metrics to diagnose agent efficiency and partial progress.

  • Overall Success: If all target objects are placed within 15cm of the goal position for the object. This is the primary metric used to judge agents the value of different agents.
  • Object Success: The ratio of objects correctly rearranged. This is only relevant for the 2-rearrange track where there is more than one object to rearrange.
  • Rearrangement Progress: The change in relative Euclidean distance of the object start to the goal from the start to the end of the episode.
  • Time Taken: The amount of time (in seconds) needed to solve the episode.

Participation Guidelines

Participate in the contest by registering on the EvalAI challenge page [Soon to come!] and creating a team. Participants will upload docker containers with their agents that are evaluated on an AWS GPU-enabled instance. Before pushing the submissions for remote evaluation, participants should test the submission docker locally to make sure it is working. Instructions for training, local evaluation, and online submission are provided below.

Valid challenge phases are habitat-rearrangement-{minival, test-standard, test-challenge}.

The challenge consists of the following phases:

  1. Minival phase: The purpose of this phase is sanity checking — to confirm that our remote evaluation reports the same result as the one you’re seeing locally. Each team is allowed a maximum of 100 submissions per day for this phase, but we will block and disqualify teams that spam our servers.
  2. Test Standard phase: The purpose of this phase/split is to serve as the public leaderboard establishing the state of the art; this is what should be used to report results in papers. Each team is allowed a maximum of 10 submissions per day for this phase, but again, please use them judiciously. Don’t overfit to the test set.
  3. Test Challenge phase: This split will be used to decide challenge winners. Each team is allowed a total of 5 submissions until the end of the challenge submission phase. The highest performing of these 5 will be automatically chosen. Results on this split will not be made public until the announcement of the final results at the NeurIPS 2022 competition.

Note: Your agent will be evaluated on 1000 episodes and will have a total available time of 48 hours to finish. Your submissions will be evaluated on AWS EC2 p2.xlarge instance which has a Tesla K80 GPU (12 GB Memory), 4 CPU cores, and 61 GB RAM. If you need more time/resources for the evaluation of your submission please get in touch.

Citing Habitat Rearrangement Challenge 2022

@misc{habitatrearrangechallenge2022,
  title         =     Habitat Rearrangement Challenge 2022,
  author        =     {Andrew Szot, Karmesh Yadav, Alex Clegg, Vincent-Pierre Berges, Aaron Gokaslan, Angel Chang, Manolis Savva, Zsolt Kira, Dhruv Batra},
  howpublished  =     {\url{https://aihabitat.org/challenge/rearrange_2022}},
  year          =     {2022}
}

Acknowledgments

The Habitat challenge would not have been possible without the infrastructure and support of EvalAI team.

References

1.
^ a b Habitat 2.0: Training Home Assistants to Rearrange their Habitat. Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra. NeurIPS, 2021

Dates

Challenge starts Jul 15, 2022
Test Phase Submissions Open Aug 1, 2022
Challenge submission deadline Oct 20, 2022

Organizer

Facebook AI Research

Participation

For details on how to participate, submit , and train agents, refer to github.com/facebookresearch/habitat-challenge/tree/rearrangement-challenge-2022 repository.

Please note that the latest submission to the test-challenge split will be used for final evaluation.

For updates on the competition, join the rearrange-challenge-neurips2022 Habitat Slack channel via this link.

Sponsors

Facebook AI Research