habitat.Benchmark

Implements evaluation of habitat.Agent inside habitat.Env. habitat.Benchmark creates a habitat.Env which is specified through the config_env parameter in constructor. The evaluation is task agnostic and is implemented through metrics defined for habitat.EmbodiedTask.

class habitat.core.benchmark.Benchmark(config_paths: Optional[str] = None)[source]

Benchmark for evaluating agents in environments.

Parameters
  • config_paths – file to be used for creating the environment.

  • config_dir – directory where config_paths is located.

evaluate(agent: habitat.core.agent.Agent, num_episodes: Optional[int] = None) → Dict[str, float][source]
Parameters
  • agent – agent to be evaluated in environment.

  • num_episodes – count of number of episodes for which the evaluation should be run.

Returns

dict containing metrics tracked by environment.