Benchmark class
Generic benchmark class for evaluating agents in environments from config.
Methods
- def evaluate(self, agent: agent.Agent, num_episodes: typing.Optional[int] = None) -> typing.Dict[str, float]
- Evaluates the provide agent in the configured environment either locally or remotely and returns the results dictionary with metrics.
- def local_evaluate(self, agent: agent.Agent, num_episodes: typing.Optional[int] = None) -> typing.Dict[str, float]
- Run evaluation of an Agent in the Env locally.
- def remote_evaluate(self, agent: agent.Agent) -> typing.Dict[str, float]
- Run remote evaluation with evalai for the instantiated Agent and Env. Runs remotely through a challenge evaluation server to prevent any potential for biased results. Imports come challenge-specific dependencies.
Special methods
- def __init__(self, config_paths: typing.Optional[str] = None, eval_remote: bool = False) -> None
- Initialize the Env from the provided config.
Method documentation
def habitat.
Evaluates the provide agent in the configured environment either locally or remotely and returns the results dictionary with metrics.
| Parameters | |
|---|---|
| agent | agent to be evaluated in environment. |
| num_episodes | count of number of episodes for which the evaluation should be run. |
| Returns | dict containing metrics tracked by environment. |
def habitat.
Run evaluation of an Agent in the Env locally.
| Parameters | |
|---|---|
| agent | The Agent to evaluate. |
| num_episodes | The number of episodes to evaluate. |
| Returns | The results dictionary containing metrics. |
def habitat.
Run remote evaluation with evalai for the instantiated Agent and Env. Runs remotely through a challenge evaluation server to prevent any potential for biased results. Imports come challenge-specific dependencies.
| Parameters | |
|---|---|
| agent | The Agent to evaluate. |
| Returns | The results dictionary containing metrics. |
def habitat.
Initialize the Env from the provided config.
| Parameters | |
|---|---|
| config_paths | file to be used for creating the environment |
| eval_remote | boolean indicating whether evaluation should be run remotely or locally |