Is it ok to directly wrap the gym env in `create_single_env` in local_evaluation.py when evaluating the model in aicrowd?

Hi, sorry to bother you again.
What’s the difference between the local evaluator and the actual one? It seems that the performance evaluated in the actual has a huge drop than in local. (LIMIT_TASKS has been changed to None in LocalEvalConfig)

View post 0

Is it ok to directly wrap the gym env in `create_single_env` in local_evaluation.py when evaluating the model in aicrowd?

Hi, is it should be the observations (not observations_agent) here?

View post 1

Is it ok to directly wrap the gym env in `create_single_env` in local_evaluation.py when evaluating the model in aicrowd?

The done is always reset to False when one episode ends in evaluate in local_evalutor.py.

View post 0

Is it ok to directly wrap the gym env in `create_single_env` in local_evaluation.py when evaluating the model in aicrowd?

Thanks for your reply.
Some feature processing methods like stacking the recent k frames need to clear some historical data when the env is reset. However, in the current evaluation framework, it is hard to do this. This problem can be solved if another parameter, such as a bool variable is_first_obs indicating whether the env is reset, is allowed to be passed into agent.act.

View post 0

Is the `agentPos` and `grid` available during evaluation in aicrowd?

It seems that in local_evaluation.py the agentPos and grid are available because vector_state=True passed into gridworld. However, these two keys are missed when I submitted my code to the aicrowd:

Traceback (most recent call last):
  File "client_launcher.py", line 20, in <module>
    start_test_client()
  File "client_launcher.py", line 17, in start_test_client
    client.run_agent()
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 198, in run_agent
    kwargs=kwargs,
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 100, in process_request
    action, target_attribute, instance_id, args, kwargs
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 129, in route_agent_request
    return self.execute(target_attribute, *args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 142, in execute
    return method(*args, **kwargs)
  File "/home/aicrowd/agents/aicrowd_wrapper.py", line 39, in register_reset
    return self.compute_action(observation)
  File "/home/aicrowd/agents/aicrowd_wrapper.py", line 43, in compute_action
    return self.agent.act(*observation,)
  File "/home/aicrowd/agents/vector_agent.py", line 29, in act
    action, early_termination = agent.act( obs, rew, done, info)
  File "/home/aicrowd/agents/iglu_agent.py", line 327, in act
    if (observation['agentPos'] == 0).all():
KeyError: 'agentPos'

View post 0

Is it ok to directly wrap the gym env in `create_single_env` in local_evaluation.py when evaluating the model in aicrowd?

If I need to process the obs and reset some data when env.reset() is called, is it ok to directly wrap the env with my own-defined wrapper in create_single_env function in local_evaluation.py?

View post 0