Error when downloading dataset

Hello,

I’m trying to download minerl dataset, but I keep receiving errors like the following one

ERROR:root:Error - HTTPSConnectionPool(host=‘minerl.s3.amazonaws.com’, port=443): Read timed out.
ERROR:root:Header from download attempt of https://minerl.s3.amazonaws.com/v3/data_texture_0_low_res.tar: {‘x-amz-id-2’: ‘N+3ju+t3ckRz+EbA36wQRlOXjuizVBvBJX1iab97sMxNjBbF1Z4tv0Q8knTtTLOWJQUN7UugMbE=’, ‘x-amz-request-id’: ‘51D21B5C1E9E825D’, ‘Date’: ‘Wed, 18 Nov 2020 11:37:23 GMT’, ‘x-amz-replication-status’: ‘COMPLETED’, ‘Last-Modified’: ‘Tue, 16 Jun 2020 12:24:29 GMT’, ‘ETag’: ‘“4bf36a445e836ff163b9d6e738d691f9-7758”’, ‘x-amz-version-id’: ‘eOnRrCN.6nfceKuVhqGSRL8aV5BlMBB1’, ‘Accept-Ranges’: ‘bytes’, ‘Content-Type’: ‘application/x-tar’, ‘Content-Length’: ‘65075773440’, ‘Server’: ‘AmazonS3’}
ERROR:minerl.data.download:IO error encountered when downloading - please try again
ERROR:minerl.data.download:None

Is there another way to download the dataset? BTW, I’m in China.

View post 0

Is there any team that still has a seat?

Hi everyone.

I just finished my competition on Procgen and ranked 3rd place in round 2. I’d like to participate in MineRL too but I don’t have many resources for this competition – I get one GPU and $300+ AWS credit. Also, I think working with people would be a lot more interesting than working alone! Therefore, I want to ask and see if any team still open to new members.

I wish to join a team that’s active and eager to improve its agent and I promise that I’ll do my best in the rest time of the competition.

View post 0

Is it a preknowledge that we should select one of existing submissions for the final evaluation?

Hi @dipam_chakraborty

The final evaluation evaluates generalization, but I did not use any regularizations such as batch normalization and data augmentations in my previous submissions. Also, in my latest few submissions, I chose to experiment with a newly introduced hyperparameter instead of using the one that performed well on my local machine.

View post 0

Is it a preknowledge that we should select one of existing submissions for the final evaluation?

Hi @vrv

Thank you for the response. Yeah, I know that was my bad after reviewing thoroughly the overview page and the answer I linked before. However, we did not always follow these, right? For example, we used 6 public and 4 private test environments in the second round instead of 4 and 1 described on the overview page. Also, this answer said we got to pick 3 submissions but at the end of the day, we only pick one.

Maybe I should ask this before instead of wishfully thinking a new submission is viable. At this point, I don’t how which submission I should use as I said before, none of them were made for the final evaluation.

The purpose I posted this was to see if there was someone else facing a similar situation. If I’m the only one, I’ll accept it.

Although some of the above words may seem like complaining, I am not meant to. I’ve learned a lot during the competition and received a lot of help from you guys. Thank you all.

View post 1

Is it a preknowledge that we should select one of existing submissions for the final evaluation?

Hi everyone,

I’m wondering if I’m the only one that learns we should select one of the previous submissions for the final evaluation. I cannot find any official statement about this and the only clue I can find now is this answer, which I’ve previously read but not paid much attention to the word “existing”. That was a mistake of mine but I humbly don’t think such an answer in the forum could be counted as a formal statement.

It’s really frustrating to learn this at this point as none of my previous solutions was prepared for the final evaluations. I thought the challenge was to find a good solution but in the end, I found myself trapped in some word game. I am not meant to complain as I definitely should be responsible for the above mistake. However, if anyone feels the same way, please say something. Maybe, together we can make the game more interesting.

View post 0

Is it possible to run the evaluation on all environments in the final week of round 2?

In the final week of round 2, is it possible to run the evaluation on all environments? To reduce the computation cost, maybe we can reduce the submission quota a bit.

View post 0

Number of environments in round 2

I am humbly against Feiyang. Not everyone gets that amount of computation resources to try out their ideas. A decent amount of daily limits on submissions are helpful.

On the other hand, I agree with @jurgisp and @quang_tran that we should relax the 2-hour limit. I think this makes the competition bias to on-policy algorithms. On-policy algorithms can take advantage of large batches and therefore use less training iterations. But off-policy algorithms usually work with a much smaller batch size and require more training iterations and more time to train.

View post 1

Round 2 is open for submissions 🚀

Hi @jyotish

What’s the use of the blind reward?

View post 1

Running the evaluation worker during evaluations is now optional

Thank you @jyotish , I see now.

View post 0

Running the evaluation worker during evaluations is now optional

Hi @jyotish

How should I change train.py to disable the evaluation worker locally?

View post 0

Inform the agent a new episode starts

Is there a way to inform the agent that a new episode starts when defining trainer using build_trainer?

View post 0

TF2 is by default not enabled?

I see. But why should it be restricted to TF1.x even though I’ve set framework=tfe in the yaml file?

View post 0

TF2 is by default not enabled?

Adding the following code to this line of train.py

from tensorflow.python import tf2
print(tf2.enabled())
assert False

it prints False. Is there any way to enable tf2?

View post 0

About Ray trainer and workers

This file may contain something what you’re looking for

View post 1

Rllib custom env

Hi @jyotish

Could you please answer my above question?

If we use frameskip, does the framework count the number of frames in the right way?
For example, if we use frame_skip=2 , the number of interactions between the agent and environment is 8e6/2=4e6 when using only 8M frames. If we use the standard configuration which set timesteps_total=8000000 , will this stop correctly?

View post 0

Rllib custom env

If we use frameskip, does the framework count the number of frames in the right way?

For example, if we use frame_skip=2, the number of interactions between the agent and environment is 8e6/2=4e6 when using only 8M frames. If we use the standard configuration which set timesteps_total=8000000, will this stop correctly?

View post 0

Unusually large tensor in the starter code

Thank you so much and sorry for the late response.

View post 0

Unusually large tensor in the starter code

I receive OOM error when running the starter code. Here’s the error message I receive

== Status ==
Memory usage on this node: 3.6/62.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 7/10 CPUs, 0.9/1 GPUs, 0.0/13.96 GiB heap, 0.0/6.4 GiB objects
Result logdir: /home/aptx4869/ray_results/procgen-ppo
Number of trials: 1 (1 RUNNING)
±------------------------------±---------±------+
| Trial name | status | loc |
|-------------------------------±---------±------|
| PPO_procgen_env_wrapper_00000 | RUNNING | |
±------------------------------±---------±------+

(pid=5272) 2020-06-24 09:26:36,869 INFO trainer.py:421 – Tip: set ‘eager’: true or the --eager flag to enable TensorFlow eager execution
(pid=5272) 2020-06-24 09:26:36,870 INFO trainer.py:580 – Current log_level is WARN. For more information, set ‘log_level’: ‘INFO’ / ‘DEBUG’ or use the -v and -vv flags.
(pid=5272) 2020-06-24 09:26:44,889 INFO trainable.py:217 – Getting current IP.
(pid=5272) 2020-06-24 09:26:44,889 WARNING util.py:37 – Install gputil for GPU system monitoring.
Traceback (most recent call last):
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/tune/trial_runner.py”, line 467, in _process_trial
result = self.trial_executor.fetch_result(trial)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py”, line 431, in fetch_result
result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/worker.py”, line 1515, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ResourceExhaustedError): ray::PPO.train() (pid=21577, ip=192.168.1.102)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1350, in _run_fn
target_list, run_metadata)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,32,32,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node default_policy_1/tower_1/gradients_1/default_policy_1/tower_1/model_1/max_pooling2d_1/MaxPool_grad/MaxPoolGrad}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

ray::PPO.train() (pid=21577, ip=192.168.1.102)
File “python/ray/_raylet.pyx”, line 459, in ray._raylet.execute_task
File “python/ray/_raylet.pyx”, line 462, in ray._raylet.execute_task
File “python/ray/_raylet.pyx”, line 463, in ray._raylet.execute_task
File “python/ray/_raylet.pyx”, line 417, in ray._raylet.execute_task.function_executor
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/rllib/agents/trainer.py”, line 498, in train
raise e
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/rllib/agents/trainer.py”, line 484, in train
result = Trainable.train(self)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/tune/trainable.py”, line 261, in train
result = self._train()
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py”, line 151, in _train
fetches = self.optimizer.step()
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/rllib/optimizers/multi_gpu_optimizer.py”, line 212, in step
self.per_device_batch_size)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/rllib/optimizers/multi_gpu_impl.py”, line 257, in optimize
return sess.run(fetches, feed_dict=feed_dict)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 958, in run
run_metadata_ptr)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1181, in _run
feed_dict_tensor, options, run_metadata)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1359, in _do_run
run_metadata)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,32,32,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node default_policy_1/tower_1/gradients_1/default_policy_1/tower_1/model_1/max_pooling2d_1/MaxPool_grad/MaxPoolGrad (defined at /.conda/envs/procgen/lib/python3.7/site-packages/ray/rllib/agents/ppo/ppo_tf_policy.py:195) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
…
== Status ==
Memory usage on this node: 17.9/62.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/12 CPUs, 0.0/1 GPUs, 0.0/13.96 GiB heap, 0.0/6.4 GiB objects
Result logdir: /home/aptx4869/ray_results/procgen-ppo
Number of trials: 1 (1 ERROR)
±------------------------------±---------±------+
| Trial name | status | loc |
|-------------------------------±---------±------|
| PPO_procgen_env_wrapper_00000 | ERROR | |
±------------------------------±---------±------+
Number of errored trials: 1
±------------------------------±-------------±-------------------------------------------------------------------------------------------------------+
| Trial name | # failures | error file |
|-------------------------------±-------------±-------------------------------------------------------------------------------------------------------|
| PPO_procgen_env_wrapper_00000 | 1 | /home/aptx4869/ray_results/procgen-ppo/PPO_procgen_env_wrapper_0_2020-06-24_09-18-39m1tpdxon/error.txt |
±------------------------------±-------------±-------------------------------------------------------------------------------------------------------+

== Status ==
Memory usage on this node: 17.9/62.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/12 CPUs, 0.0/1 GPUs, 0.0/13.96 GiB heap, 0.0/6.4 GiB objects
Result logdir: /home/aptx4869/ray_results/procgen-ppo
Number of trials: 1 (1 ERROR)
±------------------------------±---------±------+
| Trial name | status | loc |
|-------------------------------±---------±------|
| PPO_procgen_env_wrapper_00000 | ERROR | |
±------------------------------±---------±------+
Number of errored trials: 1
±------------------------------±-------------±-------------------------------------------------------------------------------------------------------+
| Trial name | # failures | error file |
|-------------------------------±-------------±-------------------------------------------------------------------------------------------------------|
| PPO_procgen_env_wrapper_00000 | 1 | /home/aptx4869/ray_results/procgen-ppo/PPO_procgen_env_wrapper_0_2020-06-24_09-18-39m1tpdxon/error.txt |
±------------------------------±-------------±-------------------------------------------------------------------------------------------------------+

Traceback (most recent call last):
File “train.py”, line 235, in
run(args, parser)
File “train.py”, line 229, in run
concurrent=True)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/tune/tune.py”, line 411, in run_experiments
return_trials=True)
File “/home/aptx4869/.conda/envs/procgen/lib/python3.7/site-packages/ray/tune/tune.py”, line 347, in run
raise TuneError(“Trials did not complete”, incomplete_trials)
ray.tune.error.TuneError: (‘Trials did not complete’, [PPO_procgen_env_wrapper_00000])

It seems that error occurs because of the unusually large tensor with shape[2048, 32, 32,32] but I have no idea where it comes from. My GPU has 12G memory. The only thing I change is the run.sh file, in which I increase the memory and CPU used by ray:

  export RAY_MEMORY_LIMIT=15000000000
  export RAY_CPUS=12
  export RAY_STORE_MEMORY=10000000000

View post 0

Several questions about the competition

Thanks for the explanations

View post 0

Several questions about the competition

Hi @jyotish and @mohanty. If we set “num_levels=0” for training, which configuration should we use for evaluation on our local machine? And I’m wondering if the difficulty is set to “easy” throughout the competition?

View post 0