Even faster batch processing (10-20 min for all scenarios using Google Cloud)

CommonRoad-Search
1

Dear all,

If you use Google Cloud free trial, you can get all scenarios evaluated in at most 20 minutes, or faster.

The steps are (15-30 minutes to configure).

  1. Subscribe to free Google Cloud trial for a year (a credit card required, but it costs you nothing).
  2. Create a preemptible instance using an N2 machine, 12 vCPUs and 16 GB RAM, Ubuntu 18.04.
  3. Allow connections to tcp 9000 port, as shown on a screenshot here: https://towardsdatascience.com/running-jupyter-notebook-in-google-cloud-platform-in-15-min-61e16da34d52
  4. Connect using ssh (using web interface).
  5. sudo apt get update
  6. Run the commands from GitHub - tom-doerr/commonroad-docker to get everything up and running (2 minutes).
  7. Connect to the public ip of your cloud instance and port 9000 using your web browser.
  8. In the Jupyter Notebook, upload the multicore batch processing notebook to the folder with the original batch processing notebook.

  1. Set NUM_PROCESSES to 12.

  2. Run and enjoy.

On 12 cores it takes at most 10 seconds instead of 120 seconds per processed scenario.

If 20 minutes is still too long, create a second VM instance and leave one half of scenarios for it, then it’ll take at most 10 minutes to evaluate all scenarios.

You can use the Jupyter web interface to edit your files. All your files are always preserved, unlike with Colaboratory.

Don’t forget to shut down your virtual machine, or you could run out of your free credit. You’d use 11 cents per hour out of the $268 free credit if you choose this solution.

I can see on Moodle that the submission deadline was extended by one week.

7 Likes
2

Thank you very much! That really works!

3

Thanks. It really works, but how can I download solutions from compute engine to my computer?

4

Hi!

First of all, Google Cloud Trial now allows only 8 N1 vCPUs. If you can not boot your machine with 12 vCPUs, do the following to change the number of vCPUs (5 minutes).

Changing CPU type of your virtual machine

Create a snapshot of your disk. It can be done under the Snapshopts tab of the Compute Engine.

Create a new VM with 8 N1 vCPUs. Choose the newly created Snapshot as the Boot disk.

Boot up to the new machine using SSH. All your files and applications are preserved.

Changing the CPU type of the Ubuntu 18.04 VM worked for me without any problems using the above-mentioned method.

In case changing CPU type did not work for your VM OS, you could also follow the instructions from https://stackoverflow.com/questions/53527063/how-do-i-get-files-from-gcp-vm to create a new VM and mount the old VM disk snapshot as a second HDD.

Downloading data from a working virtual machine

In the SSH console, zip the solutions folder.

To download files, simply click the Settings icon on the top right of the SSH window (using the standard Google SSH web client) and choose Download.

If you might encounter any problems, let me know. Don’t forget to power off your VM instance.

1 Like