Compute Tutorial

Watch the tutorial video at YouTube.

(0:24) Start at your AWS console home page. You can access the IAM account control panel by either searching for it like so or by using the pull-down menu (0:30)

and clicking on the link.

(0:36)

Click users, then click add user. (0:40)

Give your new user a name – mine will be numerai_train. Under access type, click programmatic access, then click next. (0:50)

Now click attach existing policies directly, and click the first box to grant administrator access. Click next. (0:58)

You don’t need to add tags, so click next again. One final click creates your new user. The new user has been created, and the page displays your public key and hides the secret key. I’m going to download the CSV so I can store my numerai API keys as well. (1:08)

Next, open your numerai account settings page and click Add. (1:31)

Name your key, and check “upload submissions”, “make stakes”, “view historical submission info” and “view user info”. Type in your password, then, click generate key. (1:48)

Copy the secret key which is displayed in the pop-up box in the bottom left corner. (1:53)

You can save this key in the CSV you downloaded from AWS. Also gather the public key (2:04)

Note: access and secret keys give control over your AWS and Numerai accounts. Keep them safe like you would usernames and passwords!

Save the public key in the CSV as well. Next, open your anaconda prompt and pip3 install numerai_cli. (2:23)

Once installed, type mkdir example-numerai, then cd example-numerai, then numerai setup. (2:42)

Now, copy and paste the keys as directed by the command prompt. (3:15)

You should see a bunch of print messages showing that you’ve successfully installed numerai-cli. (3:41)

Type numerai docker copy-example. This gives us all the necessary file names and example code which we can overwrite. (3:56)

Now, I am going to run a very simple regression model using RidgeCV from scikit-learn. We’ll save this model using joblib. Type numerai docker train. Docker runs the train.py file, which downloads the data, fits the model, then saves the model using joblib. I’ll spare you the boredom of watching the data download and waiting for the model to train. As you can see, the joblib file has been saved in the example_numerai folder along with the numerai dataset. (4:28)

We want to make sure that the model works correctly before we deploy our docker image to AWS. type numerai docker run. This will build the docker image and run predict.py on your machine. Once your model finishes generating predictions, numerox will then upload your predictions using the API keys you provided. (4:51)

Let’s gather the webhook address. Navigate to the .numerai folder in the example_predictions folder. (4:59)

Open submission_url.txt and copy the address. (5:03)

Open your numerai account settings once again, click Compute, and paste this address in the field, and click change webhooks. (5:15)

As you can see, the last submission shows that our local test was successful. (5:23)

Type numerai docker deploy and wait for your docker files to be pushed to AWS. (5:48)

Test the webhook by typing numerai compute test-webhook. You can get your logs by typing numerai compute logs -f. When you see the message “Task is now in the DEPROVISIONING state, your code is finished. (6:37)

Congratulations, you are now free to enjoy your Saturdays!