LogoLogo
TournamentsHedge FundDiscordGitHubForumDocs
  • Numerai Tournament
    • Overview
    • Data
    • Models
    • Submissions
      • Model Uploads
      • Compute Heavy
      • NGROK Example
      • CRON Example
    • Scoring
      • Definitions
      • Correlation (CORR)
      • Meta Model Contribution (MMC)
      • Feature Neutral Correlation (FNC)
      • Grandmasters & Seasons
    • Staking
    • Bounties
  • Numerai Signals
    • Overview
    • Data
    • Models
    • Submissions
    • Scoring
      • Definitions
    • Staking
    • Signals + QuantConnect
  • Numerai Crypto
    • Overview
    • Data
    • Submissions
    • Scoring
      • Definitions
    • Staking
  • Community
    • Discord
    • Forum
    • Twitter
    • Youtube
    • Apps & Content
      • Office Hours with Arbitrage
        • Office Hours Recaps: Season 1
          • OHwA S01E01
          • OHwA S01E02
          • OHwA S01E03
          • OHwA S01E04
          • OHwA S01E05
          • OHwA S01E06
          • OHwA S01E07
          • OHwA S01E08
          • OHwA S01E09
          • OHwA S01E10
          • OHwA S01E11
          • OHwA S01E12
        • Office Hours Recaps: Season 2
          • OHwA S02E01
          • OHwA S02E02
          • OHwA S02E03
          • OHwA S02E04
          • OHwA S02E05
          • OHwA S02E06
          • OHwA S02E07
          • OHwA S02E08
          • OHwA S02E09
          • OHwA S02E10
          • OHwA S02E11
        • Office Hours Recaps: Season 3
          • OHwA S03E01
          • OHwA S03E02
          • OHwA S03E03
          • OHwA S03E04
        • Office Hours Season 4
        • FAQ in Office Hours
        • Cited resources
      • Structure of Numerai by Wigglemuse
  • NMR
    • Coinbase
    • Uniswap
    • Etherscan
    • Github
  • Connect
    • Index
Powered by GitBook
On this page
  • Tutorials
  • Benchmark Models
  • Download
  • How they are made
  • What are they?
  • Community Models
  1. Numerai Tournament

Models

PreviousDataNextSubmissions

Last updated 6 months ago

Tutorials

The best way to learn about building models on our data is through our tutorials:

Benchmark Models

Numerai Benchmark Models are a set of standard models that the Numerai team built. Their predictions are then given out every round so that anyone can easily submit them and stake on them if they want. These models are an easy way to compare your model to the current state-of-the-art.

The list of models and their recent performance is here:

Download

The validation and live predictions are available through the .

from numerapi import NumerAPI
napi = NumerAPI()
napi.download_dataset("v5.0/train_benchmark_models.parquet", "train_benchmark_models.parquet")
napi.download_dataset("v5.0/validation_benchmark_models.parquet", "validation_benchmark_models.parquet")
napi.download_dataset("v5.0/live_benchmark_models.parquet", "live_benchmark_models.parquet")

How they are made

Walk Forward Cross Validation

All predictions are made using a Walk-Forward framework. This means all predictions are made using models which were trained only on data which was available prior to the date of the prediction being made.

Specifically, the data is split up into chunks of 156 eras. Then for each chunk of eras, the predictions are given by a model which is trained up to first_era_of_chunk - purge_eras. The number of purge_eras is always 8 for 20D targets, and 16 for 60D targets.

So a model is trained on eras 1 through 148, then purge eras 149 through 156, and then predict eras 157 through 312. Next, train on eras 1 through 304, purge 305 through 312, predict 313 through 468, and so on. Your walk-forward validation windows should look something like this:

Window
Train Start
Train End
Val Start
Val End

1

1

148

157

312

2

1

304

313

468

3

1

460

469

624

4

1

616

625

780

...

...

...

...

...

Standard Large LGBM params

Most models use the following LGBM parameters:

standard_large_lgbm_params = {
  "n_estimators": 20000,
  "learning_rate": 0.001,
  "max_depth": 6,
  "num_leaves": 2**6,
  "colsample_bytree": 0.1,
}

Deep LGBM params

After the release of v5 data, we announced the higher performance "deep" parameters we used to train the v5 benchmark models:

deep_lgbm_params = {
  "n_estimators": 30000,
  "learning_rate": 0.001,
  "max_depth": 10,
  "num_leaves": 1024,
  "colsample_bytree": 0.1,
  "min_data_in_leaf": 10000
}

Ensembles

All of the ensembles use the following steps:

  1. gaussianize each of the predictions on a per-era basis

  2. standardize to standard deviation 1

  3. dot-product the predictions with a weights vector representing the desired weight on each model

  4. gaussianize the resulting vector

  5. (if applicable) neutralize the vector

Steps 1 through 4 look something like this:

def rank_gauss_pow1(s: pd.Series) -> pd.Series:
  # do rank-normalize
  s_rank = rank_keep_ties_keep_na(s)

  # gaussianize
  s_rank_norm = pd.Series(stats.norm.ppf(s_rank), index=s_rank.index)

  # Standardize to 1 std
  result_series = s_rank_norm / s_rank_norm.std()

  return result_series


ensemble_cols = ["V4_LGBM_NOMI20", "V42_RAIN_ENSEMBLE"]
weight_vector = [0.1, 0.9]
for col in X[ensemble_cols]:
  if "era" in X.columns:
      X[col] = X.groupby("era", group_keys=False)[col].transform(lambda s1: rank_gauss_pow1(s1))
  else:
      # check X contains only a single era
      assert 1800 < X.shape[0] < 6000
      X[col] = rank_gauss_pow1(X[col])
return X[ensemble_cols].dot(weight_vector)

Neutralization

A couple of the models have some neutralization involved. This is basically doing a regression to find out your predictions' exposures to each feature, and then subtracting those exposures from your predictions vector such that the result is a vector which is orthogonal to all of those features.

Here's the code to neutralize some set of vectors (columns) by some list of features (neutralizers):

def neutralize(
  df, columns, neutralizers=None, proportion=1.0, era_col="era"
):
  if neutralizers is None:
      neutralizers = []
  unique_eras = df[era_col].unique()
  computed = []
  for u in unique_eras:
      df_era = df[df[era_col] == u]
      scores = df_era[columns].values
      scores2 = []
      for x in scores.T:
          x = pd.Series(x)
          x = (x.rank(method="first") - 0.5) / len(x.dropna())
          x = stats.norm.ppf(x)
          scores2.append(x)
      scores = np.array(scores2).T
      exposures = (
          df_era[neutralizers]
          .fillna(df_era[neutralizers].median())
          .fillna(0.5)
          .values
      )

      scores -= proportion * exposures.dot(
          np.linalg.pinv(exposures.astype(np.float32), rcond=1e-6).dot(
              scores.astype(np.float32)
          )
      )

      scores /= pd.DataFrame(scores).std(ddof=0, axis=0, skipna=True).values

      computed.append(scores)

  return pd.DataFrame(np.concatenate(computed), columns=columns, index=df.index)

What are they?

The naming formula for many benchmarks is as follows:

{data_version}_LGBM_{target}

There are many models that have some combination of a data version (V2, V3, V4, V41, V42, V43, V5) and a target (e.g. cyrusd_20, teager2b_20, etc.). These are models trained in the standard walk-forward way, with standard LGBM parameters, using the specified data version and target. That's all!

There are also unique models we created that don't have that naming scheme:

V5_LGBM_CT_BLEND (coming soon)

This is a simple 50/50 blend of V5_LGBM_TEAGER2B20 and V5_LGBM_CYRUSD20

The following models are on the Benchmark Models page, but their predictions aren't present in the predictions files because they are easily reproducible:

INTEGRATION_TEST - Submits our favorite model at the time. This has transitions through V2, V3, and V4 example predictions. It is now v5_lgbm_ct_blend.

NB_HELLO_NUMERAI - Submits the model created by the default Hello Numerai tutorial notebook.

NB_FEATURE_NEUTRAL - Submits the model created by the feature neutralization tutorial notebook.

NB_TARGET_ENSEMBLE - Submits the model created by the target ensemble tutorial notebook.

NB_EXAMPLE_MODEL - Submits the model created by the barebones example_model notebook.

Community Models

We've found that having more trees can be helpful, and we've found that having less trees with more depth can also achieve similar results with lower compute requirements. You can read more about this hyper-parameter research in .

The Numerai community has also developed , a website to buy and sell models built by and for the Numerai community. Keep in mind that Numerai does not gaurantee the performance of any model listed on Numerbay,

Hello Numerai
Feature Neutralization
Target Ensembles
numer.ai/~benchmark_models
api
this forum post
Numerbay