LogoLogo
TournamentsHedge FundDiscordGitHubForumDocs
  • Numerai Tournament
    • Overview
    • Data
    • Models
    • Submissions
      • Model Uploads
      • Compute Heavy
      • NGROK Example
      • CRON Example
    • Scoring
      • Definitions
      • Correlation (CORR)
      • Meta Model Contribution (MMC)
      • Feature Neutral Correlation (FNC)
      • Grandmasters & Seasons
    • Staking
    • Bounties
  • Numerai Signals
    • Overview
    • Data
    • Models
    • Submissions
    • Scoring
      • Definitions
    • Staking
    • Signals + QuantConnect
  • Numerai Crypto
    • Overview
    • Data
    • Submissions
    • Scoring
      • Definitions
    • Staking
  • Community
    • Discord
    • Forum
    • Twitter
    • Youtube
    • Apps & Content
      • Office Hours with Arbitrage
        • Office Hours Recaps: Season 1
          • OHwA S01E01
          • OHwA S01E02
          • OHwA S01E03
          • OHwA S01E04
          • OHwA S01E05
          • OHwA S01E06
          • OHwA S01E07
          • OHwA S01E08
          • OHwA S01E09
          • OHwA S01E10
          • OHwA S01E11
          • OHwA S01E12
        • Office Hours Recaps: Season 2
          • OHwA S02E01
          • OHwA S02E02
          • OHwA S02E03
          • OHwA S02E04
          • OHwA S02E05
          • OHwA S02E06
          • OHwA S02E07
          • OHwA S02E08
          • OHwA S02E09
          • OHwA S02E10
          • OHwA S02E11
        • Office Hours Recaps: Season 3
          • OHwA S03E01
          • OHwA S03E02
          • OHwA S03E03
          • OHwA S03E04
        • Office Hours Season 4
        • FAQ in Office Hours
        • Cited resources
      • Structure of Numerai by Wigglemuse
  • NMR
    • Coinbase
    • Uniswap
    • Etherscan
    • Github
  • Connect
    • Index
Powered by GitBook
On this page
  • Neutralization
  • The Target
  • Grandmasters
  • Canon Scores
  • Diagnostics
  • What is Churn?
  • Why calculate churn?
  • The Churn Threshold
  1. Numerai Signals

Scoring

PreviousSubmissionsNextDefinitions

Last updated 3 months ago

There are two main scores currently used for payouts

  • (FNCv4): Your prediction's correlation to the target

    • The target used for this is target_factor_feat_neutral_20 (returns to common features and factors, yielding "residual return")

  • (MMC): Your predictions contribution to the Meta Model

We also have informational scores not used for payouts:

  • (CORRv4): Your prediction's correlation to the target

  • Information Coefficient (ICv2): Your prediction's correlation to raw returns

  • Residual Information Coefficient (RIC): Your prediction's correlation to residual returns (returns to common factors)

For a full list of detailed explanations please see the .

Neutralization

A signal or target is considered "neutral" when it has zero correlation with some set of existing signals. The point of the neutralization is to isolate the original or orthogonal component of the signal that is not already present in existing signals.

If you submit a simple linear combination of a few well-known signals, there will be little to no orthogonal component after neutralization.

By neutralizing your signal before scoring, Numerai aligns it with the neutralized target which may improve its performance against the target without Numerai having to give out the data used for neutralization. For example, if your signal is not neutralized to country risks, Numerai Signals will neutralize your signal against country risks before scoring. This allows you to focus on creating an original signal without having to worry about country risk neutralization.

A signal may have strong predictive when considered alone, but could score poorly on Numerai Signals due to this neutralization. This highlights the key unique aspect of Signals: Numerai Signals is not about predicting stock returns, it is about finding original signals that Numerai doesn't already have.

The Target

We do not use shorter time horizons because signals that only work on short time horizons are difficult or impossible for large hedge funds to implement. For example, even if a signal can accurately predict the 1 hour return of stocks, it is not very useful if it takes a hedge fund 24 hours to fully trade into that position. Signals that are most useful to large hedge funds have predictive power over a long time horizon which is also known as having "low alpha decay".

Grandmasters

Grandmasters place first

Masters place in the top 10

Experts place in the top 25%

Researchers place in the top 50%

Contributors place in the top 75%

Apprentices place in the bottom 25%

Novices have not yet made 20 qualified submissions

Canon Scores

In the context of the Signals tournament, Canonical Scores (or “Canon Scores”) are particularly relevant. For example, the FNC score, which is a payout metric, has undergone updates. Initially, the payout score was 'CORR20' until round 498. It evolved into 'FNCv4' starting from round 499. The 'Canon FNC' score accounts for these changes by combining them into a unified score — it is 'CORR20' for rounds up to and including 498 and 'FNCv4' for rounds thereafter.

The 'TC' score has remained consistent throughout the tournament's history, meaning 'Canon TC' and 'TC' are equivalent.

Diagnostics

you can use the historical diagnostics of your signal to check performance and estimate the impact neutralization may have on your signal in the future. It’s important to note that signals with strong scores over the historical period may not score well in any current or future round.

Uploads over the validation time period must include one extra column:

  • A date column - historic data is weekly and the diagnostics tool assumes your predictions for a given week are made using market close data of the latest Friday

Once your upload is validated, diagnostics will start running. This usually takes 5-10 minutes depending on the number of weeks and tickers that span your submission.

These diagnostics serve as a guide for you to estimate whether your signal is good enough to be worth staking on. It is important to note that signals with strong diagnostics over the historical validation period may not score well in any current or future live periods.

Using this historical evaluation tool repeatedly will quickly lead to overfitting. Treat diagnostics only as a final check in your signal creation process.

What is Churn?

churn(t0, t1) = 1 - correlation(s(t0), s(t1))

Where s(t) is Signal's submission at time t.

Why calculate churn?

If a Signals submission has high churn, then Numerai can’t trade the signal. Many models built on the original Numerai tournament data have low churn organically, but Signals models seem to have naturally high churn.

We know that this negatively impacts the churn of the Signals Meta Model because the average churn across individual Signals models is highly correlated with the churn of the Signals Meta Model. This means Numerai must disallow high churn Signals models.

The Churn Threshold

Any model that has not submitted in the previous week will have it’s stake set to 0. This is because any model that does not submit weekly will naturally cause high churn in the Meta Model.

If your model has submitted within the last week, when you upload a new submission we calculate maximum churn with respect to this model’s submissions from the previous week. So if we treat the current upload period as time t, the max churn would be:

max_churn = max([churn(t, t-1), churn(t, t-2), ..., churn(t, t-5)])

If max_churn >= 15%, then this submissions stake is set to 0.

Numerai has a variety of existing signals including Barra factors (like size, value, momentum, etc), country and sector risk factors, and custom stock features. These existing signals are not provided to you, which makes this process "blackbox". The code used to neutralize a signal is open source and you can learn more about the neutralization process in .

To understand broader implications of feature exposure and neutralization, read .

Signals are evaluated against a custom blackbox target created by Numerai. This target is also a target like the Numerai Tournament, but are neutralized against our existing signals.

The diagnostics tool can be opened using the beaker next to your model on the page. Upload a signal over a historical validation time period and it will calculate validation metrics including performance, risk, and potential earnings. The validation time period starts on 20130104 and ends on the latest date in the validation data.

Churn is a statistic describing how much a signal changes over time. We open-sourced the code we use to calculate churn in Signals. You can find it . In short:

this example notebook
this forum post
scores
here
Feature Neutral Correlation
Meta Model Contribution
Correlation
definitions docs
neutralized
neutralized
neutralized
20D2L
A visualization of neutralization against a single known signal