Scoring
Last updated
Last updated
There are two main scores currently used for payouts
Feature Neutral Correlation (FNCv4): Your neutralized prediction's correlation to the target
The target used for this is target_factor_feat_neutral_20 (returns neutralized to common features and factors, yielding "residual return")
Meta Model Contribution (MMC): Your predictions' contribution to the Meta Model
Numerai be will transitioning to paying on different scores for rounds on or after September 2, 2025:
Alpha: Your neutral-weighted predictions multiplied by the chili target
Meta Portfolio Contribution (MPC ): Your predictions' contribution to the Meta Portfolio
We also have informational scores not used for payouts:
Correlation (CORRv4): Your prediction's correlation to the target
Information Coefficient (ICv2): Your prediction's correlation to raw returns
Residual Information Coefficient (RIC): Your prediction's correlation to residual returns (returns neutralized to common factors)
For a full list of detailed explanations please see the definitions docs.
A signal or target is considered "neutral" when it has zero correlation with some set of existing signals. The point of the neutralization is to isolate the original or orthogonal component of the signal that is not already present in existing signals.
If you submit a simple linear combination of a few well-known signals, there will be little to no orthogonal component after neutralization.
Numerai has a variety of existing signals including Barra factors (like size, value, momentum, etc), country and sector risk factors, and custom stock features. Not all of these existing signals are not provided to you, which makes this process somewhat "blackbox".
By neutralizing your signal before scoring, Numerai aligns it with the neutralized target which may improve its performance against the target without Numerai having to give out the data used for neutralization. For example, if your signal is not neutralized to country risks, Numerai Signals will neutralize your signal against country risks before scoring. This allows you to focus on creating an original signal without having to worry about country risk neutralization.
A signal may have strong predictive when considered alone, but could score poorly on Numerai Signals due to this neutralization. This highlights the key unique aspect of Signals: Numerai Signals is not about predicting stock returns, it is about finding original signals that Numerai doesn't already have.
Signals are evaluated against a custom blackbox target that is neutralized against our existing signals.
We provide both 20D2L and 60D2L targets in our dataset. We do not use shorter time horizons because signals that only work on short time horizons are nearly impossible for large hedge funds to trade. For example, even if a signal can accurately predict the 1 hour return of stocks, it is not very useful if it takes a hedge fund 24 hours to fully trade into that position. Signals that are most useful to large hedge funds have predictive power over a long time horizon which is also known as having "low alpha decay".
Grandmasters place first
Masters place in the top 10
Experts place in the top 25%
Researchers place in the top 50%
Contributors place in the top 75%
Apprentices place in the bottom 25%
Novices have not yet made 20 qualified submissions
In the context of the Signals tournament, Canonical Scores (or “Canon Scores”) are particularly relevant. For example, the FNC score, which is a payout metric, has undergone updates. Initially, the payout score was 'CORR20' until round 498. It evolved into 'FNCv4' starting from round 499. The 'Canon FNC' score accounts for these changes by combining them into a unified score — it is 'CORR20' for rounds up to and including 498 and 'FNCv4' for rounds thereafter.
you can use the historical diagnostics of your signal to check performance and estimate the impact neutralization may have on your signal in the future. It’s important to note that signals with strong scores over the historical period may not score well in any current or future round.
The diagnostics tool can be opened using the beaker next to your model on the scores page. Upload a signal over a historical validation time period and it will calculate validation metrics including performance, risk, and potential earnings. The validation time period starts on 20130104 and ends on the latest date in the validation data.
Uploads over the validation time period must include one extra column:
A date column - historic data is weekly and the diagnostics tool assumes your predictions for a given week are made using market close data of the latest Friday
Once your upload is validated, diagnostics will start running. This usually takes 5-10 minutes depending on the number of weeks and tickers that span your submission.
These diagnostics serve as a guide for you to estimate whether your signal is good enough to be worth staking on. It is important to note that signals with strong diagnostics over the historical validation period may not score well in any current or future live periods.
Using this historical evaluation tool repeatedly will quickly lead to overfitting. Treat diagnostics only as a final check in your signal creation process.
"Churn" is a statistic describing how much a signal changes over time. Similarly, "turnover" describes how much portfolio changes over time. We open-sourced the code we use to calculate churn and turnover in Signals. You can find it here. In short:
churn(t0, t1) = 1 - correlation(s(t0), s(t1))
turnover(t0, t1) = (p(t0) - p(t1)).abs().sum() / 2
Where s(t) is your signal at time t and p(t) is your portfolio at time t.
If a Signals submission has high churn/turnover, then Numerai can’t trade the signal. Many models built on the original Numerai tournament data have low churn and turnover organically, but Signals models seem to have naturally high churn and turnover.
We know that this negatively impacts the churn of the Signals Meta Model because the average churn across individual Signals models is highly correlated with the churn of the Signals Meta Model. This means Numerai must disallow high churn and turnover Signals models.
Any model that has not submitted in the previous week will have it’s stake set to 0. This is because any model that does not submit weekly will naturally cause high churn in the Meta Model and turnover in the portfolio derived from the Meta Model.
If your model has submitted within the last week, when you upload a new submission we calculate maximum churn and turnover with respect to this model’s submissions from the previous week. So if we treat the current upload period as time t, the max churn and turnover would be:
max_churn = max([churn(t, t-1), churn(t, t-2), ..., churn(t, t-5)])
max_turnover = max([turnover(t, t-1), turnover(t, t-2), ..., turnover(t, t-5)])
If max_churn >= 15% or max_turnover ≥ 20% then this submissions stake is set to 0.