Scoring

Scores

There are two main scores currently used for payouts:

• Correlation (CORR): Your prediction's correlation to the target
• Meta Model Contribution (MMC): Your prediction's contribution to the Meta Model

We also have informational scores not used for payouts:

• Feature Neutral Correlation (FNC): Your prediction's correlation to the target after neutralizing against all features
• Correlation with the Meta Model (CWMM): Your prediction's correlation to the Meta Model (stake weighted average of all predictions).
• Benchmark Model Contribution (BMC): Your prediction's correlation to the target after neutralizing against the stake-weighted Benchmark Models.

Live Scoring

Within a single round, submissions receive daily score updates until the round's configured 20-day or 60-day score is final. Let's first look at a historical 20-day weekend round:

The round opens on Saturday the 6th and closes on Monday the 8th. The first day of scoring is on Friday 12th (4 days later), with daily updates every day, Tuesday through Saturday. The final score is released about a month later, on Thursday the 8th.

This round is scored over 20 days because its configured target uses 20 days of total returns. It takes 4 days to get the first score because it takes 2 days to process data and the target lags 2 days behind trading - this means we ignore 2 days of returns before starting the 20 day clock. This timeline is commonly referred to as "20D2L" - 20 Days of returns after 2 Lag days:

The first day of scoring on Friday the 12th uses a 1D2L target, which includes returns from Wednesday 10th only. The second day of scoring on Saturday 13th uses a 2D2L target which includes returns from Wednesday 10th through Thursday 11th. The final day of scoring 4 weeks later on Thursday 8th of next month uses a 20D2L target which includes returns starting from Wednesday 10th through Tuesday 6th of the next month. Take a look at the full scoring timeline:

60D2L scores follow the same logic as 20D2L scores, but resolve 40 business days later (8 weeks later for a total timeline of about 12 weeks). Rounds paying out on 60D2L scores lock the round's stake for that longer period.

The target configured on the round is authoritative. The v5.3 Parquet target alias does not change old round results: historical Ender-20 rounds remain 20-day rounds. Starting with the separately coordinated Ender-60 scoring cutover round, CORR and MMC use the 60-day Ender target.

The Leaderboards

Only the final scores count towards a model's live performance. We rank you both on an account level and on a model level. The primary aggregate score is called your reputation.

Your model's reputation is just its 1 year average score. This directly determines the rank of models on the model leaderboard.

Your account's reputation is determined by the Stake-Weighted-Average Score of your models. Each round, we multiply each of your model's scores by its proportion of stake relative to your account and add them together. This means if you have a single model with no stake, your account score is the same as the reputation of your model.

numer.ai/leaderboard

Diagnostics

The diagnostics tool computes and charts your scores over the validation dataset. v5.3 diagnostics use explicit target_ender_60, the v53_lgbm_ender60 benchmark, and matching Ender-60 example predictions. Eras whose 60-day target has not matured are not eligible for target-dependent diagnostics.

An example diagnostics report

If you uploaded your model via Model Upload, then Numerai will automatically run your model over the validation dataset to generate diagnostics. If you wish, you may also manually run diagnostics by heading over to numer.ai/scores and clicking on the Run Diagnostics button:

WARNING

Caution: past performance is no guarantee of future performance. This is especially true in the domain of financial machine learning. Take care not to rely too heavily on validation metrics during your research process to avoid overfitting to the validation dataset. If you train on the validation dataset, then don't expect your in-sample validation metrics to generalize out-of-sample.

Your history is permanent

A model's round history is a permanent record, and there is no way to reset, clear, or zero it. None of the following erase or alter the scores a model has already earned:

• Retraining your model or changing your approach entirely.
• Re-uploading a new .pkl through Model Uploads, however many times you do it.
• Switching automation methods, or disabling model uploads.
• Archiving a model, which preserves its history but makes it inactive until you unarchive it.
• Skipping rounds, or stopping and later resuming submissions.

Changing your model or your pipeline only affects the predictions you submit from that point forward. Past rounds keep the scores they earned.

Your reputation does still move on its own, because it is a trailing 1 year average: as rounds age out of that window they stop counting toward your reputation and rank, even though their scores remain part of the model's record. That is the window rolling forward, not history being reset.

This is deliberate - a live track record is only meaningful if it cannot be selectively erased after the fact. If you want a clean slate for a new idea, create a new model and submit to that instead: see Starting from zero.