Validation & Methods

1.1 Classification Notes

CRISIS (12/13, 92% precision): The single false positive across 35 years is August 2019 (yuan devaluation, US-China trade war). CRISIS fired as the S&P 500 fell 3% that day, but VIX never reached the 35 hit threshold and the market recovered within the 10-business-day scoring horizon, so the scorer counts it as a false positive.Full case study with chart.

ELEVATED (37/49, 76% precision): Lower-confidence early-warning classification. Wider net, more false alarms. That tradeoff is intentional and disclosed.

WHIPSAW (117/123, 95% accuracy): When VIX spikes or SPX falls while the system holds STABLE, the classification is being tested and an Event Risk Brief opens. “Accuracy” means the fraction of episodes where the system was correct: the scare faded, or the system escalated within days. The 6 misses are episodes where the system held STABLE but the S&P 500 fell 5% or more within 20 sessions.

RESOLVED (52/71, 73% call-level precision): When the system returns to STABLE after an escalation episode while VIX remains elevated (≥ 20), the classification is saying the crisis has ended before conventional volatility signals confirm it. The 19 revised episodes cluster in structural bear markets (dot-com 2000–2002, GFC 2008–2009, rate cycle 2022) where the system correctly identified interim calms but structural problems resumed. In every case, the system re-escalated within a median of 10 trading days.

3.1 How an Alarm Is Evaluated

1. Fixed rules: each alarm is specified by an immutable definition. No in-sample tuning in the backtest.
2. Daily evaluation: conditions are checked per business day against historical inputs.
3. Episodes: consecutive alert days are collapsed into episodes using business-day cooldowns and windows.
4. Market events: independent market events (VIX/SPX patterns) are identified and deduplicated into event episodes.
5. Scoring: an episode is a "hit" if a qualifying event occurs within a forward-looking scoring horizon. Otherwise it is a false positive.
6. Metrics: precision, recall, and F1 are computed from episodes. All figures on the website come from these validation files.

3.2 Scope

3.3 Overfitting Controls

• No in-sample optimization: backtests run the published rule logic as-is.
• Forward-only scoring: hits require future events within a predefined BD horizon. No look-ahead.
• Deterministic time handling: all windows, horizons, and cooldowns are business-day based.
• Independent event labeling and deduplication with fixed event cooldown (10 BD).
• Episode collapsing to prevent double-counting during prolonged spells.
• Validation file isolation and regression snapshot parity checks.
• Reproducible artifacts: episode-level CSVs (alerts, events, hits) for independent verification.

3.4 Signal Robustness

MRC classifications undergo perturbation analysis (bootstrapped resampling of episode-level outcomes) to verify that precision and recall remain stable under measurement uncertainty, not dependent on specific boundary conditions. Full perturbation methodology and confidence intervals are available in the institutional validation packet.

3.5 Validation Protocols

• Held-out era evaluation on excluded historical eras; no tuning on the evaluation period.
• Walk-forward validation: expanding-window approach with 2-year training, 1-year test, 90-day roll. Era splits (SC24/SC25) validated separately.
• Leave-one-era-out to assess robustness by omitting eras and testing excluded windows.
• Era stratification to surface instability and confirm consistency across market conditions.

3.6 Data Sources

Market data: S&P 500 (Yahoo Finance) and VIX (CBOE official daily history). Environmental data: the Mindforge Signal Platform time series, derived from public datasets (NOAA, NASA, USGS). References to third-party providers are for sourcing only and do not imply endorsement.

4.1 Event Library

4.2 Analog Rules

• As-of discipline. An analog set contains only occurrences that had already happened on the morning of the report. No future information enters any report.
• De-clustering. Occurrences within 10 sessions of a prior occurrence are counted as the same event, so one episode is never counted twice.
• Outcome categories, fixed in advance. “Escalated” = system moved to a more severe state within 10 sessions. “Wrong” = no escalation within 10 sessions AND S&P 500 fell 5%+ within 20 sessions. “Resolved” = neither. The wrong category is always published, with dates.
• Data sources. VIX: CBOE official daily history. S&P 500: Yahoo Finance. Classifications: MRC daily state history. All values are stated with their as-of date.
• Evidence tiers. 1990–2024 classifications are backtested. January 2025 onward is out-of-sample. April 2025 onward was delivered in live production. Reports label these tiers wherever they are mixed.