Library/ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents
ApplicationsResearch Paper

ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents

Yicheng Tao, Yuante Li, Zipei Fan·November 17, 2026·Academic Paper
“all tested llms trail the market in forecasting accuracy”

Why It's Worth Reading

ForecastGym freezes what four commercial search APIs return for 150 open prediction-market questions — recrawled every two days, 614 evidence states — then replays that evidence offline after settlement, so models compete on identical information. On frozen evidence every tested LLM trails the market. Re-running the same queries post-settlement returns results that nearly state the outcome for up to 92% of questions, despite date filters. Swapping the search API moves Brier scores by 0.043, more than the gap across models.

Extensive technical background assumed

Concepts

Platforms mentioned: Polymarket, Kalshi

Related Reading