“all tested llms trail the market in forecasting accuracy”
ForecastGym freezes what four commercial search APIs return for 150 open prediction-market questions — recrawled every two days, 614 evidence states — then replays that evidence offline after settlement, so models compete on identical information. On frozen evidence every tested LLM trails the market. Re-running the same queries post-settlement returns results that nearly state the outcome for up to 92% of questions, despite date filters. Swapping the search API moves Brier scores by 0.043, more than the gap across models.
Extensive technical background assumed
Platforms mentioned: Polymarket, Kalshi