ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction

Accurate prediction of climate in the subseasonal-to-seasonal scale is crucial for disaster readiness, reduced economic risk, and improved policy-making amidst climate change. Yet, S2S prediction remains challenging due to the chaotic nature of the system. At present, existing benchmarks for weather and climate applications, tend to (1) have shorter forecasting range of up-to 14 days, (2) do not include a wide range of operational baseline forecasts, and (3) lack physics-based constraints for explainability. Thus, we propose <PRE_TAG><PRE_TAG>ChaosBench</POST_TAG></POST_TAG>, a large-scale, multi-channel, <PRE_TAG><PRE_TAG>physics-based benchmark</POST_TAG></POST_TAG> for S2S prediction. <PRE_TAG><PRE_TAG>ChaosBench</POST_TAG></POST_TAG> has over 460K frames of <PRE_TAG><PRE_TAG>real-world observations</POST_TAG></POST_TAG> and <PRE_TAG>simulations</POST_TAG>, each with 60 <PRE_TAG><PRE_TAG>variable-channels</POST_TAG></POST_TAG> and spanning for up-to 45 years. We also propose several physics-based, in addition to <PRE_TAG><PRE_TAG>vision-based metrics</POST_TAG></POST_TAG>, that enables for a more physically-consistent model. Furthermore, we include a diverse set of <PRE_TAG><PRE_TAG>physics-based forecasts</POST_TAG></POST_TAG> from 4 <PRE_TAG><PRE_TAG>national weather agencies</POST_TAG></POST_TAG> as baselines to our data-driven counterpart. We establish two tasks that vary in complexity: full and <PRE_TAG><PRE_TAG>sparse dynamics prediction</POST_TAG></POST_TAG>. Our benchmark is one of the first to perform <PRE_TAG><PRE_TAG>large-scale evaluation</POST_TAG></POST_TAG> on existing models including <PRE_TAG>PanguWeather</POST_TAG>, <PRE_TAG>FourCastNetV2</POST_TAG>, <PRE_TAG>GraphCast</POST_TAG>, and <PRE_TAG>ClimaX</POST_TAG>, and finds methods originally developed for weather-scale applications fails on S2S task. We release our benchmark code and datasets at https://leap-stc.github.io/<PRE_TAG><PRE_TAG>ChaosBench</POST_TAG></POST_TAG>.