Forecasting Power Price Spikes: Hurdle Models, Pareto Tails, and Calibration That Carries Money

Power prices spend most of their life in a narrow band and occasionally travel three orders of magnitude above it. Any forecasting system that feeds a storage optimizer has to represent both regimes honestly, because the optimizer places physical quantities at specific quantiles, and the annual P&L concentrates in the hours the tail governs. This reference describes an architecture for that problem: separate models for the body and the tail, spliced into one distribution, with dependence layered on top and calibration verified continuously.

The target, stated precisely

Fix an operating day and a decision time, canonically 10 AM the day before, when ERCOT's day-ahead market closes. The objects to forecast, per settlement location: day-ahead clearing prices per hour, real-time prices per hour or finer, and ancillary clearing prices per product. The target is their joint conditional distribution given the decision-time information set.

Every downstream consumer needs distributions, not points. The dispatch optimizer prices state of charge against the whole range of tomorrow's outcomes; the bid layer places quantities at specific quantiles; the hedge sizes positions off tail probabilities. A point forecast of tomorrow's average price is nearly worthless in a market where the money concentrates in a handful of extreme hours.

Two structural facts dominate the design. First, the marginal distributions are violently non-Gaussian: right-skewed, occasionally reaching an offer cap far above the median, with a meaningful point mass of negative prices in high-renewables hours. Second, dependence matters as much as the marginals: hours within a day are strongly dependent (a tight evening is tight for several hours), and products are dependent (scarcity lifts energy and every reserve price together). The architecture handles the two facts in two deliberately separated stages, marginals first and dependence second, because separating them makes each testable.

The hurdle decomposition

A single flexible model asked to fit both the $15 to $80 normal regime and the $500 to $5,000 spike regime will do both badly: spikes are rare enough to be treated as noise by any loss function dominated by typical days, yet large enough to dominate economics. The remedy is to model the regimes separately and splice them at a threshold. Three separately estimable components result: a spike probability (will this hour exceed the threshold), a body distribution (where prices land in the normal regime), and a tail distribution (how far above the threshold a spike travels).

The spike probability. This is a binary-outcome conditional probability, and the natural engine is gradient-boosted classification, because spikes live in interactions: high net load times low reserves times a binding constraint. Two disciplines matter more than the model choice. Handle class imbalance by calibration, not resampling: spike hours are perhaps 1 to 3 percent of the sample, and oversampling tricks distort predicted probabilities. Fit on the natural base rate and verify calibration explicitly, because the probability itself is the deliverable: downstream layers multiply it by expected spike size, so a classifier that discriminates well but runs miscalibrated by a factor of two is a direct pricing error. And practice regime honesty: the spike-generating process changes with market redesigns, offer-cap changes, and fleet composition drift, so prefer shorter training windows with heavier feature engineering over long windows that average across regimes.

The tail. For spike size, the sample is irreducibly small: a few dozen to a few hundred exceedances per year. This is precisely the situation extreme value theory exists for. The Pickands-Balkema-de Haan theorem says that for a broad class of distributions, exceedances over a high threshold converge to the generalized Pareto distribution (GPD), governed by a scale and a shape parameter. The shape parameter is the economically loaded one: a positive shape gives a heavy, power-law tail, and small changes in it move deep-tail quantiles enormously. Practical refinements: let the scale depend on scarcity covariates so spike severity, not just frequency, responds to conditions; keep the shape constant or nearly so, because shape is poorly identified from small samples; treat the offer cap as a truncation, with a small point mass at the cap for hours the market pins there; and penalize the likelihood, because unconstrained maximum likelihood on dozens of observations is fragile, and a weak prior on the shape buys enormous stability for negligible bias.

Threshold choice. The splice point trades bias against variance: too low and the body contaminates the tail; too high and there are too few exceedances to fit. Mean-excess plots and parameter-stability plots bracket the choice; for ERCOT real-time prices a threshold in the $150 to $300 range with sensitivity checks is defensible. Report results across a threshold grid at least once. If conclusions flip with the threshold, the model is not ready.

The left tail. High-renewables, low-load hours clear negative, which matters for the charging side of battery economics. A symmetric treatment, a small left-hurdle probability with its own much shorter tail, is usually sufficient: the left tail is bounded by bidding-floor economics rather than scarcity dynamics.

The body: quantile machinery

Below the threshold, the workhorse is gradient boosting fit to pinball (quantile) loss at a grid of levels. The grid of fitted quantiles traces the conditional distribution nonparametrically. One mechanical defect needs fixing: independently fitted quantiles can cross, which is nonsense downstream, since a scenario generator sampling from crossed quantiles produces incoherent draws. Post-hoc isotonic rearrangement (sorting the fitted quantiles) is standard, provably never worse in pinball loss, and sufficient.

Splicing body and tail per hour and location yields a complete conditional distribution: invertible for sampling, smooth enough for diagnostics, with tails governed by theory rather than extrapolated flexibility. Day-ahead prices get the same architecture with different settings: the auction averages expectations, so day-ahead is far less spiky and the body model carries more weight.

Features: encoding the decision-time information set

The feature set is where domain knowledge enters, and it matters more than the model class. All features must be evaluated strictly as knowable at decision time.

• Net load and its geometry. Forecast load minus forecast wind minus forecast solar, per hour: the single most powerful scarcity feature. Include levels, ramps, and the day's extremes; the evening net-load ramp is the canonical spike setup.
• Forecast-error risk, not just forecasts. Spikes are usually surprise phenomena: wind undershooting its forecast, load overshooting. The errors are unknown at decision time, but their conditional risk is partially predictable from recent forecast-error variance and regimes where errors historically fatten.
• Reserve margin mechanics. Committed capacity versus forecast peak, planned outages, and historical forced-outage rates by temperature, since extreme heat and cold both degrade the thermal fleet. Published AS requirements and demand curves belong here too.
• Fuel. The morning day-ahead natural gas price sets the thermal fleet's marginal cost and hence the level of the whole non-scarcity distribution.
• Temporal structure. Hour, day of week, season, and autoregressive terms: yesterday's prices carry regime information that pure fundamentals miss.

Point-in-time discipline throughout: every feature must be the vintage available at decision time, the load forecast as published then, not the later revision. Training on final vintages while trading on real-time vintages is the leakage that manufactures phantom backtest edge.

Dependence: scenarios, not just marginals

A storage optimizer's decisions couple hours through state of charge: whether to discharge at hour 18 depends on hour 19's distribution given hour 18's outcome. If hours were modeled independently, the optimizer would plan to arbitrage phantom noise between adjacent hours, and dependence misspecification would show up directly as fictitious value.

The standard construction transforms each hour's outcome through its conditional distribution to a uniform variable, then fits a copula across hours and products to capture the dependence left over after conditioning. A Gaussian copula with structured correlation (hour-distance decay plus a common scarcity factor) is the interpretable workhorse; its known weakness is zero tail dependence, so test a t-copula and prefer it if scarcity hours co-move more than the Gaussian allows. Empirically in power prices, they do.

Spike persistence deserves explicit treatment: conditional on a spike occurring, the probability that the next hour also spikes is far above the unconditional rate. This matters enormously for storage value, since a two-hour battery earns radically different money from one four-hour spike than from four scattered one-hour spikes. Scenario generation samples coherent whole-day paths from the copula, and any scenario reduction must be tail-preserving: naive clustering merges the rare expensive paths into averaged mush, deleting exactly the scenarios the battery is paid to respect.

Calibration: the diagnostics that carry money

Score with proper rules: pinball loss across the quantile grid and its integral, the CRPS, reported against baselines that keep everyone honest, seasonal climatology and persistence. A sophisticated model that fails to beat conditioned climatology is decoration.

Then verify calibration separately from sharpness. Probability integral transform values on held-out walk-forward data should be uniform: a U shape means overconfident forecasts, a hump means underconfident, a slope means biased. Verify event calibration at the levels downstream layers consume: among hours where the model said 5 percent spike probability, did 5 percent spike?

The monetary interpretation is the reason calibration outranks sharpness. The bid layer places physical quantities at prices derived from these quantiles. If the fitted 95th percentile is really the 90th, the battery systematically hoards energy for spikes that arrive half as often as priced: a quantifiable revenue leak, not a statistical blemish. When in doubt, trade sharpness for calibration; an honest wide distribution loses less money than a confident wrong one.

All evaluation is rolling-origin: train through day D, predict day D+1, roll forward, never touching future vintages. Aggregate scores by regime, with spike days scored separately from normal days, because a model can win on average by being mediocre everywhere. And carry tail humility explicitly: the deep tail rests on dozens of observations plus a theoretical extrapolation, so propagate tail-parameter uncertainty into scenarios and let risk limits, not point estimates, govern exposure to outcomes the data has barely sampled.

Failure modes, collected

Training across regime breaks without regime features. Vintage leakage through revised forecasts and outage data. Oversampled spike training that wrecks probability calibration. Unpenalized tail-shape estimates swinging deep quantiles by orders of magnitude. Quantile crossing propagating incoherent scenarios. Gaussian-copula complacency about joint scarcity. Scenario reduction that averages away the tail. Evaluating on average CRPS while the P&L lives in the spike-day subscore. Each is a named, checkable defect, and a validation harness exists to catch them before capital does.