PortBench

Five-Stage Decision Pipeline

At each rebalance date the LLM produces S1–S3. The sandbox then scores S4 and S5 from that allocation under the same execution and risk rules. The corpus has 183 market series. Portfolio replay uses the 98 series with matched price histories; weight assigned to VIX or to an ineligible asset stays in uninvested cash. LLMs and classical baselines share that replay.

CEPS

CEPS summarizes reference-based stage scores with a penalty for downstream score drops. Each stage score is between 0 and 1. The mean stage score is the average of the five scores. With penalty weight λ = 0.1,

CEPS = clip( mean(σ₁…σ₅) − λ · Σ max(σₜ − σₜ₊₁, 0), 0, 1 )

At the same mean score, CEPS assigns a lower value to a pipeline with larger downstream drops, penalizing higher upstream scores followed by lower downstream scores. The five stage scores receive equal weight.

S1 and S2 use trailing-return rules. S3 combines reference-weight similarity and the class-level correlation score with equal weight (α = 0.5). S4 scores turnover agreement with the reference. S5 compares lookback VaR, drawdown, and a rule-based rebalance flag. The lookback reference supplies a reproducible decision-time target; returns and risk are evaluated from the resulting portfolios.

CEPS ranking sensitivity over 330 normal-period decisions. Each curve compares ten model means with the default ranking. Dotted lines mark λ = 0.1 and α = 0.5.

Stress Windows and Investor Profiles

Profile constraints condition the S1/S2 prompts and the resulting signals. Normal-period CEPS is reported with a stress drawdown gate under three historical windows.

Each profile is tested under three historical stress regimes:

Static QA Evaluation Results

Per-template scores (full & restricted covariance conditions), formula vs. judgment averages, and scores by market regime. Bold = column best.

F = mean(T4, T5), the allocation tasks with supplied statistics. B = mean(T1, T2, T6, T7). T4–T5 average 0.863, versus 0.652 for T1–T2 and T6–T7; this 0.211 gap favors T4–T5 for eight of ten models. Restricted T4 and T5 withhold the covariance statistics.

Pipeline Evaluation

GLM-5.1 ranks first and Qwen3.6-Plus second by CEPS under all three profiles. S3 scores span .264–.310, while turnover agreement (S4) spans .121–.320 and risk-state agreement (S5) .000–.409. The table below reports per-stage scores, CEPS, and financial outcomes for all ten LLMs and six classical baselines (normal period, 2024), including Black–Litterman. Bold = column best within each profile. Select a profile to view results.

QA–Pipeline Rank Inversion

Spearman ρ between QA and balanced-profile CEPS is −.32. Kimi-K2.6 moves from last in QA to fourth in CEPS.

Cross-Period Evaluation

In 2020, EqW Sharpe is −0.602 and the mean model gap is +0.68. In 2024, EqW Sharpe is 0.710. In 2022, every model–profile pair trails EqW, with mean ΔSharpe −0.74.

Sharpe relative to EqW across four market periods. Light shading marks Δ>0 (beat EqW). Percentages in titles are the fraction of model–profile pairs that beat EqW in that period.

Stress Regime Results

The stress gate passes when maximum drawdown stays within the profile tolerance in every stress window. Nine models pass all three balanced-profile windows, and all ten pass under the aggressive profile. Under the balanced profile, DeepSeek-V4-Flash moves from the smallest normal-period drawdown (6.56%) to the largest COVID-window drawdown (22.19%), becoming the only model to fail that profile’s gate.

Maximum drawdown under the Conservative profile in each stress regime. The dashed line marks the 10% tolerance; red bars breach it.

Normalized NAV during the 2022 Market Decline (Conservative), when no LLM beats EqW. Vertical lines mark events during this period.

Broad Holdings Conceal Correlated Risk

The 330 saved 2024 allocations hold 73 assets on average, yet still concentrate weight in correlated assets. For each decision, a matched allocation equalizes weights among the held assets within each class while preserving asset selection, class weights, and uninvested cash.

This equalization lowers volatility estimated from lookback data by 8.5% on average, with reductions in 271 of 330 decisions. Within-class covariance accounts for 76.4% of the mean variance difference, versus 23.5% from individual-asset variance terms. Mean volatility reductions are 1.6%, 9.6%, and 14.4% for conservative, balanced, and aggressive allocations.

A: model weights have higher volatility-weighted pairwise correlation than within-class equal weighting. B: shares of the mean variance difference. C: mean relative volatility reduction from equalization; negative values indicate increased volatility.

Across the 30 normal-period cells, the mean total-variation distance is 0.306 from equal weighting and 0.890 from the profile-constrained reference. Larger deviations from equal weighting are associated with higher volatility (ρ = .65), drawdown magnitude (ρ = .61), and ΔSharpe (ρ = .51). Moving from conservative to aggressive increases distance from equal weighting by .070.

Active-allocation diagnostics across 30 normal-period cells. A: distance from equal weighting versus ΔSharpe. B: the same distance versus drawdown magnitude. Shaded regions mark ΔSharpe > 0 and drawdowns larger than equal weighting.

Under the balanced profile, every model produces less turnover than the reference. The mean model-to-reference turnover ratio is .203. Reference allocations hold 8–14 securities, with maximum weights of .24–.72, whereas model allocations spread weight more broadly. Mean S3 reference-weight similarity is .110. Mean model HHI is .0207, about 48 effective positions. Partial credit for class exposure can therefore coexist with low S4 scores.

Local Gains Can Fade or Reverse

Replacing one stage with its reference tests the effect on later scores. In this auxiliary diagnostic, S4 and S5 score generated execution and risk plans. Replacing S1 raises S2 by +.626 in normal episodes; both references follow the same trailing-return rule. Its mean effect on S3 is only +.017 and is negative in 40 of 110 episodes. Replacing S2 raises S3 by only +.023, with 39 negative cases. Under stress, replacing S2 or S3 lowers S5 by .059 and .056, although most episode-level S5 changes are zero.

Mean score change after replacing the indicated stage with its reference. Asterisks mark BH-FDR q < 0.05. Outlined cells are the stage pairs discussed above.

Real-Time Evaluation

Historical-only evaluation risks data leakage from pretraining. PortBench also supports real-time evaluation: at each decision date the model receives only data available through the previous close. We collect 72 decisions from six models with daily rebalancing over 12 trading days (2026-07-16–07-31). Mean S1 scores range from .78 to .82. This initial run demonstrates stage-level evaluation on newly arriving data.

Per-decision lookback CEPS over the 12-day real-time window (Balanced).

Mean lookback stage scores (S1–S5) over the 12 real-time decision dates.