PortBench

A Correlation-Aware, Full-Pipeline Benchmark
for LLM-Driven Portfolio Management

Yuxuan Zhao^1,2, Sijia Chen², Ningxin Su²

¹Yantai Research Institute of Harbin Engineering University
²The Hong Kong University of Science and Technology (Guangzhou)

Corresponding to: sijiachen@hkust-gz.edu.cn

arXiv 🤗 Dataset Code Raw Experimental Results Interactive Browser Evaluation Details Results & Insights

Background & Motivation

Large language models (LLMs) have shown strong performance across diverse financial tasks, yet portfolio management (PM) remains poorly benchmarked. Existing benchmarks ignore cross-asset correlation structures and fail to evaluate the complete PM decision pipeline, missing the compounding errors that arise as reasoning propagates through sequential allocation stages.

We introduce PortBench with the following key contributions:

Dataset

Market Base Dataset

183 instruments across 6 asset classes (equities, bonds, commodities, crypto, real estate, cash) spanning 2015–2025, with daily prices, returns, macro indicators, and news. Inter-class correlations are low, intra-class correlations are high: true diversification means crossing asset-class boundaries, not just picking more tickers.

Static Layer

6,269 QA Pairs

6,269 correlation-based QA pairs across 7 templates (T1–T7) and 4 difficulty levels, auto-generated from historical data via analytical formulas. Tests correlation reasoning from single-asset prediction to multi-asset constrained allocation to regime-driven rebalancing. Questions and ground truths are derived automatically (no human annotation needed), and new templates can be added on demand.

Dynamic Layer

Five-Stage Allocation Pipeline

Models execute S1 (Market Interpretation) → S2 (Signal Generation) → S3 (Weight Optimization) → S4 (Execution Simulation) → S5 (Risk Monitoring) sequentially at each rebalance date. A stateful sandbox tracks per-stage scores, weights, and NAV through time to reveal how early errors cascade into final outcomes. Evaluated under 3 investor profiles and 3 historical stress regimes.

Metrics

Dual-Layer Correlation Score + CEPS

A dual-layer correlation score that measures whether portfolios truly exploit inter-class hedging and avoid intra-class concentration. CEPS, a cross-stage error propagation score, quantifies how reasoning errors compound across pipeline stages: unlike prior benchmarks, CEPS penalizes error cascades rather than averaging scores.

PortBench framework overview: market data collection, dual-layer evaluation with static QA and dynamic pipeline, three risk profiles and stress regimes

Overview of PortBench. We first collect the Market Base Dataset (183 instruments × 6 asset classes, 2015–2025), then build a dual-layer evaluation framework on top: a static QA layer (6,269 correlation-based pairs) and a dynamic five-stage pipeline, jointly assessed under three risk profiles and three historical stress regimes.

Two-layer evaluation framework: static QA layer with 6,269 pairs above, dynamic five-stage pipeline below, both fed by market snapshots

Evaluation framework. Static QA layer (Top): seven task templates generated automatically from historical data. Dynamic five-stage pipeline (Bottom): executed sequentially at every rebalance date under three investor profiles and three stress regimes.

Key Findings

90%

LLMs Fail to Beat Equal-Weight

In 2024, 90% of model–profile cases (27 / 30) fail to beat a naive 1/N equal-weight baseline on Sharpe ratio. Across four market periods the beat rate is only 32.5% (39/120). None of the three 2024 winners also pass every profile’s stress gate.

70%

Stress Rankings Diverge from Normal-Period

Normal-period CEPS rankings do not predict stress survival. 70% of models (7 / 10) fail the Conservative stress gate, concentrated in the 2022 Crypto Collapse. Only Qwen3.6-Plus, Qwen3.7-Max, and Doubao-Seed-2.0-Lite pass all three stress regimes under Conservative. Compliant allocations can still produce double-digit drawdowns via cross-asset contagion.

ρ = −0.47

QA Performance ≠ Pipeline Competence

Spearman rank correlation between QA accuracy and pipeline CEPS is ρ = −0.47. GLM-5.1 ranks 7th in QA yet 1st in CEPS; Kimi-K2.6 ranks last in QA yet 3rd in CEPS; DeepSeek-V4-Flash ranks 1st in QA but 9th in CEPS. QA measures isolated factual recall; CEPS assesses sustained reasoning across five causally dependent stages.

Interactive Browser

The sections below let you interactively explore each layer of the PortBench. Start with the raw Market Base Dataset, then dive into the two evaluation layers that run on top of it.

Market Base Dataset

The Market Base Dataset covers 183 unique financial instruments spanning 2015–2025 across six heterogeneous asset classes, collected from Yahoo Finance, FRED, and Kaggle. Equities exhibit the broadest coverage (126 tickers), reflecting the diversity of broad-market, sector, and factor ETFs. Commodities (16) and bonds (15) provide representative cross-class hedging opportunities; cryptocurrency (12) captures major and mid-cap digital assets; real estate (10) and cash equivalents (4) round out the defensive allocation universe.

Correlation analysis reveals that inter-class average correlations are generally low while intra-class correlations are strongly positive. True diversification requires crossing asset class boundaries, not merely spreading across tickers within the same class, directly motivating the two-layer correlation scoring design.

Interactive Market Snapshot

183 instruments across 6 asset classes, daily data 2015–2025. Each monthly snapshot includes macro indicators, per-asset price summaries, and cross-class correlations. Select a date to see the full snapshot. To keep the layout compact, the six asset class tables are collapsed by default: click any class header to expand and inspect its representative tickers.

Date:

Loading market snapshots…

Bar chart showing instrument count per asset class: 126 equities, 15 bonds, 16 commodities, 12 crypto, 10 real estate, 4 cash equivalents

Number of unique tickers/series per asset class.

Pairwise Pearson correlation heatmap of daily returns across all 183 instruments, 2015–2022, showing strong intra-class and weak inter-class correlation

Pairwise Pearson correlation matrix (daily returns, 2015–2022).

Bar chart of mean inter-class correlation for each asset class vs. all others, showing low average cross-asset correlation

Mean pairwise correlation: each class vs. all others.

Normalized Price Trajectories by Asset Class (2015–2025)

Base = 100 at first listing date. Each panel shows representative instruments from one asset class.

Equities (126 tickers: broad-market, sector, factor ETFs)

Bonds (15 series: full yield curve, TIPS, credit)

Commodities (16 tickers: energy, metals, agriculture)

Cryptocurrency (12 tickers: major and mid-cap digital assets)

Real Estate (10 series: REITs and housing indices)

Cash Equivalents (4 series: money market, ultra-short duration)

1 / 6

Evaluation Framework

The evaluation framework has two complementary layers. Switch between the tabs below to explore the QA Dataset and the Pipeline Evaluation in detail. Every component is objective, traceable, and scalable: QA pairs are auto-generated without human annotation, and pipeline scoring uses a lookback oracle (trailing data available at the decision date; no future returns in the main rankings), eliminating oracle leakage and enabling seamless extension to new periods and assets.

⚙️ Pipeline Evaluation
❓ QA Dataset (T1–T7)

6,269 correlation-aware QA pairs across 7 task templates (T1–T7), spanning complexity levels 1–4 and three market regimes. Each template tests a different aspect of portfolio reasoning. Select a template below to browse sample question–answer pairs.

Loading QA samples…

At each rebalance date a MarketSnapshot is constructed and passed to the LLM for five-stage evaluation (S1–S5) under the Balanced profile. Traces use the lookback S3 oracle (no future returns), matching the paper’s main pipeline results. Select a model, market scenario, and date to see the model’s input and stage-by-stage output vs. ground truth.

Model: Scenario: Date:

Loading pipeline traces…

Evaluation Details

At a Glance

10 LLMs evaluated

6 classical baselines

6 asset classes

183 financial instruments

6,269 QA pairs

10 years of data (2015–2025)

Five-Stage Decision Pipeline

At each rebalance date the LLM executes S1–S5 sequentially. LLMs and classical baselines share the identical backtest environment for controlled comparison.

S1 · Market Interpretation Continuous sentiment views v_i ∈ [−1,+1] per asset. Scored as 1 − MAE(views, ground-truth) / 2.

S2 · Signal Generation Views discretized into buy / hold / sell signals (thresholds ±0.2). Scored as fraction of assets with the correct direction.

S3 · Weight Optimization Portfolio weights scored by the two-layer correlation score: weight accuracy (L₁ to signal-constrained max-Sharpe optimum, α = 0.5) + correlation structure (intra-class concentration penalty + inter-class hedging credit). Primary ground truth: trailing 60-day lookback max-Sharpe (no future returns); ex-post oracle only as appendix diagnostic.

S4 · Execution Simulation Deterministic. Fixed transaction costs (10 bps slippage + 5 bps commission). Scores turnover deviation from the oracle rebalancing rate.

S5 · Risk Monitoring Deterministic. 50% rebalance-trigger accuracy + 50% VaR/drawdown estimation accuracy. Rebalance trigger fires on drawdown breach of the profile tolerance, or when max weight drift from 1/n exceeds 5%.

CEPS: Cross-Stage Error Propagation Score

Prior benchmarks obscure early reasoning failures by averaging scores. CEPS penalizes error cascades, a strong stage followed by a weak one, more heavily than uniform mediocrity, capturing the operational reality that a perfectly interpreted market view is worthless if signal generation immediately fails.

CEPS = clip( mean(σ₁…σ₅) − λ · Σ max(σₜ − σₜ₊₁, 0), 0, 1 ) Default propagation weight λ = 0.1

Main-text rankings use a lookback oracle (trailing 60-day returns available at the decision date; no future returns). An ex-post variant that optimizes on realized forward returns is reported only as an appendix diagnostic.

CEPS: Cascade vs. Uniform Mediocrity

Two models with identical average stage scores (0.516) receive different CEPS scores because one cascades errors while the other is uniformly mediocre.

	S1	S2	S3	S4	S5	Avg
Model A (cascade)	0.792	0.506	0.714	0.086	0.480	0.516
Model B (uniform)	0.516	0.516	0.516	0.516	0.516	0.516

	Model A (cascade)	Model B (uniform)
Isolated avg	0.516	0.516
Cascade drops	(0.792−0.506) + (0.714−0.086) = 0.286 + 0.628 = 0.914	0
Penalty (lambda=0.1)	0.1 × 0.914 = 0.091	0
CEPS	0.516 − 0.091 = 0.425	0.516 − 0 = 0.516

          The cascade penalty (lambda=0.1) reduces Model A's CEPS by 0.086, penalizing the sharp S1 to S2 and S3 to S4 drops that indicate brittle error propagation. Model B's uniform mediocrity incurs no penalty.
        

Investor Profiles & Stress Regimes

CEPS is evaluated under three investor risk profiles with escalating risk tolerance, and back-tested across three historical stress regimes to assess robustness when market conditions deteriorate sharply.

Each profile is tested under three historical stress regimes:

2015

China Shock

Aug 2015 – Feb 2016

liquidity-driven

2020

COVID Crash

Feb 2020 – May 2020

pandemic-driven

2022

Crypto Collapse

May 2022 – Dec 2022

monetary-tightening-driven

A model passes the stress gate for a given profile if its maximum drawdown across all three regimes stays within the profile's tolerance. 70% of models (7 of 10) fail the Conservative stress gate, concentrated in the 2022 Crypto Collapse, where small crypto exposures compliant with allocation caps amplify into double-digit drawdowns (compliance trap: every process constraint satisfied, outcome safety violated).

Results & Insights

Pipeline Performance: All Three Profiles

🔑 Key Finding

Across 30 model–profile combinations (10 LLMs × 3 profiles), 90% (27 of 30) fail to beat a naive 1/N equal-weight allocation on Sharpe ratio in 2024 (EqW Sharpe 0.863), and none of the three that succeed also pass every profile’s stress gate. S1 is nearly saturated while S4 is weakest: models extract views but under-trade (balanced mean turnover ratio 10.9%). The three EqW beaters are Qwen3.6-35B-A3B (Conservative/Aggressive) and Doubao-Seed-2.0-Pro (Conservative).

The table below reports per-stage scores, CEPS, and financial outcomes for all ten LLMs and six classical baselines (normal period, 2024), including Black–Litterman. Bold = column best within each profile. Select a profile to view results. Under Conservative, 70% of models fail the stress gate; all pass under Balanced and Aggressive.

Model	Pipeline Scores					CEPS	Financial Outcomes				Gate
Model	S1	S2	S3	S4	S5	CEPS	Sharpe	Ret%	MaxDD%	Vol%	Gate
GLM-5.1	.769	.458	.751	.239	.314	.420	0.652	10.96	−7.24	10.04	✗
Qwen3.6-Plus	.780	.496	.752	.089	.319	.391	0.748	12.95	−8.52	11.22	✓
Kimi-K2.6	.771	.388	.750	.199	.331	.388	0.631	10.18	−5.87	9.16	✗
Qwen3.6-35B-A3B	.756	.395	.754	.138	.339	.376	1.068	14.28	−5.08	8.68	✗
DeepSeek-V4-Flash	.744	.302	.995	.180	.300	.375	0.250	6.19	−7.11	8.73	✗
Hunyuan3-Preview	.782	.499	.736	.033	.317	.374	0.800	11.60	−5.17	8.72	✗
DeepSeek-V4-Pro	.751	.321	.995	.115	.306	.365	0.624	9.79	−6.02	8.65	✗
Qwen3.7-Max	.752	.393	.738	.086	.299	.352	0.823	11.47	−5.25	8.30	✓
Doubao-Seed-2.0-Pro	.753	.406	.744	.057	.315	.351	0.939	12.11	−4.75	7.82	✗
Doubao-Seed-2.0-Lite	.755	.342	.745	.053	.300	.328	0.863	11.84	−5.64	8.29	✓
Equal-Weight (EqW)	N/A	N/A	N/A	N/A	N/A	N/A	0.863	11.54	−4.48	7.95	N/A
60/40	N/A	N/A	N/A	N/A	N/A	N/A	0.841	11.95	−4.27	8.65	N/A
Risk Parity	N/A	N/A	N/A	N/A	N/A	N/A	0.603	6.01	−1.39	2.77	N/A
Cov. Risk Parity	N/A	N/A	N/A	N/A	N/A	N/A	0.343	5.09	−1.43	2.39	N/A
Min-Variance	N/A	N/A	N/A	N/A	N/A	N/A	−0.043	4.11	−1.28	2.10	N/A
Black–Litterman	N/A	N/A	N/A	N/A	N/A	N/A	−0.173	3.06	−4.48	5.47	N/A

Portfolio NAV trajectories (2024, Conservative). Shaded band = range across all LLMs; dashed lines = classical baselines.

Stress Regime Results

🔑 Key Finding

Normal-period CEPS rankings do not predict stress survival. 70% of models fail the Conservative stress gate despite satisfying every allocation constraint (equity/crypto caps, bond/cash floors, drawdown/VaR limits)—a compliance trap. Failures concentrate in the 2022 Crypto Collapse. Only Qwen3.6-Plus, Qwen3.7-Max, and Doubao-Seed-2.0-Lite keep worst-case drawdowns within the Conservative 10% tolerance across all three regimes.

Covariance-aware baselines (CovRiskPar, MinVar) stay well inside the gate but at the cost of near-zero or negative normal-period Sharpe. LLMs fall behind when they fail to cut risk exposures during the initial selloff; later shocks deepen losses on an already weakened portfolio.

The stress gate: a model passes for a given profile if its maximum drawdown across all three historical stress regimes stays within that profile’s tolerance. Under Balanced and Aggressive, all models pass; under Conservative, seven of ten fail.

Worst-case MaxDD per model and baseline across three stress regimes, with Conservative 10% gate dashed line

Worst-case MaxDD (%) across three stress regimes (min over profiles). Dashed line: Conservative 10% gate; orange bars fail.

2022 Crypto Collapse NAV (Conservative)—the regime where LLMs perform worst: no LLM beats EqW. Vertical lines mark Terra/LUNA, 3AC/Celsius, Jackson Hole, and FTX.

Cross-Period Evaluation

🔑 Key Finding

Reusing the three stress windows as additional regimes alongside 2024, LLMs beat EqW in only 39 of 120 model–profile cases (32.5%). Beat rates: 77% (2020) → 43% (2015–16) → 10% (2024) → 0% (2022). LLMs look best in sharp crash-and-recovery windows rather than calm bulls or grinding bears.

In 2020, EqW Sharpe was near zero (0.045) and recovery-tilted models gained substantially (mean ΔSharpe +0.72). In 2024, EqW was strong (0.863), making active bets hard to justify. In the prolonged 2022 bear, no pair beat EqW (mean gap −0.93).

Sharpe relative to EqW across four market periods. Light shading marks Δ>0 (beat EqW). Percentages in titles are the fraction of model–profile pairs that beat EqW in that period.

Real-Time Evaluation

Historical-only evaluation risks data leakage from pretraining. PortBench also supports real-time evaluation: at each decision date the model receives only data available through the previous close. We run daily rebalancing over 12 trading days (2026-07-16–07-31) under the Balanced profile for six models. Real-time rankings differ from the 2024 monthly block: Kimi-K2.6 leads lookback CEPS (.461), while S4 remains weak (.05–.16), consistent with under-trading.

Per-decision lookback CEPS over the 12-day real-time window (Balanced).

Mean lookback stage scores S1 through S5 over the 12 real-time decision dates

Mean lookback stage scores (S1–S5). Interpretation is strong; execution and risk control remain weak.

Static QA Evaluation Results

Per-template accuracy reveals a sharp divide between formula-driven tasks (T4, T5) and judgment-driven ones (T1, T2, T6, T7)—the QA-side counterpart to the pipeline results above.

Per-template accuracy (full & restricted covariance conditions), formula vs. judgment averages, and accuracy by market regime. Bold = column best. Pink rows = Mean < 0.65.

Model	Per-Template (Full)							Mean	Restricted		Task Type		Market Regime
Model	T1	T2	T3	T4	T5	T6	T7	Mean	T4_r	T5_r	F	J	Bull	Bear	Side.
DeepSeek-V4-Flash	.520	.843	.945	1.00	.932	.652	.843	.819	.975	.860	.966	.715	.827	.823	.812
Qwen3.7-Max	.500	.859	.951	1.00	.954	.724	.742	.819	1.00	.990	.977	.706	.814	.863	.810
DeepSeek-V4-Pro	.520	.837	.963	1.00	.992	.652	.760	.818	1.00	.660	.996	.692	.844	.846	.802
Doubao-Seed-2.0-Lite	.460	.798	.957	.956	.897	.810	.747	.804	.961	.940	.927	.704	.780	.846	.806
Doubao-Seed-2.0-Pro	.440	.847	.963	.991	.912	.824	.530	.787	.979	.923	.952	.660	.764	.806	.792
Qwen3.6-Plus	.440	.858	.968	1.00	.804	.640	.768	.783	1.00	.810	.902	.677	.799	.801	.771
GLM-5.1	.440	.855	.964	1.00	.421	.882	.738	.757	1.00	.531	.711	.729	.778	.765	.746
Qwen3.6-35B-A3B	.460	.808	.961	1.00	.230	.564	.763	.684	1.00	.320	.615	.649	.714	.729	.662
Hunyuan3-Preview	.460	.386	.336	.975	.958	.468	.783	.624	.982	.974	.967	.524	.664	.663	.597
Kimi-K2.6	.420	.422	.493	.956	.280	.684	.320	.511	.978	.710	.618	.462	.556	.531	.487

F = mean(T4,T5) formula-driven; J = mean(T1,T2,T6,T7) judgment-driven. Restricted (T4_r, T5_r) withholds the covariance matrix: 8 of 10 models perform better without it, confirming format matching rather than genuine numerical reasoning.

QA–Pipeline Rank Dissociation

🔑 Key Finding

QA performance does not imply pipeline competence. Balanced CEPS falls to 0.329–0.495 despite mean QA up to 0.819, and the Spearman rank correlation is ρ = −0.47. GLM-5.1 ranks 7th in QA yet 1st in CEPS; Kimi-K2.6 last in QA yet 3rd in CEPS; DeepSeek-V4-Flash ranks 1st in QA but 9th in CEPS. QA measures isolated factual recall, whereas CEPS assesses sustained reasoning across five causally dependent stages.

Model	QA Mean	QA Rank	CEPS_bal	CEPS Rank	ΔRank
DeepSeek-V4-Flash	.819	1	.348	9	−8
Qwen3.7-Max	.819	2	.398	6	−4
DeepSeek-V4-Pro	.818	3	.399	5	−2
Doubao-Seed-2.0-Lite	.804	4	.329	10	−6
Doubao-Seed-2.0-Pro	.787	5	.386	7	−2
Qwen3.6-Plus	.783	6	.434	2	+4
GLM-5.1	.757	7	.495	1	+6
Qwen3.6-35B-A3B	.684	8	.399	4	+4
Hunyuan3-Preview	.624	9	.384	8	+1
Kimi-K2.6	.511	10	.402	3	+7

Execution Collapse & Profile Adaptation

🔑 Key Finding

S4 is the weakest stage and is largely independent of signal quality. Under the Balanced profile, every model under-trades relative to ground truth (mean turnover ratio 10.9%): unable to use covariance, models default to flat S3 weights, so implied turnover stays tiny. Separately, nearly all models default to aggressive behavior (mean PAS_agg 0.90 vs. PAS_cons 0.72); LLMs handle loose risk constraints well but struggle with strict ones—conservative profiles remain hardest.

Quadrant scatter plot of S2 signal accuracy vs. S4 execution fidelity per model, highlighting models with strong signals but poor execution

S2 (signal) vs. S4 (execution). Hunyuan3-Preview and Qwen3.6-Plus lead S2 yet sit near the bottom on S4; GLM-5.1 combines solid signals with the highest execution score.

AdaptScore vs. PAS by profile. Qwen3.7-Max achieves true cross-profile compliance (AdaptScore 0.028); high-AdaptScore models such as Qwen3.6-Plus reach near-perfect aggressive PAS but collapse under conservative constraints.

Citation

@article{zhao2026portbench,
  title={PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management},
  author={Zhao, Yuxuan and Chen, Sijia and Su, Ningxin},
  journal={arXiv preprint arXiv:2605.27887},
  year={2026}
}

PortBench

A Correlation-Aware, Full-Pipeline Benchmarkfor LLM-Driven Portfolio Management

Background & Motivation

Key Findings

Interactive Browser

Market Base Dataset

Interactive Market Snapshot

Normalized Price Trajectories by Asset Class (2015–2025)

Evaluation Framework

Evaluation Details

At a Glance

Five-Stage Decision Pipeline

CEPS: Cross-Stage Error Propagation Score

CEPS: Cascade vs. Uniform Mediocrity

Investor Profiles & Stress Regimes

Results & Insights

Pipeline Performance: All Three Profiles

Stress Regime Results

Cross-Period Evaluation

Real-Time Evaluation

Static QA Evaluation Results

QA–Pipeline Rank Dissociation

Execution Collapse & Profile Adaptation

Citation

A Correlation-Aware, Full-Pipeline Benchmark
for LLM-Driven Portfolio Management