energie-sql-bench · v0.1.1

Energie-SQL-Bench

How well do language models answer domain questions about the German power market with T-SQL on a real star schema? 70 intents in German and English, gold SQL and gold result sets, execution-based scoring, guarded read-only execution.

SMARD 2023-01-01 – 2026-09-30 · MaStR to 2026-09 · as of 2026-10-01 · monitor b399d01 · GitHub

No live run yet. Every model row reads pending: live run; the first paid runs arrive with v0.2.0. Numbers are never estimated. The grey bars below are a pipeline test (synthetic, no language model).

Execution accuracy (EX)

amber: model run with 95 % interval · dashed: pending: live run · grey: Pipeline test (synthetic, no language model) · SC: self-correction

Configurations

Configuration	EX (95 % CI)	Range	EX-strict	Validity	Blocked	Abstention P / R	LLM p50 / p95 ms	DB p50 / p95 ms	Tokens	USD / correct	Run
Claude Haiku 5.5 · v1 (full schema)	pending: live run
Claude Haiku 5.5 · v2 (full schema + Few-Shot)	pending: live run
Claude Haiku 5.5 · v3 (full schema + semantic layer)	pending: live run
Claude Haiku 5.5 · v4 (full schema + semantic layer + Few-Shot)	pending: live run
Claude Haiku 5.5 · v5 (oracle tables + semantic layer + Few-Shot)	pending: live run
Claude Haiku 5.5 · v6 (retrieved tables + semantic layer + Few-Shot)	pending: live run
Claude Haiku 5.5 · v4 (full schema + semantic layer + Few-Shot) · self-correction	pending: live run
Claude Sonnet 5.5 · v1 (full schema)	pending: live run
Claude Sonnet 5.5 · v2 (full schema + Few-Shot)	pending: live run
Claude Sonnet 5.5 · v3 (full schema + semantic layer)	pending: live run
Claude Sonnet 5.5 · v4 (full schema + semantic layer + Few-Shot)	pending: live run
Claude Sonnet 5.5 · v5 (oracle tables + semantic layer + Few-Shot)	pending: live run
Claude Sonnet 5.5 · v6 (retrieved tables + semantic layer + Few-Shot)	pending: live run
Claude Opus 5.5 · v4 (full schema + semantic layer + Few-Shot)	pending: live run
Qwen2.5-Coder 7B (Ollama) · v4 (full schema + semantic layer + Few-Shot)	pending: live run
Qwen2.5-Coder 7B (Ollama) · v5 (oracle tables + semantic layer + Few-Shot)	pending: live run

Pipeline test (synthetic, no language model)

No language model: a deterministic generator answers from the gold SQL with fixed, labelled faults. Tests guard, execution, comparison, self-correction, scoring and replay.

Run	Mode	EX	EX-strict	Blocked	Repairs	Error classes
synthetic-v4-test	single shot	68.3 % (82/120)	68.3 %	10	0	E1 8, E2 11, E11 10, E12 9
synthetic-v4-test-sc	self-correction	92.5 % (111/120)	92.5 %	10	29	E12 9

Paired comparisons

A	B	Intents	only A	only B	p (McNemar)
Claude Haiku 5.5 · v1 (full schema)	Claude Haiku 5.5 · v4 (full schema + semantic layer + Few-Shot)	–	–	–	pending: live run
Claude Haiku 5.5 · v4 (full schema + semantic layer + Few-Shot)	Claude Haiku 5.5 · v4 (full schema + semantic layer + Few-Shot) · self-correction	–	–	–	pending: live run
Claude Sonnet 5.5 · v1 (full schema)	Claude Sonnet 5.5 · v4 (full schema + semantic layer + Few-Shot)	–	–	–	pending: live run

Error classes

Class	Meaning	synthetic-v4-test	synthetic-v4-test-sc
E1	syntax or dialect	8	0
E2	hallucinated table or column	11	0
E11	blocked by the guard	10	0
E12	abstention error	9	9

Benchmark

Question set

70 × DE/EN

120 scored items, tiers A 5, T1 12, T2 16, T3 16, T4 11

Safety

SELECT only

bench_reader login, sqlglot guard, 30 s timeout, 10,000 rows

DuckDB parity (CI)

100.0 %

64/64 intents, 23 non-trivial

Provenance of the gold set

drafted by a language model

Questions, gold SQL and check queries were drafted by a language model (Claude) and verified by execution against the warehouse. The owner's review of 20 intents is pending.

Replay

offline

CI re-derives every number from the recordings, without database or API key