SQLHarness: technical brief

SQLHarness is an applied AI engineering and natural language processing research project investigating whether explicit semantic structure can improve natural-language-to-SQL systems over a direct call to a frontier large language model.

Research question

Can a system externalize the model's interpretation of a business request into a task-oriented intent graph, connect pieces of that graph to reusable SQL primitives, and use a lightweight model to assemble the final query—while preserving or improving execution accuracy and reducing cost and latency?

The project is not premised on the structured system automatically winning. Its purpose is to measure where explicit structure helps, where direct generation remains better, and which component causes each failure.

Intended architecture

  1. Input: a natural-language business question, database schema, and optional task evidence.
  2. Model A — intent parser: converts the request into a task-specific semantic graph containing entities, operations, filters, comparisons, aggregation, ordering, and relationships. The graph is inspired by Abstract Meaning Representation (AMR), but focuses on SQL-relevant intent rather than general linguistic coverage.
  3. Model B — fragment retriever: represents contextual pieces of the intent graph as numerical vectors (embeddings) and retrieves parameterized SQL primitives. It maps local semantic meaning to possible SQL realizations rather than translating the entire graph at once.
  4. Lightweight assembler: combines the graph, retrieved fragments, and schema into one executable SQL proposal.
  5. Governed execution boundary: parses the SQL abstract syntax tree (AST), checks schema references and read-only policy, applies time and row limits, and executes only allowed queries. The implemented Model Context Protocol (MCP) server controls this execution authority rather than merely wrapping an unrestricted database client.
  6. Evaluation: compares executed results with reference (gold) answers and records accuracy, validation decisions, failure stage, latency, token usage, and cost.

Proposed training approach

The proposed first learning phase would repeatedly sample a capable teacher model on unlabeled training inputs. The goal is to observe the teacher's current solution space: the alternative semantic decompositions and SQL patterns it associates with similar requests.

Those samples bootstrap two learnable relationships:

Reference question–SQL training pairs would then correct behavior inherited from the teacher model. The current working plan is to use gold data first to improve fragment selection, then obtain SQL-consistent graph targets for the gold pairs, and finally use those targets to improve the intent parser. How to derive and supervise these graph targets remains a research decision to validate, not a completed result.

Free-form chain-of-thought is not treated as ground truth. Any teacher-generated semantic representation must follow a constrained schema and be checked against the generated SQL and database schema.

Evaluation design

The principal benchmark is BIRD-SQL. Every system arm receives the same frozen question, evidence, database schema, execution limits, and evaluator.

Primary outcome: official BIRD execution accuracy.

Secondary outcomes: cost per correct query, total model tokens, end-to-end and per-stage latency, validation rejection rate, execution failure rate, abstention rate, and an error taxonomy covering intent parsing, schema linking, retrieval, assembly, and execution.

A separate untouched test set is intended for a future benchmark-scale comparison. Development uses training data and a separate validation split. Published BIRD results will use the official evaluator rather than only the project's local comparator.

Research iterations and resulting decisions

The implementation developed through separate checks of the representation, retrieval, execution boundary, and end-to-end result. Each step narrowed a different uncertainty; a working harness was not counted as evidence that the structured approach outperformed direct generation.

  1. Define the representation: Audit 25 diverse BIRD questions across 11 databases and 27 SQL demands, then revise the intent graph to version 0.2. Directed roles, schema grounding, and bounded generation made graph errors inspectable before retrieval.
  2. Check the experiment machinery: Freeze a 500-example reference-query preflight and run small direct-generation and intent-graph pilots. Reference-query executability and graph validity test infrastructure and representation, not end-to-end model quality.
  3. Run the first structured slice: Extract 48 fragments from seven examples and compare on three excluded examples. Both arms answered two correctly; parser timeout and higher structured usage argued against assuming the decomposition was already efficient.
  4. Isolate retrieval, then compare the expanded system: On the 20-train/five-held-out split, inspect catalog coverage and lexical, dense, and learned ranking separately. Direct generation scored 4/5 while both structured variants scored 3/5; ranking changes did not improve final correctness.
  5. Choose the next work from the failure evidence: Expand structural catalog coverage and distill the parser before more ranker optimization. These remain proposals; the completed study does not demonstrate an accuracy, latency, or token advantage over direct generation.

Implementation and diagnostic findings

The offline fixtures prove that the evaluation and policy paths execute. They are not presented as evidence of model quality.

Study conclusion

The development study and governed execution system are complete. The direct call wins this slice. Explicit structure currently contributes inspectability, failure localization, and governed execution, but does not improve the trade-off among accuracy, latency, and token use.

Future research is deliberately narrower: distill Model A into a smaller model so graph construction no longer requires a frontier-model call, expand exact SQL-fragment catalog coverage beyond 58.82%, and train fragment alignment on substantially more than 16 usable graphs. A full production-API BIRD run remains a separate benchmark-scale follow-on and is not yet established evidence.

Research and engineering scope

The research question is whether explicit semantic representations improve compositional language understanding. The engineering system provides reproducible evaluation, model routing, structured intermediate state, safety policy, bounded execution, tracing, and calibrated measurement.

The five-example slice does not establish benchmark-scale accuracy, cost reduction, or latency improvement. Its measured loss is reported precisely because the current structured pipeline has not earned its additional inference cost.