Research Question
Can developer language produce recognizable concepts, such as configuration, testing, and version compatibility?
When exploring an unfamiliar codebase, developers often need to find code by what it does, even when they don’t know which function names to search for. The research investigates whether real developer commit messages can provide a vocabulary for that search, with categories such as application configuration or software testing.
Using a dataset that pairs these messages with code changes, it groups descriptions of similar changes and checks whether the corresponding code shares recurring structural patterns. The code is represented as abstract syntax trees (ASTs), which describe its grammatical structure. Checking both the descriptions and the code helps distinguish categories that sound related from those that also share an implementation pattern—a step toward giving developers useful, evidence-backed labels for navigating unfamiliar code.
Method: category discovery and verification
Separate changes: Split each commit message into individual changes, each with an action and its target, for separate analysis. Ex. “add tests and update configuration” → “add tests” + “update configuration.”Evidence retained: Each extracted change linked to its original commit message, source repository, and commit.
Discover categories: Convert the target phrases from each extracted change into numerical representations of meaning (embeddings). Group similar targets, then repeat within groups to discover more specific categories.Evidence retained: Every group’s member phrases and their average embedding, which represents the group’s meaning.
Verify structure: Keep a category only when its code examples share a recurring pattern that can be expressed as a rule over a localized part of the AST.Evidence retained: The matching code-tree fragments and runnable rules for detecting the pattern.
Test transfer: Test the learned categories on code from repository families set aside before grouping began. Require both a meaning match and an AST-pattern match to check whether the categories apply beyond their source repositories.Evidence retained: Each predicted category and matching score, alongside the commit’s description of the change and the supporting code.
Evaluate acceptance thresholds: Have a human reviewer assess every accepted prediction without seeing which method produced it or its score. Compare the results with quality requirements fixed before review to decide which matching-score cutoff to use for accepting predictions.Evidence retained: The reviewer’s judgments, metric calculations, uncertainty ranges (confidence intervals), and final threshold decision.
Research scale
Category discovery: 685 code changes from 668 repository families.
Retained categories: 14 of 53 proposed groups passed both meaning and code-structure checks.
Held-out evaluation: 151 code components from 132 repository families excluded from category discovery.
Retained categories and hierarchy
The retained categories include application configuration, automated software testing, version compatibility and metadata, state flags, test-suite infrastructure, configuration setup and paths, test-case artifacts, version lookup, and unit-test infrastructure.
More specific subcategories are retained only if their broader parent category has already passed both the meaning and code-structure checks.
Human evaluation procedure
The initial evaluation used an AI model to assess the pipeline’s outputs. A later human review kept the learned categories and held-out predictions unchanged. With the producing method and score hidden, the reviewer assessed every prediction accepted by either matching-score threshold or by a baseline using exact word matches.
Coverage and precision at 0.62
Coverage at 0.62: Predictions accepted for 35 of 151 code components (23.2%), up from 11.9% at 0.72.
Strict precision at 0.62: 68.6% of accepted predictions judged fully correct, below the 80% minimum set before human review.
Acceptance checks at 0.62
| Check | Observed | Required | Result |
|---|---|---|---|
| Accepted coverage | 23.2% (35/151) | ≥ 20% | Pass |
| Strict human-reviewed precision | 68.6% (24/35) | ≥ 80% | Fail |
| Precision change vs. 0.72 | −9.2 percentage points | ≥ −5 percentage points | Fail |
| Review completeness | Complete | No missing review judgments | Pass |
Decision: retain the 0.72 threshold
Lowering the threshold to 0.62 nearly doubled coverage, but failed both precision requirements. Keeping 0.72 leaves more code components unlabeled to limit the additional errors observed at 0.62.
Context-preparation benchmark
A separate self-benchmark measured how long CodeArchitect takes to prepare code context before generating an answer. Preparing and caching repository summaries in advance reduced median context-preparation time from 33.2 to 12.9 seconds—a 61.3% reduction across three fixed questions on one 36-file, 9.7K-line repository. An immediate repeat prepared context in 0.81 milliseconds. These timings exclude answer generation; advance preparation took a separate median 18.9 seconds, so the improvement moves work ahead of the request rather than eliminating it.
Inspect the benchmark method, raw measurements, and limitations.
Research evidence
Research notebook — Category discovery, recursive grouping, code-structure checks, evaluation on excluded repositories, plots, and failure analysis.
Pre-review evaluation protocol — Evaluation set, review criteria, metrics, and acceptance requirements fixed before human review.
Human-review report — The final decision table, evaluation scope, and limits on what the results establish.
Evaluation results (JSON) — Counts, rates, Wilson confidence intervals, checks, and limitations.
Human review judgments — Individual judgments from the blinded human review, used to calculate the results.
Evidence source manifest — Source fingerprint and assembly details for tracing the review materials to their inputs.
Benchmark method and measurements — Recorded timings, exact calculation, measurement code, and limitations.
Raw timing records — The original JSON and Markdown output plus a plain-language interpretation.
Findings and limitations
| Result | Evidence | Limits |
|---|---|---|
| Fourteen human-readable categories learned from 685 code changes across 668 repository families | Research notebook, retained categories, and source records | Vocabulary concentrates on configuration, testing, versioning, and state metadata |
| 151 held-out components from 132 repository families excluded from category discovery | Fixed evaluation set and evidence source manifest | Measures correctness among accepted predictions, not how many relevant categories are missed |
| Threshold remained at 0.72 after the 0.62 candidate failed both precision requirements | Pre-review evaluation protocol, blinded human judgments, and calculated results | The evaluation set and earlier AI judgments had informed development before human review |
| 61% context-preparation latency reduction | Dated timing records, benchmark code, and three fixed questions | Three-question self-benchmark on one repository |
Limitations: The human judgments are new, but the evaluation set and earlier AI judgments had already informed development. These results do not establish whether developers understand unfamiliar code better or work more effectively with the tool.