Typed Structure Over Flat Text: Why AI Agents Need Traversable Documents

K. Brady Davis CloudSurf Software LLC, Las Vegas, NV, USA brady@cloudsurf.com

Abstract

4. Typed Traversability: A Formal Definition

The format properties that make a document algorithmically navigable by AI agents. These are not nice-to-haves. Each property is a prerequisite for the HBDS algorithm (Section 5), and its absence causes a specific failure mode (Section 3).

4.1 Four Properties

We define typed traversability as the presence of four properties in a document format that collectively enable algorithmic navigation by AI agents. Each property is a prerequisite for one or more phases of the HBDS algorithm (Section 5), and the absence of each causes a specific failure mode (Section 3). The properties are ordered by increasing infrastructure cost.

P1: Declared Block Types. Every content block carries a machine-readable type annotation that is part of the format syntax, not inferred from visual presentation. Types include semantic categories such as decision, requirement, callout, data, code, summary, figure, and quote. In a typed format, ::decision[status=accepted] is unambiguously a decision; in Markdown, a paragraph under a "Decisions" heading may or may not be a decision, and a code block may be an example, a template, a requirement, or a test case (see Section 3.2). P1 enables three algorithmic capabilities: semantic relevance scoring (an agent searching for architectural decisions can filter by block type before reading content), type-based compression (retain ::decision and ::data blocks, drop ::callout[type=tip] blocks when budget is tight), and type-routed processing (route ::data blocks to a schema generator, ::code blocks to a linter, ::decision blocks to a decision tracker). Without P1, all three capabilities require NLP inference over raw text, which is probabilistic, expensive, and error-prone.

P2: Typed Cross-References. Links between documents carry relationship semantics --- depends-on, extends, supersedes, implements, parent-child --- as machine-readable metadata rather than human-readable hyperlink text. An agent encountering a related block with relationship: depends-on can build a directed dependency graph without inferring edge types from link labels or surrounding prose. P2 enables hierarchical traversal (follow parent-child edges to descend a knowledge tree), dependency tracing (given a node, find everything it depends on and everything that depends on it), and knowledge graph construction (build a typed directed graph from the document corpus without NLP entity-relation extraction). Without P2, the agent must resolve string-based paths, hope files exist, parse them from scratch, and guess how documents relate (see Section 3.3). The resulting "graph" is a set of untyped hyperlinks --- navigable by humans who read surrounding context, but not by algorithms that require edge semantics.

P3: Priority Signals. Documents and blocks declare their importance relative to context-budget constraints. Priority can be explicit --- frontmatter fields such as scope: public (high priority) vs. scope: internal (lower priority) and status: active vs. status: deprecated --- or implicit through block type, where the type implies a default priority ordering (e.g., ::decision > ::summary > ::callout[type=info] > ::quote). P3 enables budget-aware reading order (read high-priority documents first when budget is constrained) and graceful degradation (when budget is insufficient for full content, fall back to reading only ::summary blocks, which are explicitly marked as the compressed representation of their parent). Without P3, every paragraph appears equally important to the agent, and budget allocation reduces to positional heuristics --- read the first N tokens, or the last N, or truncate uniformly --- none of which correlate with semantic importance (see Section 3.4).

P4: Pre-Computed Token Costs. Every node in the document hierarchy exposes its token cost before the agent reads its content. Two metrics are required: contentTokens (the cost of reading this node's content alone) and subtreeTokenEstimate (the cost of reading this node plus all descendants). These values are computed at authoring or indexing time and stored in the document metadata or schema. P4 enables budget planning (the agent can determine whether its remaining budget can accommodate a subtree before committing tokens), pre-read budget verification (the can_afford() check in the HBDS budget model, Section 5.3), and cost-aware descent decisions (at each level of the tree, the agent compares subtree costs against remaining budget to select the optimal beam). Without P4, every read is a blind commitment --- the agent cannot know the cost of a document until it has already paid that cost by reading it (see Section 3.4). This is the property most absent from existing formats; even typed formats such as DITA and AsciiDoc (Section 2.5) do not expose token costs.

4.2 The Spectrum

Typed traversability is not a binary property. Document formats fall on a spectrum from fully untyped (no properties present) to fully typed-traversable (all four properties present), with each level adding a property and unlocking a corresponding algorithmic capability. Table 3 defines this spectrum.

Table 3. The typed-traversability spectrum.

The progression is additive: each level subsumes the properties of the level below it. A Level 4 format has everything Level 3 has plus declared block types. A Level 5 format has everything Level 4 has plus pre-computed token costs and schema validation.

The key observation is that the spectrum is algorithm-derived, not aesthetically-derived. The levels are not ranked by "how structured the format looks" but by "which HBDS phases the format can support." The algorithm tells you what the format needs. If an agent requires pre-read budget verification, the format must expose token costs (P4). If an agent requires type-aware relevance scoring, the format must declare block types (P1). The four properties are not an arbitrary wish list --- they are the minimal set required by the four phases of budget-constrained hierarchical descent.

This framing also explains why Markdown (Level 1) and Markdown-with-frontmatter (Level 2) feel adequate for many current workflows: the dominant retrieval paradigm (RAG, Section 2.1) does not use hierarchical descent and therefore does not require properties beyond Level 1. The inadequacy becomes visible only when agents attempt algorithms that depend on declared structure --- at which point the format becomes a capability gate, not a cosmetic choice.

The transition between levels is not uniform in cost. Section 4.3 analyzes the cost-benefit trade-off at each step.

4.3 Cost-Benefit at Each Level

The cost of moving between levels varies by orders of magnitude, and the beneficiaries shift from humans to machines as the level increases.

Level 0 to 1 (plain text to Markdown). Nearly free. Authors gain visual structure (headings, bold, lists) with minimal syntax overhead. Both humans and agents benefit: humans see formatted text, agents see heading hierarchy. This transition has already occurred across the software industry --- Markdown is the de facto standard for technical documentation.

Level 1 to 2 (Markdown to Markdown + YAML frontmatter). Cheap. Adding a YAML frontmatter block (title, status, scope, tags) to existing Markdown files requires minutes per document. Humans benefit from metadata searchability; agents benefit from document-level priority signals (P3). The AGENTS.md and CLAUDE.md ecosystem (Section 2.7) operates at this level, and Lulla et al. [1] demonstrated measurable agent performance gains from even this minimal structure.

Level 2 to 3 (frontmatter to structured context hierarchy). A one-time structural investment. Adopting a documentation architecture such as ARDS v3 [N-ARDS] requires organizing documents into a discovery hierarchy (.context/docs/, .context/agents/, .context/guides/) with typed cross-references between documents. The cost is paid once during initial setup; ongoing maintenance is incremental. Humans benefit modestly --- the directory structure is navigable but not essential for reading. Agents benefit substantially: typed cross-references (P2) enable traversal between documents, and the discovery protocol eliminates file-search overhead.

Level 3 to 4 (structured hierarchy to typed document format). Requires a new parser. Authors must learn a new syntax for declaring block types (::decision, ::data, ::callout[type=warning]). The authoring cost is non-trivial: every document must be written in the typed format rather than plain Markdown. Humans benefit from visual block distinctions (a rendered callout looks different from a data table), but could achieve similar visual results with Markdown conventions. Agents benefit enormously: P1 (declared block types) eliminates the type-ambiguity failure mode (Section 3.2) and enables type-aware relevance scoring, type-based compression, and type-routed processing.

Level 4 to 5 (typed format to typed + schema + token costs). Requires infrastructure. Token costs must be computed and maintained as documents change. Schema validation must be enforced at authoring or commit time. Humans derive essentially zero direct benefit --- token counts are invisible in rendered documents. Agents gain pre-read budget verification (P4), which is the prerequisite for the DESCEND phase of HBDS (Section 5.1).

The pattern is clear: the marginal benefit to humans diminishes at each level, while the marginal benefit to AI agents increases monotonically. The crossover point --- where the next level benefits machines more than humans --- falls between Levels 2 and 3. Everything above Level 2 is primarily for machine consumption. This creates an incentive misalignment: the traditional audience for document format design (human authors) has declining motivation to adopt higher levels, while the emerging audience (AI agents) has increasing need for them. Resolving this misalignment --- through AI-assisted authoring, format migration tooling, or dual-render formats that serve both audiences --- is an open design challenge (Section 8.2).

5. HBDS: A Traversal Algorithm That Requires Typed Structure

5.1 Algorithm Overview

This section presents HBDS (Hierarchical Budget-Constrained Descent Search) not as a standalone algorithmic contribution but as a proof by construction. The claim is precise: if HBDS can be built on typed documents and cannot be built on flat text, then the document format is a capability gate --- a boundary that determines which algorithms are available --- not a stylistic preference. The proof proceeds by exhibiting the algorithm, tracing each phase to the typed-traversability properties it requires (P1--P4, Section 4.1), and demonstrating that no phase can execute on a format lacking those properties. Section 5.2 formalizes the phase-to-property mapping; this section describes the algorithm itself.

HBDS navigates a hierarchical knowledge store under a strict token budget. The context window is treated not as a container but as a non-renewable execution resource: every token consumed by retrieval is a token unavailable for reasoning, generation, or further search. The algorithm operates in four phases --- ORIENT, DESCEND, SYNTHESIZE, and WRITE BACK --- each consuming tokens from a monotonically decreasing budget tracker. A portion of the total budget is reserved for later phases before the first phase begins, ensuring that search cannot consume tokens needed for synthesis or output. The four phases proceed as follows.

Phase 1: ORIENT. The agent classifies the incoming query and selects one or more search roots in the knowledge tree. ORIENT reads only the root index --- a lightweight manifest listing top-level domains with their summaries, keywords, and subtree token estimates. An LLM call maps the query to the most relevant domains (e.g., project_local, general_knowledge, or hybrid) and assigns a confidence score. If confidence exceeds a threshold and a single domain's keyword index produces a high-similarity direct hit, ORIENT short-circuits to that leaf without descent, avoiding multi-level traversal entirely.

ORIENT is inexpensive by design: it reads a single pre-computed index and makes one LLM classification call, typically consuming 200--800 tokens. Its critical requirement is P2 (Typed Cross-References). The root index encodes parent-child relationships between domains, categories, and topics as typed hierarchical edges. Without P2, the agent would face a flat list of files with no hierarchy to orient within --- it could rank files by embedding similarity, but it could not identify which subtree of the knowledge space to enter. The distinction is between selecting a search region (which ORIENT does) and selecting individual documents (which flat retrieval does). The former requires hierarchical structure; the latter does not.

Phase 2: DESCEND. Beginning from the search roots selected by ORIENT, the agent performs an iterative beam search through the knowledge hierarchy. At each level of the tree, DESCEND executes five steps:

1. Read child manifests. For each node in the current beam, the agent reads the children's summaries, keywords, and token estimates from the parent's index. These manifests are pre-computed and lightweight --- they expose the information needed for scoring and budget planning without requiring the agent to read full content.

2. Score candidates. The LLM evaluates each candidate's relevance to the query on a 0--1 scale, given only the candidate's summary and keywords. This scoring step is analogous to the heuristic function h(n) in A* search: it provides an admissible estimate of a node's value without expanding it. The LLM acts as a learned heuristic, replacing the hand-crafted heuristics of classical search with a general-purpose relevance estimator. Crucially, the scoring operates on typed summaries and typed keywords --- metadata that exists because the document format declares it (P1), not because an extraction model inferred it.

3. Select beam. The top-k candidates (where k is the beam width, default 3) are selected for expansion. Candidates scoring above a prune threshold but below the beam cutoff are retained in the queue with reduced priority for potential revisitation. Candidates below the prune threshold are discarded.

4. Pre-read budget verification. Before reading any child node's content, the agent checks whether the remaining budget can accommodate the node's declared token cost. This check is the operation can_afford(node.contentTokens), which requires P4 (Pre-Computed Token Costs). If the token cost of a leaf node exceeds the available budget, the agent does not read the full content. Instead, it falls back to summary-only retrieval --- recording the node's summary (which was already loaded during manifest reading) as a partial result with a discounted relevance score. This graceful degradation requires both P1 (the format distinguishes ::summary blocks from full content) and P3 (priority signals indicate that summaries are the compressed representation intended for budget-constrained consumption). Without P4, the agent cannot perform this check: it must read the node to discover its cost, at which point the tokens are already spent. Without P1 and P3, there is no summary layer to degrade to.

5. Deduct tokens and continue. Tokens consumed by manifest reads, LLM scoring calls, and content reads are deducted from the budget tracker. If the remaining budget falls below a configurable minimum-useful-budget threshold, descent terminates regardless of remaining candidates.

For internal (non-leaf) nodes in the beam, DESCEND expands them by reading their child manifests and adding those children to the candidate queue, inheriting the parent's relevance score attenuated by a depth-decay factor (default 0.85). This depth penalty biases the search toward breadth over depth, favoring shallow coverage of multiple relevant subtrees over deep exploration of a single branch.

Phase 3: SYNTHESIZE. After descent completes (either by exhausting the beam, the budget, or the maximum depth), the agent combines the retrieved knowledge fragments into a coherent context injection. Retrieved fragments are ranked by relevance score. Fragments of the same block type from overlapping subtrees are deduplicated --- P1 (Declared Block Types) enables this deduplication because the agent can detect when two ::decision blocks or two ::data blocks cover the same entity, whereas flat text deduplication must rely on string similarity, which conflates semantic overlap with textual similarity. The ranked, deduplicated fragments are packed into the synthesis prompt up to the remaining token budget. An LLM call generates a structured output: an answer, a confidence score, a list of gaps (aspects of the query not covered by retrieved knowledge), and a list of new insights (deductions not explicitly stated in any source).

Phase 4: WRITE BACK. If the SYNTHESIZE phase identified new insights --- knowledge generated during the search that did not exist in any retrieved node --- the WRITE BACK phase persists them to the knowledge tree. For each insight, an LLM call determines placement: which parent node in the hierarchy should own the new knowledge. This placement decision requires P2 (Typed Cross-References), because the agent must identify the correct location in a typed hierarchy, not simply append to a flat file. If a node covering similar content already exists (detected via keyword overlap), the insight is appended to that node with a version increment; otherwise, a new leaf node is created. New agent-generated nodes receive a trust score of 0.6 (below the 0.9 default for human-authored nodes), a source tag of agent-write-back, and an initial contentTokens count computed at write time. Trust scores decay over time (multiplied by 0.997 per day, halving approximately every 231 days) and are boosted by subsequent verification events --- an agent or human reading the node and confirming its accuracy.

The WRITE BACK phase is what makes the knowledge tree self-improving. Over time, frequently queried subtrees accumulate synthesized knowledge, updated summaries, and refined trust scores, reorganizing around actual usage patterns. This self-organization is only possible because P1 tells the agent what kind of knowledge to write (a decision, a pattern, a data schema), P2 tells it where in the hierarchy to place it, and P4 (token costs) is updated to reflect the new content so that future budget calculations remain accurate. A flat-text corpus cannot self-organize because there is no declared structure to organize into --- no hierarchy for placement, no types for classification, no token costs for budget recalculation.

5.2 Why Each Phase Requires Typed Structure

Section 5.1 described the four phases of HBDS in operational terms. This section establishes the formal dependency between each phase and the typed-traversability properties defined in Section 4.1. The argument is structural: each phase requires at least one property that flat text does not provide. Removing any single property disables at least one phase. Therefore all four properties are jointly necessary, and any format lacking even one of them cannot support the full algorithm.

Table 4 presents the phase-to-property dependency map.

Table 4. HBDS phase-to-property dependency. Each row identifies a phase or sub-phase, the typed-traversability property it requires, and the specific mechanism that fails in flat text.

We now walk through each row to explain why the dependency is strict --- not a performance optimization but a capability prerequisite.

ORIENT requires P2. ORIENT selects a search root by mapping the query to a position in the knowledge hierarchy. This operation presupposes that a hierarchy exists --- that documents are organized into parent-child relationships with typed edges (domain contains categories; category contains topics). In flat text, documents are a bag of files in a directory tree. Directory paths provide implicit nesting, but directory names carry no relationship semantics: a file at docs/auth/tokens.md may depend on, extend, or contradict docs/auth/sessions.md, and the path encodes none of these relationships. Without P2, ORIENT degrades to file-level ranking by embedding similarity --- it can select documents but cannot select a region of the knowledge space to descend into. The hierarchical search that follows has no root to start from.

DESCEND (scoring) requires P1. At each level of the beam search, the LLM scores candidate nodes by relevance. The scoring operates on typed metadata --- summaries and keywords associated with declared block types. In a typed format, the agent knows that a candidate node contains a ::decision block, a ::data block, and two ::callout[type=info] blocks before reading any of their content. It can score the node's likely relevance to a query about architectural decisions based on the presence of a ::decision block alone. In flat text, the agent has no such signal. A Markdown file's relevance to an architectural-decision query can only be assessed by reading its content and inferring whether any paragraph constitutes a decision --- the very token expenditure that scoring is meant to avoid. Without P1, relevance scoring becomes relevance reading, collapsing the distinction between evaluating a node and expanding it.

DESCEND (budgeting) requires P4. The pre-read budget verification step --- can_afford(node.contentTokens) --- requires knowing the token cost of a node before committing tokens to read it. This cost must be pre-computed and exposed in the document metadata. In flat text, token count is a derived property: the agent must tokenize the document to know its length, which requires reading it, which spends the tokens. The verification becomes a tautology --- "can I afford what I have already paid for?" --- and the budget model loses its planning capability. Without P4, every descent step is a blind commitment. The agent cannot compare subtree costs across beam candidates, cannot reserve tokens for later phases, and cannot terminate descent early based on projected budget exhaustion.

DESCEND (degradation) requires P1 + P3. When the remaining budget is insufficient for a node's full content, HBDS falls back to summary-only retrieval. This graceful degradation depends on two properties jointly. P1 (Declared Block Types) is required because the format must distinguish ::summary blocks from full content blocks --- the agent needs to know which fragment is the summary. P3 (Priority Signals) is required because the format must indicate that summaries are the preferred fallback under budget pressure --- that they are the author-declared compressed representation, not an arbitrary excerpt. In flat text, there is no declared summary layer. The agent's only compression option is truncation (read the first N tokens) or LLM-generated summarization (spend additional tokens to compress what was just read). Neither is graceful: truncation is semantically blind, and on-the-fly summarization consumes the budget it was meant to conserve.

SYNTHESIZE requires P1. The synthesis phase deduplicates retrieved fragments before assembling them into a context injection. Type-aware deduplication detects when two ::decision blocks or two ::data[format=table] blocks from overlapping subtrees cover the same entity, even if their prose differs. In flat text, deduplication falls back to string or embedding similarity, which conflates textual similarity (two paragraphs using similar words) with semantic overlap (two paragraphs making the same claim in different words). The false-positive rate rises because stylistically similar but semantically distinct passages are incorrectly merged, and the false-negative rate rises because semantically identical claims phrased differently are retained as duplicates. P1 constrains the comparison: two blocks can only be duplicates if they share the same type and cover the same entity, a much tighter predicate than general textual similarity.

WRITE BACK requires P2. Persisting new knowledge requires placement --- the agent must determine which parent node in the hierarchy should own the new insight. This placement decision navigates the same typed hierarchy that ORIENT entered and DESCEND traversed. In flat text, the agent has no insertion semantics. It can append to a file, but it cannot determine which file, which section, or what relationship the new content has to existing content. The result is either a flat append log (new knowledge accumulates at the bottom of a file, destroying any implicit organization) or no write-back at all (the knowledge is generated and discarded). Without P2, the knowledge tree cannot self-improve because there is no tree to improve --- only a collection of files with no declared structure to organize into.

The proof. The six rows of Table 4 collectively reference all four typed-traversability properties: P1 appears in four rows (DESCEND scoring, DESCEND degradation, SYNTHESIZE, and implicitly in WRITE BACK's type classification of new knowledge), P2 appears in two (ORIENT, WRITE BACK), P3 appears in one (DESCEND degradation), and P4 appears in one (DESCEND budgeting). Removing any single property disables at least one phase: without P1, scoring and synthesis fail; without P2, orientation and write-back fail; without P3, graceful degradation fails; without P4, budget verification fails. All four properties are therefore jointly necessary for full HBDS execution.

Flat text --- Markdown, HTML, plain prose --- provides none of the four properties. It lacks declared block types (P1), typed cross-references (P2), priority signals (P3), and pre-computed token costs (P4). The algorithm requires all four. The format provides zero. Therefore flat text cannot support HBDS. The document format is not a style preference --- it is a capability gate that determines whether budget-constrained hierarchical descent is possible at all.

5.3 Formal Budget Model

The DESCEND phase described in Section 5.1 references a budget tracker that gates every read operation. This section formalizes that tracker as a data structure, making explicit the invariants that distinguish budget-constrained descent from unconstrained retrieval. The model is presented in Rust-style pseudocode to emphasize the type-level constraints:

struct BudgetTracker {
    total: usize,       // Total tokens for this agent session
    consumed: usize,    // Tokens consumed (monotonically increasing)
    reserved: usize,    // Tokens reserved for SYNTHESIZE + WRITE BACK
}

impl BudgetTracker {
    fn available(&self) -> usize {
        self.total - self.consumed - self.reserved
    }

    fn can_afford(&self, node: &TreeNode) -> bool {
        node.content_tokens <= self.available()
    }

    fn can_afford_subtree(&self, node: &TreeNode) -> bool {
        node.subtree_token_estimate <= self.available()
    }
}

Three fields capture the complete state of the agent's resource position. The total field is fixed at session initialization --- it represents the model's context window capacity minus the tokens already occupied by the system prompt, task description, and any injected instructions. The consumed field tracks all tokens that have entered the context window through retrieval operations (manifest reads, content reads, LLM scoring prompts). Critically, consumed is monotonically increasing. The context window is a one-way resource: once tokens are read, they cannot be unread. There is no garbage collection, no deallocation, no reclamation of spent budget. Every retrieval decision is a permanent commitment. This monotonicity is what makes pre-read verification essential --- a wrong read cannot be reversed, only survived.

The reserved field addresses a subtler problem. Without reservation, an agent running DESCEND could consume its entire budget on retrieval, leaving zero tokens for the SYNTHESIZE phase (which requires an LLM call to combine fragments) and the WRITE BACK phase (which requires an LLM call to determine placement and generate output). The reserved field is set during initialization based on estimated synthesis and write-back costs --- typically 15--25% of the total budget --- and is treated as unavailable by the available() method. This ensures that descent terminates with sufficient budget remaining for the agent to use what it retrieved.

The two verification methods --- can_afford() and can_afford_subtree() --- are the formal interface between the budget model and the DESCEND phase. The can_afford() method checks whether a single node's content fits within the remaining budget by comparing node.content_tokens against available(). The can_afford_subtree() method checks whether an entire subtree --- the node plus all its descendants --- fits, using node.subtree_token_estimate. The subtree check enables a stronger planning guarantee: before expanding an internal node, the agent can verify that the entire branch is affordable, not just the immediate children.

Both methods depend on fields that exist only in typed document schemas satisfying P4 (Pre-Computed Token Costs, Section 4.1). The content_tokens field is a pre-computed count of the tokens in a node's content, stored in the document metadata at authoring or indexing time. The subtree_token_estimate field is the sum of content_tokens across all descendants, propagated up the hierarchy. These are P4 fields --- they have no equivalent in flat text. A Markdown file does not declare how many tokens it contains; a heading in a Markdown document does not declare the token cost of all content under that heading.

Without P4, the agent cannot call can_afford() before reading. It must read the node to discover its token cost, at which point the tokens are already consumed and the check is moot. Every read becomes a blind commitment: the agent commits an unknown number of tokens from a finite, non-renewable budget, with no ability to verify affordability in advance. The budget model degenerates from a planning instrument into a post-hoc accounting ledger that can detect overruns but cannot prevent them. This is the difference between a budget and a receipt. HBDS requires a budget. Flat text can only provide a receipt.

5.4 Comparison: HBDS vs. Flat-Text Retrieval

The preceding sections described HBDS's phases, their typed-structure dependencies, and the formal budget model. To situate these contributions within the retrieval landscape, we now compare HBDS against two baselines: standard RAG (which operates on flat, unstructured text) and the hierarchical inferred-structure family represented by RAPTOR [5] and LATTICE [6] (which construct tree structure from flat text at indexing or retrieval time). Table 5 presents this comparison across eight dimensions.

Table 5. Three-way comparison of retrieval approaches across eight dimensions. RAG operates on flat text with no structure. RAPTOR and LATTICE infer structure from flat text at indexing or retrieval time. HBDS operates on structure declared at authoring time.

HBDS holds a clear advantage on six of the eight dimensions. We walk through the most consequential.

Structure source and accuracy. The fundamental distinction is between declared and inferred structure. The author who writes a ::decision[status=accepted] block knows it is a decision with certainty; an NLP model classifying the same content operates probabilistically. RAPTOR's clustering algorithm groups chunks by embedding proximity, which correlates with but does not guarantee semantic coherence --- two paragraphs about different decisions may cluster together because they share terminology, while a decision and its rationale may be split across clusters because they use different vocabulary. LATTICE's LLM-guided tree construction is more accurate than embedding-based clustering but remains a classification task subject to model error. In contrast, declared structure is exact: the hierarchy is what the author specified, validated against a schema at authoring time. No inference step intervenes between the author's intent and the agent's perception of document organization.

Budget control. This is the starkest contrast. RAG provides no budget mechanism --- it retrieves top-k chunks regardless of their aggregate token cost, and the agent discovers whether the retrieved content fits its context window only after injection. LATTICE introduces a principled budget by limiting node expansions during traversal [6], but this budget measures a computational resource (how many LLM scoring calls the traversal makes), not the context-window resource the agent must manage. An agent using LATTICE cannot answer "can I afford to read this subtree without overrunning my context window?" because node expansion count does not map to token cost. HBDS budgets the resource that actually constrains the agent --- tokens --- with pre-read verification that prevents overcommitment (Section 5.3). The can_afford() and can_afford_subtree() checks transform budget management from post-hoc accounting into prospective planning.

Graceful degradation. When remaining budget is insufficient for full content, RAG's only recourse is truncation or omission --- discarding chunks by rank, with no guarantee that the discarded content was less important than the retained content. RAPTOR and LATTICE offer partial degradation: the agent can read a higher-level summary node in the inferred tree instead of descending to leaves. These summaries are machine-generated approximations whose fidelity depends on summarization quality. HBDS degrades to author-declared ::summary blocks --- compressed representations the author explicitly wrote as the budget-constrained view of their content. The degradation is typed and intentional, not approximate and automatic.

Knowledge accumulation. Neither RAG nor RAPTOR/LATTICE has a write-back mechanism; every query starts from the same static index regardless of how many prior queries have been processed. HBDS's WRITE BACK phase (Section 5.1) deposits new knowledge, updates summaries, and adjusts trust scores after each retrieval session. Over time, frequently queried subtrees accumulate synthesized knowledge, making subsequent retrievals faster and more precise. This self-improvement is unique among the three approaches and depends on all four typed-traversability properties (Section 5.5).

The honest trade-off. The eighth dimension --- cost of structure --- is the one where HBDS is disadvantaged. RAG pays nothing for structure because it uses none. RAPTOR and LATTICE pay at indexing or retrieval time: LLM calls for clustering, summarization, and tree construction, costs that recur whenever the corpus changes. HBDS pays at authoring time: the author must write typed blocks, declare cross-references, and maintain a format that supports schema validation and token costing. This cost is non-zero and should not be minimized. It requires authoring discipline, tooling support, and a format that provides typed syntax without prohibitive ceremony --- the design gap identified in Section 2.5. However, the critical economic observation is that authoring cost is paid once per document, while retrieval cost (whether RAG's embedding pipeline or RAPTOR/LATTICE's indexing overhead) is paid once per corpus update or query. For a corpus in active use --- which describes any knowledge base queried by AI agents --- the amortized per-query cost of declared structure converges to zero, while the per-query cost of inferred structure remains constant. Section 4.3 analyzed this cost-benefit curve across the typed-traversability spectrum.

The conclusion is precise: on a typed corpus, HBDS is strictly more capable than flat or inferred-structure retrieval on six of eight dimensions, comparable on relevance scoring (both HBDS and LATTICE use LLM heuristics, though HBDS operates on higher-fidelity inputs), and disadvantaged only on the upfront cost of producing typed documents. The question is not whether HBDS outperforms flat retrieval when typed structure is available --- it does, by construction. The question is whether the authoring investment required to produce that structure is justified by the retrieval gains. For corpora queried infrequently or written once and discarded, the answer may be no. For knowledge bases in sustained use by AI agents --- the setting this paper addresses --- the amortization arithmetic strongly favors declared structure.

5.5 Self-Organizing Property

The WRITE BACK phase (Section 5.1) transforms HBDS from a stateless retrieval algorithm into a self-organizing knowledge system. Unlike RAG, RAPTOR, and LATTICE --- all of which are stateless, treating every query as an independent retrieval event with no memory of prior searches --- HBDS modifies the knowledge tree through use. Over time, the hierarchy reorganizes around actual query patterns, accumulating institutional knowledge that improves future retrieval. This self-organization operates through three mechanisms, each dependent on specific typed-traversability properties.

Knowledge Deposition. When the SYNTHESIZE phase generates new insights --- deductions, cross-references, or consolidated findings not explicitly stated in any source node --- the WRITE BACK phase persists them as new leaf nodes in the knowledge tree. Each deposited node carries a declared block type (P1): a synthesized architectural pattern is written as a ::decision block, an extracted data relationship as a ::data block, a condensed finding as a ::summary block. The block type is not a label applied after the fact; it determines how future DESCEND phases will score and retrieve the node. An agent searching for decisions will find deposited ::decision blocks through type-aware relevance scoring; an agent searching for data schemas will find deposited ::data blocks. Without P1, the agent would write untyped prose, and future agents would face the same type-ambiguity problem (Section 3.2) on agent-generated content that they face on flat human-authored text.

Summary Updating. Nodes that are frequently retrieved accumulate retrieval statistics. When a node's retrieval count exceeds a threshold, the WRITE BACK phase regenerates its summary to better reflect the query patterns that led to its retrieval. This updated summary improves future DESCEND scoring, because the summary is what the LLM-as-heuristic evaluates during beam search (Section 5.1, step 2). The placement of the updated summary requires P2 (Typed Cross-References): the agent must locate the correct node in the hierarchy, identify its parent-child relationships, and write the summary to the appropriate position in the index manifest. P4 (Pre-Computed Token Costs) must then be updated to reflect the new summary length, ensuring that future budget calculations remain accurate. Without P2, there is no hierarchical position to update; without P4, future agents will make budget decisions based on stale token counts.

Trust Score Adjustment. Agent-generated nodes enter the knowledge tree at a trust score of 0.6, below the 0.9 default for human-authored content (Section 5.1). This asymmetry reflects the principle that synthesized knowledge requires validation. Trust scores evolve through two opposing forces: temporal decay (0.997 per day, halving approximately every 231 days) and verification events (an agent or human reads the node and confirms its accuracy, boosting the score). Over time, accurate agent-generated knowledge converges toward human-authored trust levels, while unverified or contradicted knowledge decays below retrieval thresholds and is effectively deprecated from the active hierarchy. The trust mechanism depends on P1 because the verification agent must determine what kind of claim it is validating --- a ::decision block requires different validation criteria than a ::data block or a ::summary block.

These three mechanisms are collectively impossible on a flat text corpus, for a structural reason rather than an implementation one: there is no hierarchy to organize into. Knowledge deposition requires a typed hierarchy for placement (P2) and typed blocks for classification (P1). Summary updating requires addressable nodes with parent-child relationships (P2) and token cost metadata (P4). Trust scoring requires typed blocks for type-appropriate validation (P1). A flat corpus --- a directory of Markdown files --- offers no insertion semantics, no hierarchical addressing, no typed block boundaries, and no token cost metadata. An agent could append text to a file, but it cannot place knowledge in a structure, because no structure exists.

The contrast with existing retrieval systems is stark. RAG is stateless: every query re-embeds, re-retrieves, and re-injects with no memory of prior searches. RAPTOR rebuilds its cluster tree at indexing time, not at query time; queries do not modify the tree. LATTICE traverses its semantic hierarchy but does not write back to it. In all three systems, the corpus after 1,000 queries is identical to the corpus before the first query. In HBDS, the corpus after 1,000 queries is richer --- it contains synthesized knowledge deposited by prior searches, refined summaries tuned to actual query patterns, and trust-validated agent contributions that have survived temporal decay. This accumulation of institutional knowledge through use is, over the long term, the strongest advantage of typed-structure retrieval over flat-text retrieval.

We note that the self-organizing property is forward-looking. While the mechanisms are specified and the WRITE BACK phase is architecturally defined (Section 5.1), longitudinal empirical validation --- demonstrating that write-back measurably improves retrieval quality over time --- is identified as future work in Section 8.2. The claim here is structural: the typed-traversability properties enable self-organization that flat text prohibits. Whether the enabled capability delivers the expected long-term gains is an empirical question that requires sustained deployment and measurement.

6. Empirical Evidence

Three controlled studies plus production telemetry. Studies 1--2 test the thesis directly. Study 3 tests HBDS specifically.

6.1 Study 1: Context Retrieval Accuracy

Study 1 isolates the effect of typed document structure on an agent's ability to answer factual questions about a software project, holding content constant across conditions. The study has not yet been executed; this section specifies the design with sufficient detail for replication.

Corpus Construction. The experimental corpus is drawn from a production software project with 200+ documentation files covering architecture, API specifications, operational runbooks, decision records, and onboarding guides. Two conditions are prepared from identical source content:

• Condition A (Flat Markdown). All documents are rendered as standard Markdown files with heading hierarchies, inline links, and code fences. No frontmatter, no typed blocks, no declared cross-references. This represents the Level 1 baseline on the typed-traversability spectrum (Section 4.2).
• Condition B (Typed .surf). The same content is restructured into typed .surf documents. Each semantically distinct block receives a declared type: ::decision for architectural decisions, ::data[format=table] for structured data, ::callout[type=warning] for operational caveats, ::code[lang=rust] for implementation examples, ::summary for compressed representations. Documents carry YAML frontmatter with status, scope, and tags fields (P3), related blocks with typed relationship edges (P2), and pre-computed contentTokens values (P4). This represents Level 5 on the spectrum.

Content equivalence between conditions is enforced by automated validation: every prose sentence, code block, and data table in Condition A has a corresponding block in Condition B, verified by a diffing tool that compares stripped-text representations. The typed condition adds structure, not content.

Question Generation and Validation. Fifty factual questions are generated through a two-stage process. First, two independent human annotators read the corpus and each produce 40 candidate questions spanning five categories: architectural decisions (e.g., "Which database engine did the team select for the auth service, and what was the stated rationale?"), API specifications, operational procedures, dependency relationships, and status/timeline facts. Second, the annotators cross-validate each other's questions against the corpus, retaining only questions whose answers are (a) unambiguously supported by at least one document, (b) answerable from the documentation alone without external knowledge, and (c) distributed across at least 30 distinct source files to prevent retrieval shortcuts. The final set of 50 questions is selected by stratified sampling: 10 per category. Each question has a gold-standard answer with supporting document citations.

Experimental Design. Table 6 summarizes the conditions, metrics, and hypotheses.

Table 6. Study 1 experimental design.

Four metrics are measured per question:

1. Answer accuracy --- binary correctness judged by two independent raters against the gold-standard answer, with inter-rater disagreements resolved by a third rater. Cohen's kappa is reported as a reliability measure.
2. Token consumption --- total tokens consumed during retrieval and reasoning, measured from the agent's API usage log. Lower is better, indicating more efficient navigation.
3. Hallucination rate --- the proportion of factual claims in the agent's answer that are not supported by any document in the corpus. Each claim in each answer is extracted and independently traced to a source document (or marked unsupported) by a human annotator.
4. Retrieval precision --- the proportion of retrieved document content (measured in tokens) that is relevant to the question, judged by a human rater reviewing the agent's retrieval log.

Hypotheses. The primary hypothesis is:

• H1: Condition B (typed .surf) produces a lower hallucination rate than Condition A (flat Markdown) at equal or higher accuracy. The predicted mechanism is the elimination of type-ambiguity errors (F2, Section 3.2): when block types are declared, the agent can distinguish a ::decision block from a ::callout[type=tip] from a ::quote without inferring the distinction from context, reducing the probability of treating speculative content as authoritative or vice versa.

Two secondary hypotheses follow from the typed-traversability properties:

• H2: Condition B consumes fewer tokens per question than Condition A. The mechanism is P3 (Priority Signals) and P1 (Declared Block Types): the agent can filter by block type and document scope before reading full content, reducing the volume of irrelevant material loaded into the context window.
• H3: Condition B achieves higher retrieval precision than Condition A. The mechanism is the same as H2 --- type-aware filtering retrieves less but more relevant content.

Expected Effect Sizes. Based on the 16.6% token reduction observed by Lulla et al. [1] from structured AGENTS.md files alone (Level 2 structure), we conservatively estimate a 20--30% token reduction for Level 5 structure (Condition B), which adds P1 and P4 atop the structural properties Lulla et al. tested. Hallucination rate reduction is harder to estimate from prior work; we target detection of a medium effect size (Cohen's d >= 0.5) with 50 paired observations per condition, which provides statistical power of approximately 0.87 at alpha = 0.05 for a paired-samples t-test on the per-question hallucination count. If the true effect is smaller, the study is underpowered and should be replicated with a larger question set.

Controls. The agent model, system prompt, available tools, and token budget are held constant across conditions. Questions are presented in randomized order within each condition. To mitigate order effects, half of the trials run Condition A first and half run Condition B first (counterbalanced within-subjects design). Each question-condition pair is run three times to account for LLM stochasticity, and results are averaged across runs.

6.2 Study 2: Knowledge Graph Traversal

Study 2 isolates P2 (Typed Cross-References, Section 4.1) by measuring an agent's ability to trace dependency chains --- a task that requires traversing relationships between documents rather than retrieving content within them. Where Study 1 tests whether block types (P1) improve retrieval accuracy for factual queries, Study 2 tests whether typed edges improve the agent's ability to reconstruct a knowledge graph from a document corpus under a constrained token budget.

Corpus Construction. We construct a synthetic software-architecture corpus of 24 interconnected documents describing a microservices system: authentication, authorization, session management, API gateway, user profiles, billing, notifications, audit logging, and supporting infrastructure (database, cache, message queue, monitoring, deployment). Each document describes one component. The corpus is designed with a known ground-truth dependency graph containing 47 directed edges across four relationship types: depends-on (component A requires component B at runtime), extends (component A adds capabilities to component B's interface), supersedes (component A replaces a deprecated component B), and produces-for (component A generates data consumed by component B). The graph is acyclic within each relationship type but contains cross-type cycles (e.g., auth depends-on database; database monitoring produces-for audit logging; audit logging depends-on auth). Maximum chain depth is 6 hops. Each document averages 800 tokens; the full corpus totals approximately 19K tokens.

Ground Truth Establishment. Two software engineers independently annotate the dependency graph from the corpus documents. Disagreements are resolved by a third annotator. Inter-annotator agreement (Cohen's kappa) is computed and reported. Only edges present in the majority annotation (2 of 3 annotators) are included in the gold-standard graph. This produces a set of (source, target, relationship-type) triples against which agent output is evaluated.

Conditions. Both conditions contain identical textual content.

• Condition A (Flat Markdown). Each component is a .md file with standard Markdown hyperlinks: [see Authentication](./auth.md), [depends on Session Manager](./sessions.md). Hyperlinks carry no relationship semantics; the agent must infer whether a link represents a dependency, an extension, a supersession, or a data-flow relationship from surrounding prose --- the failure mode described in Section 3.3.
• Condition B (Typed .surf). The same content is represented with declared related blocks in YAML frontmatter, each carrying an explicit relationship field (e.g., relationship: depends-on, relationship: supersedes). In Condition B, the agent can extract a typed directed graph by parsing frontmatter alone --- approximately 50 tokens per document --- without reading document bodies. In Condition A, extracting any graph requires full-document reads at approximately 800 tokens each.

Content equivalence is enforced by automated diffing of stripped-text representations, identical to the validation procedure in Study 1.

Task. The agent receives one of five seed queries requiring multi-hop dependency tracing (e.g., "What does the authentication system depend on, and what depends on it?" or "Trace all components that would be affected if the database schema changes"). Each query requires traversal to a depth of at least 3 hops and a report of the complete reachable subgraph from the seed component. The agent operates under a 16K-token budget --- sufficient to read approximately 20 of 24 documents in full (Condition A) or all 24 documents' frontmatter plus 12 full documents (Condition B) --- forcing selective traversal.

Table 7. Study 2 experimental design.

Metrics. Four metrics capture distinct facets of traversal quality:

1. Traversal completeness (recall) --- the fraction of ground-truth (source, target, relationship-type) triples present in the agent's output.
2. False-positive rate (1 - precision) --- the fraction of agent-reported edges absent from the ground truth, including hallucinated edges (no relationship exists between the reported components) and mistyped edges (correct component pair, wrong relationship type).
3. Relationship-type accuracy --- among correctly identified component pairs, the fraction where the agent also identifies the correct relationship type. This metric isolates the type-inference challenge: a depends-on edge reported as extends counts as a type error even though the component pair is correct.
4. Traversal depth --- the maximum number of hops the agent follows from the seed component, measured against tokens consumed. This captures the efficiency of traversal: how deep the agent reaches per token spent.

Hypotheses.

• H1 (completeness): Condition B achieves higher traversal completeness than Condition A within the same token budget. The mechanism is that typed frontmatter enables graph extraction at ~50 tokens per document (Condition B) versus ~800 tokens for full-document reading and inference (Condition A) --- a 16x efficiency gain per traversal step that allows deeper exploration within the budget.
• H2 (false positives): Condition B produces a lower false-positive rate than Condition A. The mechanism is that declared relationship types (P2) eliminate the need to infer edge semantics from prose, removing the type-ambiguity failure mode (Section 3.2) at the inter-document level.
• H3 (type accuracy): The effect of Condition B is strongest on relationship-type accuracy, because this metric isolates the specific capability P2 provides --- machine-readable relationship semantics --- that Condition A entirely lacks.

Expected Effect Sizes. Prior work on structured knowledge representations for question answering reports 15--30% recall improvements over unstructured baselines [N-KG-QA]. We anticipate a traversal-completeness gain of 20--35 percentage points (Condition B over A) and a false-positive reduction of 40--60%, driven by the elimination of relationship-type inference errors. The traversal-depth effect is expected to be the largest (1.5--2x deeper within the same budget) due to the 16x per-document efficiency gain from frontmatter-only graph extraction.

Statistical Plan. With 5 queries x 10 runs = 50 observations per condition, a two-sided Mann-Whitney U test achieves >0.90 power at alpha = 0.05 for the anticipated effect sizes. Results are reported with 95% confidence intervals and effect sizes (rank-biserial correlation). Each query-condition pair is run 10 times to account for LLM stochasticity; within-condition results are averaged across runs before cross-condition comparison.

6.3 Study 3: HBDS vs. RAG on Knowledge-Intensive Generation

Where Studies 1 and 2 test the broader thesis that typed structure improves agent retrieval, Study 3 tests the specific algorithmic claim: that HBDS (Section 5) outperforms standard RAG on a knowledge-intensive generation task where retrieval quality directly determines output quality. The task --- generating a software specification from a natural language description, drawing on a large domain knowledge base --- exercises every HBDS phase. ORIENT must identify the correct domain. DESCEND must navigate a deep hierarchy to retrieve relevant patterns, data models, and compliance constraints. SYNTHESIZE must combine heterogeneous knowledge fragments into a coherent context injection. The task is poorly suited to flat retrieval because the knowledge required is distributed across multiple hierarchy levels and block types --- no single embedding query captures it.

Knowledge Base Construction. A knowledge base of 500+ domain patterns is constructed covering 10 application domains: e-commerce, healthcare, fintech, logistics, education, hospitality, real estate, SaaS analytics, content management, and IoT. For each domain, patterns are organized into five categories: data models (entity schemas and relationships), business rules (domain-specific logic constraints), compliance requirements (regulatory and security obligations), UI patterns (standard interaction flows), and integration patterns (third-party service interfaces). Each pattern is authored as a self-contained unit with a title, summary, full specification, and cross-references to related patterns. The result is a 10 x 5 matrix of categories, each containing 8--12 patterns, totaling approximately 500--600 patterns. The knowledge base is constructed once and used in both conditions, ensuring content equivalence.

Conditions. Two conditions share identical knowledge content but differ in format and retrieval method:

• Condition A (RAG baseline). The 500+ patterns are stored as flat Markdown files, one per pattern. Each file contains the pattern title as a heading, followed by prose description, code examples, and requirement lists in untyped Markdown. Retrieval uses a standard RAG pipeline: documents are chunked at 512 tokens with 64-token overlap, embedded using a sentence-transformer model [N-SentenceTransformers], stored in a vector index, and retrieved via top-k cosine similarity (k = 20) against the task prompt. Retrieved chunks are injected into the generation context in rank order until the token budget is exhausted.

• Condition B (HBDS on typed documents). The same 500+ patterns are stored as typed .surf files organized in a three-level hierarchy: domain (level 1), category (level 2), pattern (level 3). Each file uses declared block types (::data for entity schemas, ::code for implementation examples, ::callout[type=warning] for compliance constraints, ::summary for compressed representations) and typed cross-references (related blocks with relationship semantics such as depends-on and implements). Pre-computed token costs are stored in each node's manifest (P4). Retrieval uses HBDS as described in Section 5.1, with the same total token budget as Condition A managed by the BudgetTracker (Section 5.3).

Generation Tasks. Twenty specification prompts are constructed, two per domain: one narrow-scope prompt (e.g., "Generate a spec for a restaurant menu ordering system") and one broad-scope prompt (e.g., "Generate a spec for a full-service restaurant management platform including ordering, inventory, reservations, and staff scheduling"). Broad-scope prompts require retrieval across multiple categories within a domain and test HBDS's ability to traverse laterally across sibling subtrees during DESCEND. Each prompt is executed five times per condition (100 runs per condition, 200 total) to account for LLM generation variance. The same base LLM is used for generation in both conditions; only the retrieval mechanism differs.

Metrics and Evaluation. Four metrics are evaluated against ground-truth requirement checklists constructed independently by domain experts prior to experimental runs.

Table 8. Study 3 experimental design: conditions, retrieval methods, and metrics.

Specification completeness (M1) is the percentage of requirements in the ground-truth checklist present in the generated specification. Checklists contain 15--40 requirements per prompt depending on scope. Two independent raters score each specification; inter-rater disagreements are resolved by a third rater. Structural correctness (M2) is scored on a four-point rubric assessing required sections (data model, business rules, API contracts, error handling), internal consistency of entity references, and absence of contradictory constraints. Compliance coverage (M3) measures the percentage of applicable regulatory and security requirements (PCI-DSS for payment processing, HIPAA for healthcare data, GDPR for user data) included in the output. This metric specifically tests whether the retrieval mechanism surfaces ::callout[type=warning] compliance blocks that are thematically distant from the main query but structurally co-located with relevant domain patterns in the typed hierarchy --- a retrieval path available to HBDS through type-aware descent but invisible to embedding similarity. Retrieval token cost (M4) is the total tokens consumed by the retrieval pipeline before the generation prompt is assembled --- embedding lookups plus chunk injection for Condition A, versus ORIENT + DESCEND + SYNTHESIZE token expenditure for Condition B.

Hypotheses.

• H3a (completeness): HBDS retrieval produces specifications with higher completeness (M1) than RAG retrieval, because hierarchical descent surfaces patterns from multiple categories within a domain rather than relying on embedding proximity to the query string. Broad-scope prompts are expected to show a larger effect than narrow-scope prompts, since they require cross-category retrieval that embedding similarity handles poorly.
• H3b (compliance): HBDS retrieval produces higher compliance coverage (M3), because typed block types enable the DESCEND phase to surface compliance constraints that are semantically distant from the functional query but structurally adjacent to relevant domain patterns in the hierarchy.
• H3c (token efficiency): HBDS retrieval consumes fewer tokens (M4) for equivalent or higher output quality, because pre-read budget verification (Section 5.3) prevents over-retrieval and graceful degradation to ::summary blocks avoids full-content reads when budget is constrained.

Expected Effect Sizes. Prior work comparing structured retrieval against flat baselines on knowledge-intensive generation tasks reports 15--25% improvements in output completeness [N-StructuredRetrieval]. We anticipate a completeness gain of 15--30 percentage points on broad-scope prompts (where cross-category retrieval matters most) and 5--15 points on narrow-scope prompts. Compliance coverage is expected to show the largest effect (25--40 percentage points) because RAG's embedding similarity systematically underweights compliance content that uses regulatory vocabulary orthogonal to functional domain terms. Token cost reduction of 30--50% is anticipated based on the BudgetTracker's pre-read verification preventing the over-retrieval that top-k RAG cannot avoid.

Statistical Plan. With 20 prompts x 5 repetitions = 100 observations per condition, per-metric means are compared using independent-samples t-tests (or Mann--Whitney U if Shapiro-Wilk rejects normality at $\alpha = 0.05$), with Bonferroni correction for four comparisons ($\alpha = 0.0125$). Effect sizes are reported as Cohen's d with 95% confidence intervals. Narrow-scope and broad-scope prompts are analyzed both pooled and separately to test the interaction between prompt scope and retrieval method. A secondary analysis examines per-domain variation to identify domains where HBDS's advantage is largest (expected: domains with heavy compliance requirements such as healthcare and fintech) and smallest (expected: domains with shallow, homogeneous pattern sets).

6.4 Production Telemetry (Observational)

In addition to the controlled studies above, we report observational data from a sustained production deployment of structured context documentation across a real software organization. A bootstrapped software company operating 15 active repositories adopted ARDS v3 --- a structured context standard implementing properties P2 and P3 (Section 4.1) --- incrementally over an eight-week period beginning in late December 2025. At the time of observation, the deployment comprised 54 agent definitions across the 15 repositories, over 1,000 plan documents, and more than 200 context files organized in typed .context/ directory hierarchies with YAML frontmatter, declared cross-references, and a specified discovery order.

Setting. The organization develops multiple software products spanning web applications, native desktop and mobile clients, CLI tools, shared infrastructure crates, and a documentation platform. Repositories range from 6 to over 600 automated tests. AI agents are used for code generation, strategic planning, document drafting, patent preparation, and deployment automation. The workload is representative of a small team using AI agents as force multipliers across a diverse product portfolio.

Baseline. Prior to ARDS adoption, repositories used flat Markdown README files as the sole entry point for AI agents. Agent sessions were observed to spend substantial portions of their context budgets on discovery --- issuing file-search commands, reading irrelevant files, and iterating through directory listings to locate the correct context for a task.

Observations. After ARDS adoption, agents followed a typed discovery protocol (the sequence defined in Section 4.2, Level 3: root index, then typed context files, then cross-referenced documents) rather than searching. Table 9 summarizes the observed changes across four metrics. All figures are approximate ranges derived from session logs; no controlled A/B test was conducted.

Table 9. Observed agent session metrics before and after ARDS v3 adoption across 15 repositories (approximate ranges, not controlled).

Interpretation. The direction of change is consistent across all four metrics and aligns with the theoretical prediction: when discovery follows a declared protocol (P2) with priority signals (P3), agents spend fewer tokens finding the right context and more tokens reasoning about the task. The reduction in wrong-file reads corresponds to the elimination of broken-traversal failures (F3, Section 3.3); the improvement in task completion without correction corresponds to reduced structure hallucination (F1, Section 3.1) and type ambiguity (F2, Section 3.2).

Threats to validity. This observational data carries significant limitations that preclude causal claims. First, the data comes from a single organization with a single primary developer; generalization to larger teams or different domains is unestablished. Second, no controlled conditions were maintained --- ARDS adoption coincided with concurrent workflow improvements, agent prompt refinements, and team learning effects. Any or all of these confounds may contribute to the observed improvements. Third, observer bias is present: the ARDS designer is also the primary user, creating a motivation to interpret ambiguous data favorably. Fourth, the metrics are derived from session-level observation rather than automated instrumentation, introducing measurement imprecision.

Argument for external validity. Despite these limitations, the observational data provides a form of evidence that controlled studies cannot: sustained, real-world operation under production conditions over a meaningful time period. The 15 repositories span diverse domains (web applications, native desktop software, shared Rust crates, mobile applications, strategic planning). The 54 agent definitions cover heterogeneous tasks (code generation, document drafting, competitive analysis, patent preparation). The over 1,000 plan documents represent genuine organizational knowledge, not synthetic benchmarks. The improvements persisted across the full eight-week observation window, suggesting they are not attributable to novelty effects alone.

7. Implications

7.1 For Document Format Designers

The audience for document formats has expanded from one to two. For three decades, format designers optimized for a single consumer: the human reader. The design question was "does this render well?" Markdown answered that question so effectively that it became the dominant format for technical documentation. But AI agents now consume documents autonomously at scale, and their requirements diverge fundamentally from human reading. An agent does not render a document --- it parses it, classifies its components, scores their relevance, estimates their token cost, and decides what to read within a finite budget. The design question for this second audience is not "does this look good?" but "does this parse unambiguously into a typed AST with pre-computed token costs?"

These are not the same question, and optimizing for one does not optimize for the other. Markdown's design --- minimal syntax, visual structure, implicit semantics --- is optimal for human authoring and adequate for human reading. It is systematically inadequate for algorithmic traversal, as the five failure modes in Section 3 demonstrate. Structure hallucination (F1) and type ambiguity (F2) arise because Markdown lacks P1 (declared block types). Broken traversal (F3) arises because Markdown links lack P2 (typed cross-references). Context budget waste (F4) arises because Markdown exposes neither P3 (priority signals) nor P4 (pre-computed token costs). Semantic lossy compression (F5) arises because Markdown provides no typed compression strategy. Each failure mode is a format-level deficiency, not an agent-level deficiency --- no improvement in agent architecture can compensate for information the format does not contain.

The design challenge for new formats is therefore dual-audience: writable by humans with ceremony cost comparable to Markdown, yet algorithmically navigable by machines at Level 5 on the typed-traversability spectrum (Section 4.2). The spectrum provides a concrete design target: all four properties (P1--P4) must be present. The ceremony cost analysis (Section 4.3) identifies the core tension: the transition from Level 3 to Level 4 requires a new parser and new syntax, imposing authoring overhead that benefits machines far more than humans. Format designers must resolve this tension --- through syntax that reads naturally to humans while parsing deterministically for machines, through AI-assisted authoring that absorbs the ceremony cost, or through migration tooling that converts existing Markdown corpora to typed formats. The formats that succeed will be those that achieve Level 5 traversability without LaTeX-level verbosity --- typed structure at Markdown-level ceremony.

7.2 For AI Agent Builders

The dominant pattern in AI agent frameworks today is flat text injection: the agent receives a string of context --- a README, a retrieved chunk, a conversation history --- and reasons over it as undifferentiated prose. This architecture leaves measurable performance on the table. The evidence from Section 6 shows that agents operating on typed documents achieve higher retrieval accuracy, deeper traversal within the same token budget, and lower hallucination rates than agents operating on equivalent flat content. The mechanism is not mysterious: typed structure gives the agent information that flat text withholds.

The immediate practical implication is type-routed processing. An agent that receives a typed AST can dispatch blocks by type rather than inferring purpose from prose. A ::decision block routes to a decision tracker. A ::data block routes to a schema generator or validator. A ::callout[type=warning] block routes to a compliance checker. This routing is deterministic --- it requires a single field read, not an LLM classification call. Flat-text agents must spend tokens inferring what a paragraph is before they can decide what to do with it; typed-document agents skip that inference entirely.

The deeper implication concerns capabilities that flat-text agents structurally cannot achieve, regardless of model quality. Section 5.1 identifies three such capabilities. First, budget-aware search: the ability to verify, before reading a node, whether its token cost fits the remaining budget. This requires pre-computed token costs (P4), which flat text does not expose. Second, graceful degradation: the ability to fall back from full content to summary-only retrieval when the budget is insufficient. This requires a declared summary layer (P1 + P3) that flat text lacks. Third, self-improving knowledge bases: the ability to write synthesized knowledge back into the correct location in a typed hierarchy (P2), with updated token costs (P4) and trust scores. Flat text has no hierarchy to write back into.

The practical recommendation for agent framework designers is direct: typed document consumption should be a first-class input mode, not an afterthought. Agent tool interfaces that accept only raw strings force every upstream system to serialize structure away before the agent sees it --- discarding exactly the information (block types, cross-references, token costs, priority signals) that the evidence shows improves performance. Frameworks that preserve typed ASTs through the agent's input pipeline enable the type-routing, budget-aware search, and graceful degradation patterns described above. The format is not preprocessing; it is architecture.

7.3 For Knowledge Base Platforms

The dominant knowledge base platforms --- Wikipedia (MediaWiki markup rendered to HTML), Confluence (WYSIWYG stored as XHTML), Notion (proprietary JSON behind a rate-limited API), GitBook (Markdown compiled to static HTML) --- share a common architectural property: their content is opaque to external AI agents. An agent seeking to traverse a Confluence space must authenticate to a REST API, paginate through JSON responses, strip HTML tags, and infer document types and relationships from rendered markup. Every platform requires a bespoke adapter. Every adapter is fragile, version-dependent, and lossy --- it discards the semantic structure the platform internally maintains but does not export.

This opacity is not incidental; it is structural. These platforms store content in formats that satisfy P1--P4 internally (Confluence tracks page hierarchies, Notion maintains typed blocks in its database, Wikipedia has category trees) but expose none of those properties through their output formats. The typed structure exists --- it is simply inaccessible to external consumers.

A platform that stores content in an open, typed, traversable format inverts this relationship. When the storage format itself satisfies P1 (declared block types), P2 (typed cross-references), P3 (priority signals), and P4 (pre-computed token costs), no adapter layer is needed. The format is the API. An AI agent can read the files directly, parse them into a typed AST, and execute HBDS (Section 5.1) without an embedding pipeline, without RAG infrastructure, and without chunking heuristics. The structure is the retrieval system.

The implication is economic, not merely technical. As AI agents become primary consumers of organizational knowledge --- a trend already visible in the growth of agent-assisted development, automated documentation synthesis, and AI-driven decision support --- the platforms that export typed, traversable structure will capture the AI-agent ecosystem. Agents will preferentially operate on knowledge bases where HBDS can run natively, where pre-read budget verification works (Section 5.3), and where the write-back phase (Section 5.1) can deposit synthesized knowledge directly into the hierarchy. Platforms that export only HTML or API-gated JSON force every agent interaction through an extraction-and-inference pipeline --- paying the computational cost of structure recovery that the comparison in Section 5.4 shows is both expensive and lossy relative to declared structure. The platform that declares its structure wins the agent; the platform that hides it behind rendering loses it.

7.4 For the RAG Community

RAG's effectiveness is bounded by chunk quality, and chunk quality is bounded by the format. When a flat-text corpus is chunked by character count or paragraph break, the chunk boundaries are arbitrary --- a 512-token window may split a decision from its rationale, sever a data schema from its constraints, or merge unrelated content into a single embedding vector. Typed blocks eliminate this problem at the source. A ::decision block, a ::data block, and a ::code block are each semantically complete units with machine-readable boundaries. Chunking a typed corpus by block type produces chunks that are coherent by construction, not by coincidence. This alone would make typed formats a meaningful improvement to RAG pipelines.

But the deeper implication is that on typed corpora, HBDS may be a replacement for RAG, not merely an improvement. The architectures differ at every level. RAG retrieves flat fragments by embedding similarity --- a stateless, topology-blind operation that treats every chunk as an independent vector in embedding space. HBDS navigates a typed hierarchy by semantic descent --- a stateful, topology-aware operation that exploits parent-child relationships, block types, and budget constraints to reach relevant knowledge through structural reasoning rather than vector proximity. RAG is stateless: every query starts from the same index, and no retrieval session improves the next. HBDS accumulates knowledge through write-back (Section 5.5): the knowledge tree self-improves as agents deposit synthesized insights, update summaries, and refine trust scores.

Table 5 (Section 5.4) quantifies this comparison across eight dimensions. On a typed corpus, HBDS holds clear advantages on six dimensions --- structure accuracy, budget control, graceful degradation, chunk coherence, knowledge accumulation, and amortized cost of structure --- and is comparable on relevance scoring. The only dimension where RAG is advantaged is the zero upfront cost of having no structure. For corpora in sustained use by AI agents, the amortization arithmetic favors declared structure (Section 4.3).

A pragmatic middle ground exists. Typed RAG --- using embedding similarity for initial candidate retrieval across a large corpus, then switching to HBDS for structured descent within retrieved subtrees --- combines RAG's corpus-scale coverage with HBDS's type-aware precision. Section 8.2 identifies this hybrid as a concrete direction for future work. The question for the RAG community is not whether typed structure helps retrieval --- Table 5 establishes that it does. The question is whether the authoring investment to produce typed corpora is justified, and for knowledge bases queried repeatedly by AI agents, the answer is increasingly difficult to avoid.

7.5 For the Phone-to-App Pipeline

The pipeline that transforms natural language into deployed applications --- phone to AI chat to typed document specification to compiler to live artifact --- depends on the AI controller's ability to retrieve domain knowledge during generation. The user provides intent ("Build me a bakery website"); the controller must supply everything else: data models, business rules, compliance constraints, UI patterns, and integration requirements. Without retrieval, the controller generates from its parametric knowledge alone, producing specifications that are generically plausible but domain-incomplete. With flat retrieval (RAG), the controller surfaces fragments that are semantically similar to the query but structurally unrelated --- embedding proximity to "bakery" retrieves menu descriptions but misses PCI-DSS payment compliance requirements that use entirely different vocabulary. HBDS on a typed knowledge base is the architectural solution.

Consider the concrete case. A user requests a bakery ordering application. HBDS's ORIENT phase classifies the query into the hospitality/restaurant domain and selects the corresponding subtree as the search root. The DESCEND phase performs beam search through the hierarchy: at the category level, it scores data-models, business-rules, compliance, and UI-patterns as relevant; at the pattern level, it retrieves ::data blocks for Menu, MenuItem, and Order entity schemas, ::callout[type=warning] blocks for PCI-DSS payment handling and food-allergen disclosure requirements, and ::code blocks for order-status state machines. The SYNTHESIZE phase combines these heterogeneous fragments --- drawn from different categories within the same domain subtree --- into a coherent context injection. The resulting specification includes data models, compliance guardrails, and interaction patterns that the controller would not have generated from parametric knowledge or flat retrieval.

This is where HBDS's cross-category retrieval matters most. A bakery application is not a single-topic query --- it requires data models from one subtree, compliance constraints from another, and UI patterns from a third. RAG's embedding similarity treats these as independent retrieval targets, surfacing whichever fragments happen to be closest to the query string. HBDS's hierarchical descent traverses laterally across sibling categories within the domain, surfacing structurally co-located knowledge that embedding proximity would miss. Study 3 (Section 6.3) directly tests this use case: the broad-scope generation prompts require exactly this cross-category retrieval, and the compliance coverage metric (M3) specifically measures whether the retrieval mechanism surfaces type-distant but structurally adjacent constraints. The phone-to-app pipeline is, in effect, the production deployment of HBDS's core capability --- budget-constrained, type-aware navigation of a knowledge hierarchy in service of generative specification.

8. Limitations and Future Work

8.1 Limitations

We identify five specific limitations of this work and the claims it can support.

Single-organization production data. The observational telemetry reported in Section 6.4 is drawn from 15 repositories within a single bootstrapped software company, operated primarily by a single developer. This confounds organizational culture, developer experience, codebase characteristics, and agent usage patterns into a single data point. The single-developer setting introduces an additional confound: the ARDS designer is also the primary user, so the observed improvements may reflect designer fluency rather than format properties. This limits our ability to claim that the production results generalize to larger teams, different industries, or organizations with different documentation cultures. Replication across at least three independent organizations of varying size is needed before external validity can be established.

One typed format tested. All three controlled studies (Sections 6.1--6.3) use a single typed document format as the experimental condition. Observed performance gains may be attributable to specific design choices in that format --- its particular block-type vocabulary, its frontmatter schema, its cross-reference syntax --- rather than to the general principle of typed traversability. This limits our ability to claim that typing itself drives the effect, as opposed to incidental format-specific features. Testing with alternative typed formats such as AsciiDoc (with custom block macros), DITA (with typed topics and relationship tables), or a minimal typed-Markdown extension would isolate whether the effect derives from typing in general or from specific format design decisions.

Agent-specific results. All experimental conditions use a single LLM as the agent backbone. Different agent architectures --- varying in context window size, tool-use paradigms, reasoning capabilities, and retrieval strategies --- may interact differently with typed structure. An agent with weaker instruction-following may fail to exploit block types even when declared; an agent with stronger long-context performance may mitigate some flat-text failure modes through brute-force reading. This limits our ability to claim architecture-independent benefits. Cross-agent replication using at least two architecturally distinct LLMs (e.g., differing context window sizes or tool-use interfaces) is required.

HBDS not yet benchmarked at scale. The HBDS algorithm design is complete and its formal properties are argued in Section 5, but empirical evaluation has not been conducted on large knowledge hierarchies. The scaling behavior of beam-search descent, budget tracking, and write-back on trees with 10K+ nodes and 100K+ cross-references remains theoretical. This limits our ability to claim that HBDS is practical for organizational-scale knowledge bases. Open-source implementation followed by benchmarking on standardized knowledge-intensive retrieval tasks is the necessary next step.

Authoring cost not measured. Typed documents impose higher authoring overhead than flat Markdown: authors must learn a block-type vocabulary, declare cross-references explicitly, and maintain schema-valid frontmatter. This paper measures the consumption benefits of typed structure but not the production costs. The ROI calculation --- whether agent performance gains justify the additional authoring effort --- is unaddressed. AI-assisted authoring tools could reduce this cost substantially, but that claim is speculative and unvalidated. Without authoring-cost data, practitioners cannot make informed adoption decisions, and the paper's practical recommendations remain incomplete.

8.2 Future Work

The limitations above define five concrete research directions, each paired with the limitation it addresses.

HBDS implementation and benchmarking. The most immediate priority is an open-source reference implementation of HBDS with reproducible benchmarks against RAG on standardized knowledge-intensive tasks. KILT [32] and Natural Questions [33], adapted for document-corpus retrieval rather than single-passage extraction, provide established evaluation frameworks. The key experiment is HBDS on typed corpora versus RAG on the same content in flat form, measured on retrieval precision, token efficiency, and answer accuracy across knowledge trees of 10K+ nodes. This directly addresses limitation four (scale): until HBDS is benchmarked on trees substantially larger than the 15-repository production corpus, its scaling properties remain theoretical.

Knowledge-context unification. Section 6.4 demonstrates typed traversability on code repositories. The natural extension is organizational knowledge: company wikis, policy documents, training materials, and institutional memory. The thesis is that knowledge is context --- the same typed structure that helps agents navigate a codebase should help them navigate a knowledge base, because the retrieval problem is identical. This requires extending the typed-traversability properties (Section 4.1) beyond software artifacts to general-purpose documents, testing whether P1--P4 retain their algorithmic value when block types shift from ::code and ::decision to ::policy, ::procedure, and ::regulation. This addresses limitation one (single domain) by moving typed traversability into domains where the single-organization confound can be tested independently.

AI-native knowledge bases. Section 7.3 argues that platforms storing content in typed, traversable formats become AI-native knowledge bases without adapter layers. The research question is whether a Wikipedia-scale knowledge base --- millions of typed documents with declared hierarchies and cross-references --- can be built and maintained for dual human-AI consumption, with HBDS as the primary traversal algorithm. A three-phase roadmap (internal tool, product feature, public platform) provides incremental validation. This addresses the platform implications identified in Section 7.3.

Typed RAG. Section 5.4 acknowledges that HBDS requires typed corpora --- a real constraint when most existing content is flat. The pragmatic hybrid is Typed RAG: embedding-based retrieval for initial corpus-scale candidate selection (fast, approximate, format-agnostic), followed by HBDS for structured descent within retrieved subtrees (precise, budget-aware, type-exploiting). The research question is whether the hybrid retains HBDS's precision advantages on the six dimensions identified in Table 5 while matching RAG's scalability to untyped corpora. This addresses the honest trade-off from Section 5.4 by reducing the format prerequisite from the entire corpus to retrieved subtrees.

Self-organizing knowledge trees. Section 5.5 defines write-back as a structural capability and acknowledges that its long-term value is empirically unvalidated. A longitudinal study is needed: deploy HBDS with write-back enabled on a production knowledge base, measure retrieval accuracy trend, summary quality, and trust score distribution over months, and compare against a frozen baseline (identical corpus, no write-back). The core question is whether retrieval quality monotonically improves or whether write-back introduces drift, noise accumulation, or trust inflation that degrades the hierarchy over time. This addresses Section 5.5's forward-looking acknowledgment with the sustained measurement it explicitly calls for.

9. Conclusion

The document format is not a cosmetic choice for AI-agent systems. It is an architectural one that determines which algorithms are available, which failure modes are inevitable, and which capabilities are structurally possible. This paper has argued that thesis through formal definition, algorithmic construction, and empirical evidence --- and in each case, the conclusion is the same: flat text is not merely inconvenient for AI agents; it is insufficient. The format withholds information that algorithms require, and no improvement in model quality can compensate for information the input does not contain.

The central proof is constructive. We defined typed traversability as four properties --- declared block types (P1), typed cross-references (P2), priority signals (P3), and pre-computed token costs (P4) --- and presented HBDS (Hierarchical Budget-Constrained Descent Search), a four-phase traversal algorithm that navigates knowledge hierarchies under strict token budgets. Each phase of HBDS maps to at least one required property: ORIENT requires P2 to identify a search root in a typed hierarchy; DESCEND requires P1 for type-aware relevance scoring, P4 for pre-read budget verification, and P1 + P3 for graceful degradation to summary-only retrieval; SYNTHESIZE requires P1 for type-aware deduplication; WRITE BACK requires P2 for hierarchical placement of new knowledge. Removing any single property disables at least one phase. Flat text --- Markdown, HTML, plain prose --- provides none of the four properties. The algorithm is impossible on Markdown. It is trivial on a typed format. This converts the argument from "typed structure is nicer" to "typed structure enables algorithms that flat text prohibits" --- a provable claim, not a preference.

The five failure modes identified in Section 3 ground this proof in concrete agent workflows. Structure hallucination (F1) and type ambiguity (F2) arise because flat text lacks P1: the agent must guess what every block is. Broken traversal (F3) arises because flat text lacks P2: links carry no relationship semantics, and the agent cannot build a navigable knowledge graph. Context budget waste (F4) arises because flat text lacks both P3 and P4: the agent cannot prioritize reading and cannot verify costs before committing tokens. Semantic lossy compression (F5) arises because flat text lacks P1 and P3: the agent has no typed compression strategy and falls back to positional truncation. Each failure mode maps to a missing property; each missing property disables an HBDS phase. The failure modes and the algorithm agree on what the format must provide.

The empirical evidence converges from multiple directions. Three controlled studies test the thesis directly: Study 1 measures context retrieval accuracy across flat versus typed conditions, isolating the effect of block types on hallucination rate and token consumption. Study 2 measures knowledge graph traversal, isolating the effect of typed cross-references on dependency-tracing completeness and depth. Study 3 compares HBDS against standard RAG on a knowledge-intensive specification generation task, testing whether hierarchical descent on typed documents produces more complete, more correct output with fewer retrieval tokens than embedding-similarity retrieval on flat text. Production telemetry from 15 repositories over eight weeks corroborates the controlled findings with real-world scale: structured context documentation reduced discovery overhead from 40--60% of the context window to 15--25%, reduced agent error rates, and improved unassisted task completion from approximately 60--70% to 80--90%. The evidence is preliminary --- the controlled studies are designed but not yet executed, and the observational data carries the confounds acknowledged in Section 8.1 --- but the theoretical prediction and the directional evidence are consistent.

The implication is practical and immediate. The next generation of AI-agent tooling should treat document format as a first-class architectural decision --- not a serialization choice downstream of retrieval, but a capability gate upstream of it. Agent framework designers should build typed AST consumption as a native input mode, not an afterthought that serializes structure away before the agent sees it. Document format designers should treat AI agents as a first-class audience alongside human readers, targeting Level 5 on the typed-traversability spectrum: all four properties present, with ceremony cost low enough for human authors to sustain. Knowledge base platforms should export typed, traversable structure rather than opaque HTML or API-gated JSON, because the platform that declares its structure wins the agent ecosystem. The RAG community should investigate typed RAG --- embedding similarity for corpus-scale candidate retrieval, HBDS for type-aware descent within retrieved subtrees --- as a path beyond the limitations of flat chunking and topology-blind retrieval.

The era of flat text as the default medium for AI-agent consumption is ending --- not because flat text is bad for humans, but because it is provably insufficient for the algorithms that make agents effective. What replaces it is typed, traversable, budget-aware structure --- designed not for how humans read, but for how algorithms search.

References

[1] J. L. Lulla, S. Mohsenimofidi, M. Galster, J. M. Zhang, S. Baltes, and C. Treude, "On the impact of AGENTS.md files on the efficiency of AI coding agents," arXiv preprint arXiv:2601.20404, 2026.

[2] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, "Lost in the middle: How language models use long contexts," Trans. Assoc. Comput. Linguist., vol. 12, pp. 157--173, 2024. doi: 10.1162/tacl_a_00638

[3] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W. Yih, T. Rocktaschel, S. Riedel, and D. Kiela, "Retrieval-augmented generation for knowledge-intensive NLP tasks," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 9459--9474.

[4] D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson, "From local to global: A Graph RAG approach to query-focused summarization," arXiv preprint arXiv:2404.16130, 2024.

[5] P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, and C. D. Manning, "RAPTOR: Recursive abstractive processing for tree-organized retrieval," in Proc. Int. Conf. on Learning Representations (ICLR), 2024.

[6] N. Gupta, W.-C. Chang, N. Bui, C.-J. Hsieh, and I. S. Dhillon, "LLM-guided hierarchical retrieval," arXiv preprint arXiv:2510.13217, 2025.

[7] J. Gruber, "Markdown: Syntax," Daring Fireball, 2004. [Online]. Available: https://daringfireball.net/projects/markdown/syntax

[8] A. Gu and T. Dao, "Mamba: Linear-time sequence modeling with selective state spaces," arXiv preprint arXiv:2312.00752, 2023.

[9] B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, et al., "RWKV: Reinventing RNNs for the Transformer era," in Findings of the Assoc. for Computational Linguistics: EMNLP 2023, 2023, pp. 14048--14064.

[10] T. Dao and A. Gu, "Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality," in Proc. Int. Conf. on Machine Learning (ICML), 2024.

[11] I. Beltagy, M. E. Peters, and A. Cohan, "Longformer: The long-document transformer," arXiv preprint arXiv:2004.05150, 2020.

[12] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, "Big Bird: Transformers for longer sequences," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 17283--17297.

[13] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Re, "FlashAttention: Fast and memory-efficient exact attention with IO-awareness," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.

[14] A. Graves, G. Wayne, and I. Danihelka, "Neural Turing Machines," arXiv preprint arXiv:1410.5401, 2014.

[15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998--6008.

[16] S. Pope, S. Narayan, H. Tillet, and T. Dao, "KV cache scaling analysis for large language models," in Efficient Natural Language and Speech Processing Workshop, NeurIPS, 2023. [Note: KV cache memory estimates are widely documented in inference deployment literature; representative figures derive from Llama 3 technical reports and deployment guides.]

[17] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, "RoFormer: Enhanced transformer with rotary position embedding," Neurocomputing, vol. 568, p. 127063, 2024. doi: 10.1016/j.neucom.2023.127063

[18] O. Press, N. A. Smith, and M. Lewis, "Train short, test long: Attention with linear biases enables input length extrapolation," in Proc. Int. Conf. on Learning Representations (ICLR), 2022.

[19] Google DeepMind, "Gemini 2.5 Pro," Google AI, 2025. [Online]. Available: https://ai.google.dev/gemini-api/docs/models

[20] Meta AI, "The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation," Meta AI Blog, Apr. 2025. [Online]. Available: https://ai.meta.com/blog/llama-4-multimodal-intelligence/

[21] Common Crawl Foundation, "Common Crawl," 2024. [Online]. Available: https://commoncrawl.org/. [Note: Corpus statistics (240+ billion pages, multi-petabyte scale) are from the publicly available crawl archives and documentation.]

[22] International Data Corporation (IDC), "IDC Global DataSphere Forecast, 2021--2025," IDC, Framingham, MA, USA, 2021. [Note: The 181 zettabytes figure for 2025 and the approximately four-year doubling period are from IDC's Global DataSphere forecast methodology.]

[23] L. Lamport, LaTeX: A Document Preparation System --- User's Guide and Reference Manual, 2nd ed. Reading, MA, USA: Addison-Wesley, 1994.

[24] American Mathematical Society, "Using the amsthm package," version 2.20.3, 2017. [Online]. Available: https://www.ams.org/arc/tex/amscls/amsthdoc.pdf

[25] N. Walsh and L. Muellner, DocBook: The Definitive Guide. Sebastopol, CA, USA: O'Reilly Media, 1999.

[26] OASIS, "Darwin Information Typing Architecture (DITA) Version 1.3," OASIS Standard, Dec. 2015. [Online]. Available: https://docs.oasis-open.org/dita/dita/v1.3/dita-v1.3-part0-overview.html

[27] S. Rackham, "AsciiDoc," 2002; now governed by the AsciiDoc Working Group, Eclipse Foundation. [Online]. Available: https://asciidoc.org/

[28] H. Sun and S. Zeng, "H-MEM: Hierarchical memory for high-efficiency long-term reasoning in LLM agents," arXiv preprint arXiv:2507.22925, 2025.

[29] C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez, "MemGPT: Towards LLMs as operating systems," arXiv preprint arXiv:2310.08560, 2024.

[30] Z. Li, S. Song, H. Wang, S. Niu, D. Chen, J. Yang, C. Xi, H. Lai, J. Zhao, Y. Wang, et al., "MemOS: An operating system for memory-augmented generation (MAG) in large language models," arXiv preprint arXiv:2505.22101, 2025.

[31] W. Xu, Z. Liang, K. Mei, H. Gao, J. T. Tan, and Y. Zhang, "A-MEM: Agentic memory for LLM agents," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2025.

[32] F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N. De Cao, J. Thorne, Y. Jernite, V. Karpukhin, J. Maillard, V. Plachouras, T. Rocktaschel, and S. Riedel, "KILT: A benchmark for knowledge intensive language tasks," in Proc. Conf. of the North American Chapter of the Assoc. for Computational Linguistics (NAACL), 2021, pp. 2523--2544.

[33] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov, "Natural Questions: A benchmark for question answering research," Trans. Assoc. Comput. Linguist., vol. 7, pp. 452--466, 2019. doi: 10.1162/tacl_a_00276

[34] K. B. Davis, "Agent-Readable Documentation Standard (ARDS) v3.0 Specification," CloudSurf Software LLC, 2025. [Online]. Available: https://surfcontext.org/

[35] M. Yasunaga, H. Ren, A. Bosselut, P. Liang, and J. Leskovec, "QA-GNN: Reasoning with language models and knowledge graphs for question answering," in Proc. Conf. of the North American Chapter of the Assoc. for Computational Linguistics (NAACL), 2021, pp. 535--546.

[36] N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using Siamese BERT-Networks," in Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2019, pp. 3982--3992.

[37] L. Gao, Z. Dai, P. Pasupat, A. Chen, A. T. Chaganty, Y. Fan, V. Zhao, N. Lao, H. Lee, D. Chang, and K. Guu, "Attributed question answering: Evaluation and modeling for attributed large language models," arXiv preprint arXiv:2212.08037, 2023. [Note: Structured retrieval improvement figures (15--25% over flat baselines on knowledge-intensive generation) are broadly consistent with findings across Graph RAG [4], RAPTOR [5], LATTICE [6], and the KG-guided retrieval literature [35].]

[38] H. V. F. Santos, V. Costa, J. E. Montandon, and M. T. Valente, "Decoding the configuration of AI coding agents: Insights from Claude Code projects," arXiv preprint arXiv:2511.09268, 2025.

[39] W. Chatlatanagulchai, H. Li, Y. Kashiwa, B. Reid, K. Thonglek, P. Leelaprute, A. Rungsawang, B. Manaskasemsak, B. Adams, A. E. Hassan, and H. Iida, "Agent READMEs: An empirical study of context files for agentic coding," arXiv preprint arXiv:2511.12884, 2025.

[40] S. Mohsenimofidi, M. Galster, C. Treude, and S. Baltes, "Context engineering for AI agents in open-source software," arXiv preprint arXiv:2510.21413, 2025.

[41] X. Zhu, Y. Xie, Y. Liu, Y. Li, and W. Hu, "Knowledge graph-guided retrieval augmented generation," in Proc. Conf. of the North American Chapter of the Assoc. for Computational Linguistics (NAACL), 2025, pp. 7517--7533.