// Architecture // 2026-07-03

Building Agentic Search over Vectors and a Database

Most search bars quietly hard-code one strategy — usually vector similarity — and lose every query that strategy can't express. Agentic search puts a small router in front instead: it reads the query, picks the retrieval strategy that actually fits, and compiles a typed plan that runs against your relational tables, your vector store, or both.

This is a generalized version of a real design. The point is not to replace vector search — it's to stop forcing every question through it.

Why one strategy isn't enough

Vector search is superb at topical questions — "find passages that talk about X." But a large share of real questions are structural: they ask about relationships that already live in your database as foreign keys, tags, and join tables. Answering those with embeddings is guessing at something you can look up exactly.

Take a concrete query:

"which projects use documents tagged GDPR?"

Vector search alone matches documents whose text resembles "GDPR" — so it misses correctly-tagged docs that never spell out the phrase, and includes docs that mention it once in passing. The tag relationship isn't in the text, so it's simply not expressible this way.

Structural retrieval answers it exactly, with no embeddings, because the answer already lives in your schema as a join:

tag: GDPR  →  document_tag  →  document  →  project

The router's whole job is to notice which kind of question just arrived.

Three retrieval modes

Sort every query into three buckets by a single question: does it name a known entity, a free-text topic, or both?

Structural — deterministic

  • When: the query is anchored on a known entity plus a relationship verb — owns, tagged, assigned to, which teams use…
  • How: resolve anchor → id, walk the schema to the target grain, SELECT / JOIN through relations, return exact rows.
  • Property: exact, no embeddings, no hallucination.

Semantic — similarity

  • When: a free-text, topical query with no entity anchor — "passages explaining rate-limiting trade-offs."
  • How: embed the query, run vector / hybrid search (dense + BM25 keyword), apply metadata filters.
  • Property: this is your existing vector store, unchanged.

Hybrid — structural ∩ semantic

  • When: an entity anchor and a topic — "onboarding videos in projects owned by the Platform team."
  • How: run the structural pass to get a scoped id set, feed those ids as a filter, then run the semantic search within that scope.
  • Property: structural scope constrains the vectors.

The key move in hybrid: the structural pass narrows the candidate set before the vector search runs, so similarity ranks within an already-correct scope instead of across your whole corpus. That's how you get "on-topic and actually related" instead of one or the other.

How the router decides

The router is one LLM call with a tight prompt. It classifies the query and, when an entity is named, marks it for resolution. The decision reduces to a 2×2 over two signals: is there an entity anchor? and is there a topical phrase?

  • anchor only — "which projects use GDPR-tagged docs" → structural
  • phrase only — "docs about token bucket rate limiting" → semantic
  • anchor + phrase — "security reviews for the Platform team" → hybrid
  • anchor ambiguous / unresolvable — "the billing one" → 3 matches → clarify

Note the fourth outcome. When the named entity resolves to zero or many rows, the right answer isn't a guess — it's a follow-up question. Treat clarify as a first-class outcome, not an error.

The Search IR

Don't let the LLM emit SQL, or free-form tool calls, or prose you hope to parse. Have it emit a small, typed intermediate representation over a closed vocabulary — a fixed set of target grains, anchor kinds, and scope filters that your compiler knows how to execute. The IR is the contract between the probabilistic part (the LLM) and the deterministic part (your database). Anything outside the vocabulary is rejected before it can touch a query.

{
  "mode":   "structural",
  "target": "project",        // grain to return
  "anchor": {
    "kind": "tag",            // one of a closed set
    "term": "GDPR"            // resolved to an id before execution
  },
  "semantic_query": null,     // set this and the mode becomes hybrid
  "scope": {
    "time_range": null,
    "org_id": "<from auth>"   // injected from the session, never the body
  }
}

The fields, generalized:

  • mode — structural / semantic / hybrid, chosen by the router.
  • target — the grain to return (project / document / user …), one of your real entity types.
  • anchor — {kind, term} where kind is a closed set of resolvable entity types; term is resolved to an id before execution.
  • semantic_query — the topical phrase, if any. Its presence is what flips the mode to hybrid.
  • scope — time / status / category filters, plus the tenant id injected from the authenticated session, never accepted from the request body.

Why a closed vocabulary? It makes the LLM's output checkable. A plan that names a grain or filter you don't support fails validation instantly — you never hand an improvised query to your database.

The turn pipeline

Everything above assembles into a single, bounded pipeline. Each stage has one job and hands a well-typed value to the next. The stages marked (reused) are infrastructure you very likely already have.

  1. Query arrives — tenant / role from the authenticated session, a session id for multi-turn clarify, and the natural-language query.
  2. Guard — injection & abuse pre-filter (reused) — a rule-based screen before any LLM spend: prompt-override, role-hijack, data-exfiltration, cross-tenant, and raw-SQL probes. Blocked queries return a refusal at zero model cost.
  3. Route — mode + IR classifier (LLM) — loads any pending intent from prior turns, classifies the query into a retrieval mode, and emits the typed Search IR over the closed vocabulary. A missing or ambiguous anchor routes to a clarify turn.
  4. Resolve & plan the traversal (reused) — turn "GDPR" into a tag id via a fuzzy, tenant-scoped lookup (ILIKE / trigram); then prove a join path from the anchor to the requested target grain. Ambiguous name → clarify.
  5. Retrieval compiler — executes the IR — dispatches by mode: walks the join path to SELECT rows, calls the vector backend, or does both. Every query is tenant-scoped; the structural answer never passes back through the LLM. Results are shaped into your existing result DTOs, so the frontend contract doesn't change.
  6. Assemble & ground (optional, best-effort) — group results (e.g. project → its matching documents) and optionally add a 1–2 sentence summary that cites only returned facts — dropped whole if it invents anything.
  7. Response — mode · results (grouped) · narrative? · trace — or a clarification.

Inside the structural pass

The structural mode is where agentic search beats vector search outright, and it needs no ML. It has two parts: resolve the fuzzy human term to a concrete id, then traverse your existing relationships to the grain the user asked for.

Resolve — a fuzzy, tenant-scoped lookup:

-- "GDPR" is a human label, not an id
SELECT id, name
FROM   tag
WHERE  org_id = :tenant        -- from auth, not the query
  AND  name  ILIKE '%GDPR%'
LIMIT  5;

-- 0 rows  -> clarify ("no matching tag")
-- 1 row   -> proceed with that id
-- >1 rows -> clarify (disambiguate)

Traverse — walk the relations you already have to the target grain:

tag  (anchor: "GDPR" -> id)
  │  via document_tag
document_tag
  │  document_tag.document_id
document  (the tagged docs)
  │  document.project_id
project   (target -> returned)

Model the join paths once. Give the compiler a tiny map of how your core entities connect (a hand-written graph of tables and their join keys is plenty). The router chooses a target grain; the compiler finds the path from anchor to target and emits the joins. New question shapes then cost nothing, as long as the path already exists in the schema.

What you reuse vs. what you build

If you already run a database and a vector store, the agentic layer is mostly a router and a compiler bolted onto machinery you have.

Reuse what exists:

  • vector store — your current embeddings + hybrid search, untouched.
  • relational schema — the foreign keys and join tables are the "graph."
  • entity lookup — fuzzy, tenant-scoped name → id search.
  • auth / tenancy — org & role already on every request.
  • result DTOs — reuse them so the frontend contract holds.

Build new:

  • router prompt — the mode classifier + IR emitter over the closed vocabulary.
  • Search IR type — {mode, target, anchor, semantic_query, scope} plus a validator.
  • join-path map — how core entities connect, for the compiler.
  • retrieval compiler — dispatch by mode: SELECT rows / call vectors / both.
  • hybrid scoper — structural id set → vector metadata filter.
  • clarify loop — multi-turn state for ambiguous anchors.

Pitfalls to design against

  • The LLM never writes SQL. It emits a typed IR you validate; your code writes the query. An improvised query is both an injection surface and a correctness risk. Keep the model on the plan side of the line.
  • Scope comes from auth, not the query. Inject org_id from the session into every query the compiler builds. Never let the router or the request body set the tenant, or you've built a cross-tenant leak.
  • Screen before you spend. Run the rule-based guard first, and skip the LLM entirely for the structural path once the IR is built — a join is milliseconds and needs no model to read the rows back.
  • Clarify beats guessing. Zero or many matches for an anchor is a question, not an answer. Grounded narration is best-effort and gets dropped whole if it cites anything outside the returned rows.
  • Test the router, not just recall. Your new failure mode is misrouting. Keep a labeled set of queries → expected mode and assert on it; a query sent to the wrong backend fails no matter how good that backend is.
  • Keep the vocabulary closed. Every new target grain or anchor kind is a deliberate addition to the IR and the join map. An open vocabulary is an unvalidatable plan.

The whole pattern in one line: Guard → Route (mode + IR) → Resolve → Compile (structural · semantic · hybrid) → Assemble.