Who Builds That Graph? On Managing AI Agent Context as a Knowledge Graph

I was mid-demo at a public agency. Someone watching the AI read an issue and attach a fix proposal asked: "Shouldn't it look at the connected sources too, not just the file it's changing?" Yes, I said. It follows the graph to find them. The next question came immediately.

"Who builds that graph?"

Nobody does. I've watched the air in the room change at that answer more than once. This post explains the structure behind it.

What does it mean to give an AI context?

Giving an AI agent context means taking what a senior engineer carries around in their head and making it queryable by a machine. The issue that records why the code looks the way it does. The incident a similar change caused last time. The other sources that reference this module. An agent's output quality is model capability times context quality, and since the model labs keep raising the first factor, the second is effectively the only one an organization can act on.

The problem is that in most organizations this context is scattered across people's heads and a handful of tools. Which is why the first step of "adopting AI" turns out to be organizing context, not choosing a model.

Why isn't vector search enough?

Vector search (RAG) falls short because search returns similar documents, not connected facts.

Search for "payment module timeout" and you get document fragments that mention timeouts. Useful. But what the agent actually needs is a path: the commits linked to this issue → the module those commits touched → the other sources referencing that module → the past incidents tangled up with those sources. That's relationship, not similarity. No matter how close two things sit in embedding space, "this commit exists because of that issue" won't fall out of it. That fact either exists in the data as a relationship, or it doesn't exist at all.

So we chose a structure that follows a graph instead of searching a flat table. Issues, documents, code, builds, and deployments all carry stable IDs and reference each other. Documents link to one another and open in a graph view, the way Obsidian does it. There's a reason the old word "ontology" keeps resurfacing lately: represent knowledge as entities and relationships, and reasoning over it works the same way whether the reasoner is a person or an AI.

So who builds the graph?

Nobody builds the graph. It accumulates automatically as people work.

Create a branch from an issue and one edge appears. Attach a commit to a change request, another. The pipeline builds that change request, another. The deployment ships, another. None of these is "graph-building work." It's all work people were doing anyway. The relationships fall out as a byproduct, simply because the work happens on one plane.

Now consider building this graph in a fragmented toolchain. You issue API tokens for the issue tracker, the repositories, CI, and the wiki; write glue code to stitch four response schemas together; and parse commit-message strings to guess at relationships. We've done it. It works, technically. But everything from token management to schema drift becomes an operational burden, and relationships built on guesses fail silently. This is why we ship GitHub and GitLab integrations and still keep telling people about the limits of that approach.

How does the agent use the graph?

The agent pulls only the context it needs from the graph. It doesn't read everything.

Given an issue, it walks a few hops out from it: linked commits, related documents, similar past issues. Only what it finds there goes into the context window. This matters for two reasons. Accuracy, first: an agent that hasn't read irrelevant things is more precise. Cost, second: for organizations running open-source LLMs in air-gapped networks, the context window is GPU money, so infrastructure that selects the right context affects outcomes as much as model quality does.

To describe what actually runs inside our own team: when a pipeline breaks, the agent doesn't relay the error message. It walks the graph, narrows the cause candidates, and attaches a diagnosis. When it generates a small change request from an issue, it looks at the connected sources before proposing the fix. Neither is a flashy feature. They work because the necessary context exists in a queryable form.

What happens if you adopt AI without a knowledge graph?

Adopt AI without a knowledge graph and the agent starts from a blank page every time. Your organization may have ten years of history; to the agent it might as well not exist. What comes out is code that's plausible but ignorant of your situation. The time reviewers spend catching it eats the time the agent saved, and a few months later the verdict arrives: "AI didn't work for us." It wasn't the model. Nobody gave it the context.

Gathering context onto one plane was valuable before AI — it's the integrated-platform argument we've been making for years. But the moment AI arrives, the nature of that value changes: from something nice for humans into a precondition for agents to work at all. If you're evaluating AI adoption and started with model benchmarks, try reversing the order. Whether the context your agent will stand on exists in queryable form comes first.