Decision infrastructure and enterprise AI agents

Knowledge Graph for LLMs: How to Build One

Written by Navdeep Singh Gill | 21 August 2024

A knowledge graph for an LLM is a store of entities and the relationships between them that the model queries before it answers or acts. It gives the model facts it can cite and relationships it can follow, which a plain text index cannot. Build one when your questions depend on how things connect, such as which supplier feeds which plant, or which policy applies to which product. For a wider view of the technology, see our guide to graph analytics and knowledge graphs.

The build is a sequence of seven decisions: the questions it must answer, the schema, extraction, entity resolution, validation, storage and retrieval. This guide walks through each, compares vector RAG, a knowledge graph and GraphRAG, and sets out the governance and cost points that decide whether it reaches production. It is written for CTOs, CDOs, architects and platform leads.

Why do LLMs need a knowledge graph?

An LLM is fluent but does not hold your enterprise facts, and it cannot show where an answer came from. A knowledge graph supplies the facts as entities and relationships, so the model answers from governed records and the reasoning can be traced. The table maps each common limitation to what the graph adds.

LLM limitationWhat a knowledge graph suppliesWhat you still need
Plausible but wrong answersFacts held as entities and relationships that the answer can be checked againstSource records that are accurate and current
Knowledge fixed at training timeFacts updated in the graph without retraining the modelA refresh process with an owner
No link to enterprise policy or dataPolicies, assets and people modelled as connected nodesAccess control on the graph itself
Opaque reasoningA path through named relationships that can be shown to a reviewerLogging of which paths were used for each answer
Weak multi-hop questionsTraversal across several relationships in one queryA schema that includes the relationships people ask about

Vector RAG, knowledge graph or GraphRAG: which fits?

Vector RAG retrieves text passages that look similar to the question. A knowledge graph retrieves connected facts. GraphRAG uses an LLM to build a graph from your documents and then answers from it. The Microsoft Research GraphRAG paper (first submitted 24 April 2024, revised 19 February 2025) describes GraphRAG as aimed at global questions over a whole corpus, such as the main themes in a dataset, where conventional RAG struggles. On datasets of about 1 million tokens, the authors report that GraphRAG beats a conventional RAG baseline on the comprehensiveness and diversity of answers.

ApproachBest whenMain trade-offStatus to note
Vector RAGQuestions are answered by one or two passagesWeak on relationships and corpus-wide questionsThe default starting point; the GraphRAG documentation keeps it as Basic Search
Curated knowledge graphEntities and relationships are known, governed and queried oftenSchema design and upkeep need peopleBuilt on RDF or property graph standards
GraphRAG (LLM-built graph)Questions span many documents and need themes or community summariesThe LLM runs over the corpus at indexing timeOpen source from Microsoft; the documentation lists Global, Local, DRIFT and Basic search
HybridSome questions are lookups, others need connectionsTwo retrieval paths to test and monitorCommon once a pilot shows where vector search falls short

Start with vector RAG and add the graph where evaluation shows it fails, for example on multi-hop or corpus-wide questions. Our note on how graph databases work covers the storage side.

How do you build a knowledge graph for an LLM?

1. What questions must the graph answer?

Write ten to twenty real questions from the business first. They decide which entities and relationships you model, and they become your evaluation set. A graph with no target questions grows without a test.

2. What schema or ontology will you use?

Define entity types, relationship types and the properties that matter. Keep the first version small and reuse an existing vocabulary where one exists in your industry. Record who owns each entity type, because ownership decides who fixes bad data later.

3. How do you extract entities and relationships?

Structured sources map to the schema directly. For documents, an LLM can extract entities and relationships from text. The GraphRAG indexing pipeline in the GraphRAG documentation splits the corpus into text units, extracts entities, relationships and key claims, clusters the graph with the Leiden technique, then summarises each community. The documentation recommends prompt tuning, because default prompts may not suit your data.

4. How do you resolve duplicate entities?

The same supplier may appear under three spellings. Entity resolution merges them into one node. The neo4j-graphrag package lists semantic-match and fuzzy-match resolvers among its graph-construction extras. Whatever tool you use, keep a log of merges so that a wrong merge can be undone.

5. How do you validate the graph before an LLM uses it?

Check the graph against rules before it feeds answers. For RDF data, SHACL, a W3C Recommendation dated 20 July 2017, is a language for checking that an RDF graph meets a set of conditions written as shapes. For property graphs, use database constraints and your own test queries drawn from the questions in step 1.

6. Where do you store it and how do you query it?

ISO/IEC 39075:2024 is the published ISO standard for property graphs and the GQL query language (Edition 1, April 2024; the ISO page lists a corrigendum issued in 2026 and a revision in development). Check whether your chosen store supports GQL, or its own dialect, before you commit, because that affects portability.

7. How does the LLM retrieve from it?

Common patterns are to run a graph query for entities named in the question, to combine vector search with graph traversal, or to let the model write a query that your code validates before running. Whichever you pick, return the path used with the answer so that a reviewer can see it.

Which graph model and tooling should you choose?

OptionUse it whenCurrent status
RDF with SHACLYou need shared vocabularies, linked data and formal validationSHACL is a W3C Recommendation (2017)
Property graph with GQLApplication teams want node and edge properties and traversal queriesISO/IEC 39075:2024 published April 2024
Microsoft GraphRAGYou want an LLM to build a graph and community summaries from documentsDocumented with Global, Local, DRIFT and Basic search; no version number given on the documentation site
neo4j-graphrag (Python)You run Neo4j and want construction pipelines and retrieversNeeds Neo4j 5.18.1 or later (Aura 5.18.0 or later) and Python 3.10 to 3.14; replaces the deprecated neo4j-genai package

How does this architecture support agentic AI?

Agents need to act, not only answer, so the graph sits between the model and the systems it can change. A common layered pattern puts the graph in the middle, checking and enriching what the model proposes.

LayerRoleControl to add
InputCaptures queries, events and alertsAuthentication and logging
LLMInterprets intent and summarisesPrompt and output filtering
Knowledge graphValidates claims, adds context, links systemsAccess control, versioned schema
Agent orchestrationPlans tasks and coordinates agentsApproval rules for disruptive actions
ExecutionRuns actions in IT, support, security and operationsLeast-privilege credentials
FeedbackWrites confirmed outcomes back to the graphReview before write-back

This is the context layer described by ElixirData Context OS, which keeps governed context, versioned policy and decision traces in one decision layer above AI agents. ElixirClaw is an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control. For retrieval over unstructured material, AgentSearch offers context-aware search across enterprise knowledge.

What are the governance, security and cost risks?

  • Access. A graph can reveal relationships that individual records do not. Apply permissions to nodes and edges, not only to source systems.
  • Lineage. Store the source and extraction date on every fact, so a wrong answer can be traced to a record.
  • Extraction errors. LLM extraction can invent or mislabel relationships. Sample and review before the graph is used for decisions.
  • Cost. GraphRAG-style indexing runs an LLM over your corpus, so cost grows with corpus size. The GraphRAG documentation does not give cost figures, so measure on a bounded pilot set before scaling.
  • Decision review. Where an agent acts on graph facts, route material decisions through review. The Decision Review Agent is described on its page as a way to review AI decisions before execution.

When should you not build a knowledge graph?

  • The questions are answered by single documents and vector RAG already scores well on your evaluation set.
  • No team owns the data, so the graph would go stale within a quarter.
  • The relationships are few and stable enough for a relational table with joins.
  • You cannot yet say which decisions the answers will support.

Which enterprise use cases fit best?

Use caseWhat the graph connectsFit
Supply chain riskSuppliers, shipments, sites and external eventsStrong: multi-hop by nature
Security operationsAlerts, assets, identities and threat patternsStrong: relationships carry the signal
Compliance and governanceRegulations, policies, controls and evidenceStrong where answers must cite sources
Customer supportAccounts, products, contracts and historyMedium: often solved with RAG first
Enterprise searchDocuments, people, projects and topicsMedium: GraphRAG for corpus-wide questions

Related reading: how a Google Knowledge Graph is built and exposed, which is a different problem from an enterprise graph.

What is a sensible adoption sequence?

  • Weeks 1 to 2: collect the question set, pick one domain and name the data owners.
  • Weeks 3 to 4: build a small schema and load structured sources; run vector RAG as the baseline.
  • Weeks 5 to 8: add document extraction on a bounded corpus, resolve entities and validate.
  • Weeks 9 to 12: compare graph-grounded answers with the baseline on the question set, then decide whether to extend.

The timings are our planning guide, not a benchmark. Adjust them to your data volume and team size.

To map one decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions

What is a knowledge graph for an LLM?

It is a store of entities and relationships that the model queries to ground its answers in governed facts and to show the path behind them.

Is GraphRAG better than vector RAG?

It depends on the question. The Microsoft Research paper reports better comprehensiveness and diversity on global questions over datasets of about 1 million tokens. Its own documentation keeps Basic (vector) search for queries that baseline RAG serves well.

Which standards matter when building a knowledge graph?

SHACL, a W3C Recommendation from 2017, validates RDF graphs. ISO/IEC 39075:2024 standardises property graphs and the GQL query language.

How do you reduce wrong relationships from LLM extraction?

Tune the extraction prompts, resolve duplicate entities, validate against rules, record the source of each fact and sample the output before agents act on it.

When should you avoid a knowledge graph?

Avoid it when single documents answer your questions, when no team owns the data or when the relationships are few enough for relational joins.