Building a Knowledge Graph
for AI: Enterprise Guide to
Architecture, RAG & AI Agents

You may already have an AI assistant connected to company documents, and it may still get questions about your own customers and contracts wrong. Building a knowledge graph for AI assistants improves the accuracy of answers to questions that involve numerous connected records, though it adds design work and infrastructure costs.

We explain how knowledge graphs fit into AI agents and RAG (retrieval-augmented generation) and when plain vector search is enough. You’ll also see how to create a knowledge graph step by step and what a pilot costs, from hosting to salaries.

A knowledge graph is a database built around connections. It represents customers, suppliers, and products as nodes, with edges describing how they relate to one another. These connections have labels, such as “supplies” or “purchases,” that define the semantic relationships between records.

Suppose a supplier announces a two-week delay, and a manager needs a list of every affected customer. Normally, the team would export data from two systems and match it in a spreadsheet. Now, analysts can pull the same list from a knowledge graph in one search, because the developers recorded those links when they built it.

Engineers call the graph’s rulebook an ontology: the agreed list of record types and the ways they can connect, and planning how to create a knowledge graph for AI begins with this. Developers can use taxonomies to classify entities and graph schemas to define which types of relationships the database permits. Importantly, company leadership settles what each term means, because if (for example) “customer” includes free-trial users, the customer count in the AI assistant’s answers will differ from the finance department’s figure.

The Role of Knowledge Graphs in GraphRAG and AI Agents

Developers use knowledge graphs to give AI assistants connected facts with traceable sources, and to supply AI agents with company context for their tasks.

Plain retrieval-augmented generation (RAG) finds the passages closest to a question and hands them to the AI model. Yet, Microsoft Research recorded weak answers from it on two kinds of questions: ones that need facts spread across documents and ones about a whole collection, such as “What are the main themes here?”

GraphRAG can also support multi-hop reasoning, where an answer depends on following connected facts. The retrieval system uses graph traversal to find paths between entities that a search of individual passages might miss.

Microsoft’s researchers tested their own take on how to build a knowledge graph for RAG, called GraphRAG: they had an AI model read thousands of news articles and map out who and what the articles mention and how those things connect. Plain RAG came back empty on a question that needed facts from several articles, while GraphRAG answered it and linked each claim to its source article for a reviewer to check.

GraphRAG’s answers to big-picture questions also scored as more complete and more varied when Microsoft had an AI model grade both methods on large document collections of about 1 million tokens.

Zep built its Graphiti engine to give AI agents memory: a knowledge graph of conversation history and business data, with dates on each change to a relationship. In Zep’s tests on the LongMemEval benchmark, agents using it improved accuracy by up to 18.5% and cut response latency by 90% compared with baseline setups.

Gartner analysts applied the idea to AI agents in February 2026 research. They describe context graphs (knowledge graphs extended with business decision rules and a trace of workflow steps) as essential infrastructure for grounding AI agents in company context.

Such graphs can also provide shared context for AI agent orchestration, so agents working on related tasks can retrieve information about the same business entities and decisions.

Companies build a knowledge graph for enterprise AI to fix the most common problem with AI in business use: wrong answers. McKinsey’s 2025 survey found inaccuracy to be the AI-related risk organizations experience most often, and nearly one-third of all respondents reported consequences from it.

Researchers at the data.world AI Lab tested whether business context reduces those errors, so they asked GPT-4 the same questions about an insurance database in two ways. GPT-4 answered 16.7% correctly when using SQL queries against the relational database, compared with 54.2% when querying a knowledge graph of the same data in SPARQL, a query language for Resource Description Framework (RDF) graphs.

Companies use knowledge graphs when the answer depends on how records connect. For example:

• Customer support. LinkedIn turned its past support tickets into a graph that kept the links between related tickets, and after about six months of use, median resolution time per issue fell by 28.6%.
• Product recommendations. Amazon researchers used a knowledge graph to capture which products people buy for particular situations, such as slip-resistant shoes for pregnant women. They reported improvements of up to 60% in offline product search results, showing how such connections can be useful for recommendation engines.
• Drug research. Drug developers at AstraZeneca work from a single graph of 10.9 million entities and over 118 million connections, built from public and internal data to support analytics and machine learning.
• Financial investigations. Journalists use ICIJ’s Offshore Leaks Database, built on Neo4j, to follow who owns what across more than 810,000 offshore entities, including companies that hold shares in other companies or trusts.
• Fraud detection. Banks use fraud detection systems that connect accounts, devices, and transaction histories in a graph. Investigators can then examine networks of accounts that share identifying details or transfer funds between one another.
• Content management. Companies exploring how to build a content knowledge graph can connect articles and internal documents to the products or policies they describe. As a policy changes, content managers can find the documents that reference it and need reviewing.

Vector search uses embedding models to represent text numerically and retrieve passages based on semantic similarity. It works well for questions answered in a single passage, while a knowledge graph is built for questions that need facts from distributed but connected records.

The two often work well together. Researchers from BlackRock and NVIDIA tested both methods on 400 analyst questions about company earnings calls. And while 96% of the context the graph retrieved was relevant against 84% for vector search, a combination of the two gave the most relevant answers overall.

Companies can settle the choice by listing the questions the AI should answer and sorting them into two groups: ones with the answer in a single document, and ones that need connected records. Those with a long second group have a case for a graph, usually alongside vector search, while those with a short one can start with vector search alone.

The process of building a knowledge graph runs from choosing the questions the graph has to answer to keeping it current after launch. The steps below explain how to create a knowledge graph for AI applications, starting with a small pilot that developers can test against real business questions.

1. Choose the questions the graph has to answer

When deciding how to develop a knowledge graph, start by collecting the questions your AI needs to answer and a sample of correct responses. Developers can use these as a test set throughout the project.

The test set should include natural language questions phrased as employees would ask them, including variations that express the same user intent.

2. Map the data sources

The development team inventories both structured data in CRM and ERP tables and unstructured data in shared drives, contracts, and support tickets. You can only create a knowledge graph from sources your developers can reach, so each source needs an owner on your side who opens it up to the team and explains what the fields mean.

Developers also check which data connectors are available and how frequently each source changes. They use this information to plan data ingestion, including how to import the initial records and keep them updated.

3. Design the ontology

Developers deciding how to build a tailored knowledge graph can start with existing industry standards and adapt them to the company’s data. For instance, finance has FIBO, an open-source ontology the EDM Council maintains with industry experts, and SNOMED CT, a clinical terminology for healthcare, is used in more than 80 countries.

The ontology engineering process also involves defining which properties and relationships each entity type can have. Developers working with RDF can use RDF Schema to describe classes, properties, and basic inheritance rules.

4. Load and connect the data

The first of two knowledge graph building techniques covers structured data: developers map database tables and columns to entity types or relationships defined in the ontology. They can import records from a CSV file or use an ETL tool to extract and prepare data from existing systems.

To build a knowledge graph from text, such as contracts or emails, developers use the LLM extraction approach from GraphRAG. Content ingestion involves parsing files, chunking longer documents into passages, and entity and relationship extraction to identify the facts and connections described in them.

5. Test and maintain the graph

Developers test the finished graph by asking the AI the questions from step one and checking each response against the managers’ reference answers.

Developers also test the graph construction process with missing fields and malformed data to check whether import errors create incomplete records or incorrect relationships.

The ongoing cost of knowledge graph development depends partly on how developers add new data after launch. Microsoft’s GraphRAG users had to re-index the whole dataset every time they added new information until GraphRAG 1.0 arrived in December 2024 with an update command that merges only the changes into the existing index.

Example: Finding Customers Affected by a Supplier Delay

Let’s take one path through the records from the supplier example given earlier to illustrate how to build a knowledge graph:

Acme Ltd. (supplier) → supplies → X-200 brake pad (product) → included in →
Order #4512 → placed by → Northline Auto (customer)

Developers store a path like this for every order, using four record types: Supplier, Product, Order, and Customer.

The contract archive lists the same supplier as “ACME Limited,” so developers merge both names into one Supplier record based on their shared tax ID. Orders filed under either name then lead back to Acme.

When Acme announces a two-week delay, the manager asks the AI assistant which customers it affects. Through graph traversal, a query follows the edges connecting Acme to the relevant product, order, and customer nodes. It returns 37 customers across 120 open orders.

Developers check this during testing by listing the customers on Acme’s open orders straight from the ERP, and both lists name the same 37 customers.

To build a knowledge graph for enterprise AI, developers connect a number of components between the company’s existing software and the assistant or agent employees use.

Each component has a specific role in storing company information or retrieving it when someone asks a question.

• Source systems. The company’s CRM and ERP keep running as before, with records stored in a relational database or another existing system. Scheduled synchronization jobs copy changes into the graph, while the original systems remain responsible for their usual data management tasks.
• Graph database. Developers store entities and relationships using a property graph, as in Neo4j, or the Resource Description Framework. RDF data is commonly stored in a triple store, which represents each fact as a subject, predicate, and object.
• Text search index. Developers store original document passages in a separate search index, often alongside vector embeddings for semantic retrieval. They also preserve the original text so users can retrieve the exact wording of a contract clause. OWASP warns that users of a shared index can retrieve documents meant for another group, so developers tie access to each person’s role.
• Retrieval layer. Developers working out how to build a knowledge graph for RAG connect graph queries with text search in this layer. The retrieval code combines both sets of results and selects the information to pass to the LLM.
• AI application. This is the only layer users see: a chat assistant or an AI agent that employees, and sometimes customers, work with.

To build a knowledge graph, you need a graph database to store connected records, document processing software to prepare and search text, language models to extract entities and write answers, and cloud or on-premises servers to host the system.

Engineering time accounts for most knowledge graph development costs, while infrastructure for a small pilot runs to a few hundred dollars. Analyst firms rarely publish full project benchmarks, so these totals come from published prices for infrastructure and engineering time.

Database hosting is the easiest piece to price, because vendors charge by the hour. The monthly bill is simply the hourly rate times 730 — the number of hours in an average month. Neo4j AuraDB Professional starts at $0.09 per GB per hour, so its smallest 1 GB instance runs about $65.70 a month. Amazon Neptune charges $0.348 per hour for a db.r5.large server in the US East, roughly $254 a month, plus $0.10 per GB of storage each month.

AI model processing comes next, and OpenAI bills it per token (a piece of a word). Reading 40 million tokens of documents once with gpt-6.1-sol, for example, costs $80 at $2 per million, and writing out 10 million tokens of extracted entities and relationships adds another $100 at $10 per million. Full GraphRAG makes extra passes to summarize each group of related entities, and each pass adds to the bill. Microsoft’s lighter LazyGraphRAG indexes data for 0.1% of what full GraphRAG costs.

Finally, team costs. Two in-house developers on a three-month pilot, for example, cost about $68,000 in salary at the U.S. median of $135,980 a year reported by the Bureau of Labor Statistics (May 2025), before benefits and overhead.

Outsourcing can lower that number. Development companies in Eastern Europe charge $25 to $70 an hour, so the same two developers at 160 hours a month would bill $24,000 to $67,200 over three months.

Assuming two developers spend three months on one department’s data, team costs come to roughly $24,000 to $68,000, depending on where the team is hired. Hosting and AI services are billed separately, as is any extra work, such as cleanup.

Neo4j keeps billing about $788 a year for the smallest instance after the pilot ends, and developers run each new batch of documents through extraction at the same per-token rates. A company adding 4 million tokens of new documents a month, for example, pays about $8 a month in input charges for them.

DevCom has been building custom software since 2000, and its team brings that experience to knowledge graph solutions development. Every project starts with discovery: engineers look at the client’s existing data and identify which records the AI needs to retrieve. For enterprise knowledge management projects, this can include information spread across internal document repositories and business systems.

Next, the team works out how the AI will find information, and CTO Andrew Dovgal has described one approach DevCom engineers use: Neo4j to follow connections between records, paired with semantic search, so the AI can draw on linked records and the original documents together. DevCom handles hosting too, and for GBS Enterprises it set up an AWS environment that included an RDF database.

The team also cleans up and anonymizes client data before any AI processing starts.

DevCom stays on after launch to monitor performance and roll out upgrades. Companies working out how to create a custom knowledge graph can bring their data sources and the questions they want answered to DevCom and plan a first pilot together.