One of the biggest problems with AI today is its inability to respond to precise domain-specific questions and its tendency towards hallucinations. Try to upload a bunch of PDF documents to any AI chatbot and ask “What is it about?”. Most of the systems will either provide a wrong answer or a very generic response. The reason is that most of them don’t have a deep understanding of the relations and topics in the context you’re analyzing. A standard system will use a so-called RAG system (retrieval-augmented generation) that will retrieve the chunks of the text that are most similar to the user’s query and use them in the response.

Graph RAG is a loose term for a collection of methods that use knowledge graphs that address this problem and introduce reasoning into model’s responses. In this article, we explain how it works using visualizations made with InfraNodus — the tool that has been used to represent text as graphs since 2014. We also demonstrate how these graphs can be helpful not only for AI but also for the humans who use them, similar to how navigation instructions that we use on Google maps can help us discover the shortest paths to our destinations and the most interesting places to explore.

How GraphRAG Works and Its Difference from RAG

A standard LLM will provide the most likely response to a user’s query. Essentially, it’s a stochastic engine that attempts to predict the most likely continuation of a text sequence. When additional context is provided, the model will perform better because the probabilities will be narrowed down, but that makes the quality of this context very important.

Standard approach to add context to a user’s query is a so-called retrieval-augmented generation (or RAG). The additional context is split into overlapping text chunks and each chunk is given a vector representation. The user’s query is then also transformed into a vector and the nearest chunks that relate to the user’s query are added into the model’s context. Here is where the problem lies.

Suppose a user uploads a Microsoft’s paper on GraphRAG and asks “What is it about”? Here are the text chunks they’re going to get:

The standard RAG will look for all the chunks that relate to “it is about” and retrieve unrelated chunks, such as some domain-specific use cases cited in that paper. These are not the main ideas and topics expressed in this paper and if we add these chunks as the “context” to the user prompt we’re going to get a wrong response.

Some LLM systems like ChatGPT, for example, will not use RAG and simply feed the whole document into the context, but that will not work if you have multiple papers (ChatGPT’s limit is 20) and the bigger context becomes, the more generic the model’s responses will be as it will have to rely more and more on its general world knowledge.

Graph RAG is a general term for the various methodologies that were developed to address this problem. The approach has multiple implementations but they are all based on representing the underlying knowledge as a graph where the nodes represent the entities and the edges represent the relations between them.

There are many different ways to represent these relations. Some stricter approaches may rely on existing ontologies: for example, “GPT-4 is [anInstanceOf] LLM” and “RAG is [aComponentOf] LLM”. Some others may even use document chunks as the nodes and generate loose relation descriptions using LLMs themselves. The main point is that GraphRAG deals with interconnected data where the relations are described in some sort of way.

In InfraNodus, users can create graphs using word lemmas or extracted entities following our graph representation protocol from 2014. The edge descriptions are represented in the statements where these words or entities co-occur: this provides for a rich description that is not limited by the ontological dictionary. The advantage of this approach is that a statement can act as a descriptor for multiple edges and more than two nodes. So the user can see a simplified version of the graph visualized while the actual relation descriptions can be very detailed and incorporate clusters of nodes.

Another advantage of the InfraNodus system is the application of the graph theory and community detection algorithms to identify the most influential nodes, edges, and clusters. This structural information can be used to reveal the patterns in data (both to humans and to the AI) and provide additional context to the models that extends beyond the original user’s query.

These structural insights can be used both by humans and by LLMs. Both can use the graph to

• Get an overview of the knowledge base or context
• Retrieve the main topics and ideas on the global level
• Zoom into specific concepts and topical clusters without losing the context
• Identify the gaps between different clusters and generate ideas and prompts to bridge them

In the example above, the user (or an LLM) can select “LLM” and “context window” and get the statements that contains the both terms, effectively describing the relation between them. These statements can then be used to better understand the text or to augment the prompts to develop these ideas further. The model now “knows” that this particular paper talks about the problem of exceeding the LLM’s context window and how both RAG and GraphRAG can be useful in addressing this problem. Not only that, it will also see the connection to the term “community”, and after extracting the relation to that term from “LLM” and “context window” it will “know” that a particular way of fixing this problem proposed in this paper is to summarize the communities detected in the graph and use them to augment model’s responses. It would not be possible to get such a rich information just using the standard RAG approach searching for “LLMs” in this paper.

In real-life scenarios, multiple approaches are used. For example, GraphRAG can be combined with traditional search and RAG (e.g. HyperRAG), which will retrieve both the most likely matches and chains of relations for a particular query from a graph and provide them to the model’s prompt using various syntaxes to rank these by importance.

Augmenting User’s Queries with GraphRAG

Apart from using the graphs to improve information retrieval, GraphRAG can also be used to augment user’s queries. The basic approach is to transform the query into a graph and then traverse the graph of the context used to find relational patterns that relate to the original query.

For example, if the user asks “How can AI be used for?” using the paper on GraphRAG as an example, the query would be converted into a graph with two nodes: “AI” and “use” and the actual query is the description of the relation.

The system will then perform a search on the underlying knowledge graph for any nodes that relate to “AI” and “use” and also the edges that may be semantically close to the user’s query. It can be further augmented by including the nodes that connect to the two original nodes, expanding the breadth of search and providing a rich collection of possible results based on the edges’ descriptions.

These statements are then added to the original user’s query to generate better responses, helping the model take into account not only the standard semantic matches it could find but also some chain-of-thought reasoning distilled from the original document that can provide much better, context-aware responses.

…

We encourage you to try GraphRAG in an intuitive way with your own documents using InfraNodus. We also have an Obsidian graph view plugin and a browser extension which enable you to use the plugin on your own data, web pages, ChatGPT conversations, and YouTube videos.

Please, contact us if you need to set up GraphRAG workflows within your organization.