This blog post is based on three great YouTube videos: 1) Adam Lucek: Knowledge Graph or Vector Database… Which is Better? 2) Data Heroes: LightRAG & LongRAG Explained: Cutting-Edge RAG Techniques in AI 3) W.W. AI Adventures: GraphRag vs Normal RAG – Summarise a Whole Book in python!
Ever asked a search engine something complex like, “How does the hero’s journey in The Lord of the Rings compare to Star Wars?” You’d probably get separate summaries of Frodo’s and Luke’s adventures, leaving you to connect the dots. Wouldn’t it be amazing if AI could grasp the parallels – good vs. evil, mentors like Gandalf and Obi-Wan, the corrupting influence of power – and give you a truly insightful answer?
This tantalizing possibility is no longer a mere fantasy, thanks to a new wave of AI research that’s pushing the boundaries of traditional search. While current search engines are undeniably powerful, they primarily rely on matching keywords. They often stumble when faced with intricate relationships between ideas, the subtleties of context, and the deeper meaning embedded within text. They can tell you what is in a document, but not necessarily why it matters.
This is where Retrieval-Augmented Generation (RAG) enters the scene. RAG represents a significant leap in AI, combining the prowess of large language models (like those powering ChatGPT) with the ability to retrieve information from external sources. Think of it as having a super-smart research assistant at your disposal. Instead of generating text solely from its internal knowledge, a RAG system can “read” a collection of documents you provide—whether it’s a database of articles, a company’s internal documents, or even the entire text of The Lord of the Rings—and utilize that information to answer your questions. This isn’t your everyday search; it’s a significant advancement.
However, basic RAG still has its limitations. It can be akin to a research assistant who merely skims the surface of documents, overlooking crucial connections. In this post, we’ll delve into advanced RAG techniques that are striving to overcome these limitations and build a web of interconnected knowledge. It is not always a straight line.
(As an intriguing side note, during the writing of this very blog post, a Bengali word meaning “separated” made a surprise appearance, almost convincing me I’d stumbled upon a hidden Elvish term.)
The Limitations of Conventional Search (and Basic RAG)
To truly appreciate the power of advanced RAG, let’s first examine the shortcomings of traditional search and even basic RAG systems. We’ll use The Lord of the Rings as a recurring example to make these concepts more relatable.
Fragmented Knowledge
Traditional search, and even basic RAG, often chop information into isolated fragments—like a paragraph about Frodo leaving the Shire, then another about Aragorn at the Council of Elrond, বিচ্ছিন্ন (*) from Gollum’s story in the Misty Mountains. It’s like examining individual puzzle pieces without seeing the whole picture. While you might uncover information about each character, you’d miss the intricate tapestry of connections that weave their stories together.
Missing the Big Picture
Now, consider the question, “How does the One Ring affect the different characters who carry it?” A basic RAG system might locate separate instances of Frodo, Bilbo, and Isildur possessing the Ring. However, it might struggle to connect the dots and elucidate the Ring’s insidious, corrupting influence that permeates their experiences. It might fail to recognize that “Sauron”, “The Dark Lord” and “Lord of Mordor” all refer to the same malevolent entity if these terms appear in different text chunks. In essence, basic RAG has difficulty grasping the overarching themes, summaries, or abstract concepts that span a document or a collection of documents.
The “Lost in Translation” Effect
Basic RAG systems often falter when confronted with abstract concepts, themes, and summaries. For instance, if you were to request a summary of the distinct ruling systems of Gondor, Rohan, and the Shire, a basic RAG system would likely struggle to synthesize the information effectively. It might be able to state that “Denethor is the Steward of Gondor,” but it may not be able to connect that fact to the broader concept of Gondor’s governance, nor articulate how it differs from the systems in place in other realms of Middle-earth.
Enter the Knowledge Graph: Making Connections Like a Human Brain
So, how do we transcend these limitations? How do we impart to AI the ability to discern connections between ideas, much like the human mind does? The key lies in a concept called a Knowledge Graph.
Envision a Knowledge Graph as an expansive mind map—a network of interconnected ideas. Unlike traditional systems that merely process words, a Knowledge Graph comprehends entities—people, places, things, concepts—and the intricate relationships that bind them together.
Let’s return to the realm of Middle-earth to illustrate this concept:
- Instead of simply noting “Frodo” and “Ring” in separate text fragments, a Knowledge Graph grasps that “Frodo” carries the “One Ring.”
- It discerns that “Gandalf” serves as a mentor to “Frodo.”
- It understands that “Gollum” was corrupted by the “One Ring” and obsessively seeks to possess it once more.
Analogy: Picture a detective’s evidence board, where photographs of suspects are linked by strings to locations, motives, and other pertinent clues. A Knowledge Graph operates in a similar fashion, connecting the dots to unveil a more comprehensive and insightful picture.
LLMs to the Rescue
Historically, the creation of these Knowledge Graphs was a laborious, manual undertaking. However, this is where things become truly exciting: Large Language Models are now being harnessed to automatically construct Knowledge Graphs from textual data. This represents a monumental leap forward, as it dramatically simplifies the process of generating these rich, interconnected representations of information. With LLMs automatically building these knowledge graphs, we can now use them to power up our search engines.
Advanced RAG Techniques: Supercharging Search with Knowledge Graphs
Now that we’ve established the concept of a Knowledge Graph, let’s explore how it’s being leveraged to enhance RAG systems significantly. Here are some of the most promising advanced RAG techniques:
Graph RAG: Connecting the Dots
Graph RAG takes the knowledge graph and runs with it. Here’s how:
How it Works: Graph RAG employs a Knowledge Graph, constructed from a collection of documents, to answer your queries. It leverages the intricate relationships within that Knowledge Graph, enabling it to make connections that would elude basic RAG.
Two Search Strategies:
- Local Search: Imagine posing the question, “What is the significance of the palantíri?” In Graph RAG, a local search would resemble a detective meticulously examining the portion of the evidence board related to the palantíri (seeing stones). It would trace the connections surrounding that entity, uncovering details about how they were employed by Saruman, Denethor, and Sauron for communication and the exertion of influence.
- Global Search: Now, consider a broader question like, “How does the theme of power manifest throughout The Lord of the Rings?” This is where global search comes into play. It’s akin to stepping back to survey the entire evidence board. Graph RAG would identify the key communities or clusters of related concepts within the Knowledge Graph—perhaps one cluster revolving around the One Ring’s corrupting power, another around the struggle of the Free Peoples against Sauron—and then synthesize information from those clusters to furnish a comprehensive response concerning the various forms that power assumes in the narrative.
Benefits: Graph RAG excels at tackling complex questions that necessitate an understanding of relationships between entities. It can synthesize information from multiple sources and provide a more “global” understanding of a topic, much like a human expert who has immersed themselves in the entirety of The Lord of the Rings. It can also provide a summary of an entire corpus.
Drawbacks: Constructing and maintaining a Knowledge Graph can be more intricate than basic RAG, and the search process, particularly global search, can sometimes be slower.
LongRAG: Giving AI More Context
LongRAG offers a simpler, yet surprisingly powerful, solution: give the AI more context to work with.
The Idea: Instead of dissecting documents into minuscule snippets, it operates on much larger chunks of text—for example, entire chapters of The Lord of the Rings rather than isolated paragraphs.
Analogy: This is analogous to reading an entire chapter to grasp the context of a scene, rather than merely perusing a single paragraph out of context. If we only read the paragraph where Sam and Frodo encounter Gollum in the wilderness, we might not fully comprehend the history between the characters. However, if we read the entire chapter, we would understand how Gollum came to follow them and how he is manipulated into guiding them toward Mordor.
Benefits: By furnishing the AI with a more expansive context, LongRAG facilitates a more nuanced understanding of the narrative flow. It diminishes the likelihood of the AI overlooking crucial information or drawing erroneous inferences. Because it is such a simple change to make, it is easy to add to an existing RAG system.
LightRAG: Combining the Best of Both Worlds
LightRAG combines the strengths of knowledge graphs with a more streamlined approach.
Graph-Powered Retrieval: LightRAG draws inspiration from Knowledge Graphs but in a more streamlined manner. It constructs a simplified graph that links related topics and ideas within a document or a set of documents.
Two-Level Search: Similar to Graph RAG, it amalgamates focused searches for specific details with a broader search for overarching context.
Why It’s Powerful: LightRAG has demonstrated remarkable efficacy in handling complex information and diverse perspectives. For instance, if you were to inquire about the various cultures in Middle-earth, LightRAG could adeptly synthesize information about the Hobbits, Elves, Dwarves, and Men, highlighting both their unique characteristics and their intricate interactions.
Drift Search: Following the Breadcrumbs
Drift Search takes a more exploratory, ‘what if’ approach to finding information.
The “What If?” Approach: Drift Search adopts an innovative approach by generating hypothetical documents related to your query. Think of it as a form of brainstorming. If you ask, “What are the key battles in The Lord of the Rings?” Drift Search might generate hypothetical summaries of battles like the one at Helm’s Deep or the one on the Pelennor Fields, based on its understanding of the narrative.
Follow-Up Questions: It then employs these hypothetical documents as a springboard and formulates follow-up questions to delve into different facets of the topic. For example, it might ask, “Who were the key figures in the Battle of Helm’s Deep?” or “What were the ramifications of the Battle of the Pelennor Fields?” It will then seek out this information and integrate it with the knowledge it has already retrieved.
Benefits: By exploring diverse angles and tracing the “breadcrumbs” of information, Drift Search endeavors to enhance the relevance, comprehensiveness, and insightfulness of search results. It is also able to use information from across communities.
Comparing the Approaches (A Quick Summary)
Let’s briefly recap the strengths of each technique:
- Traditional RAG: Straightforward to implement and suitable for basic questions, but it grapples with complex relationships, context, and deeper meaning.
- Graph RAG: Potent for comprehending relationships and global context, furnishing insightful answers to intricate queries. However, it can be more complex to establish.
- LongRAG: A simple yet effective method for augmenting context and mitigating errors by processing larger text segments.
- LightRAG: A robust all-around performer, particularly adept at navigating complex topics and diverse perspectives by fusing focused and broad searches.
- Drift Search: An innovative technique for more dynamic and pertinent searches, achieved by exploring various angles of a query and posing follow-up questions.
It is crucial to bear in mind that Knowledge Graphs do not supplant the more traditional vector embeddings-based retrieval methods but rather serve to enhance them, either as a supplementary component or as the foundation for more sophisticated searches.
The Future of Search: From Keywords to Understanding
What implications do these advancements hold for the future of search? We are transitioning from a paradigm where we simply input keywords into a search box and hope for the best to an era where AI can genuinely comprehend the meaning underlying our questions and the documents we’re exploring.
Envision the transformative possibilities:
- More Intuitive and Insightful Search Results: Instead of merely receiving a list of links, imagine obtaining synthesized answers that directly address your queries, drawing connections between disparate sources of information.
- AI That Can Summarize Complex Topics: Imagine posing the question, “What are the primary arguments for and against utilizing the One Ring to vanquish Sauron?” and receiving a lucid, concise summary of the differing perspectives, drawing upon the entirety of The Lord of the Rings.
- Personalized Learning Experiences: Imagine an AI tutor that can tailor its explanations to your specific needs and knowledge level, predicated on its understanding of your learning style and the material you’re studying.
While these advanced RAG techniques are still in a state of evolution, they signify a monumental stride toward the realization of truly intelligent search. They are ushering us closer to a world where we can interact with information in a more natural, intuitive, and insightful manner. The future of search isn’t just about finding information; it’s about understanding it.
Notes: (*) A funny thing happened on the way to publishing this post. Due to a strange technological quirk, the Bengali word “বিচ্ছিন্ন” (bichchhino), meaning “separated,” accidentally snuck into the text. For a moment, I thought I’d discovered a new Elvish word for the Council of Elrond, worthy of Tolkien himself! It turns out even AI can have its moments of linguistic confusion.
This blog post was written through compounding and later distilling the YouTube-videos with NotebookLM and Gemini 2 Advanced. The podcast by NotebookLM takes a slightly different approach to the topic: https://notebooklm.google.com/notebook/82e205d1-e21c-4340-b20f-d43ff409c288/audio
Leave a Reply