Visual RAG — with ColPali and Screenshot embeddings
Quick overview: VisRAG is a novel vision-language model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the document to obtain text, the document is directly embedded using a VLM as an image and then retrieved to enhance the generation of a VLM.Compared to traditional text-based RAG, VisRAG maximizes the retention and utilization of the data information in the original documents, eliminating the information loss introduced during the parsing process.
In the last few months, a variety of projects have come up that rely only on visual information for answering queries about documents with complex layouts, say, from the legal or finance domains.
Screenshot embeddings like ColPali allow you to avoid lot of complex document layout pre-processing and enable Visual RAG - retrieval and query answering using only the images, no structured text extraction! There's no free lunch though!
The answer generation phase depends on visual LLMs which aren't mature yet, embedding and ingestion isn't fast on CPU yet. I think there's a long way to go before we can do truly Visual RAG! read here
Ragpipe, a lean Agentic RAG framework I’m building to enable fast experiments, now supports multiple visual RAG pipelines. See the sec10k analysis example. More coming!
Evaluating Visual RAG or, in general, vision-based retrievers, requires validating answers while being tolerant to OCR errors. Many variants and extensions of Levenshtein distance (ANLS) have been proposed in literature to systematically measure difference between expected and returned response. For validating documents with complex layouts, we need extended distance measures over not only strings but also nested data structures over strings. Read more here.
Agent Frameworks and Computer Use
Computer Use refers to the ability of an autonomous agent to understand and replicate human-like behavior in using a computer, such as navigating interfaces, executing tasks, and interacting with software applications. Computer Use is quite challenging - understanding and navigating a variety of user interfaces, lack of user's complete context, adapting to application UIs, guessing user intent correctly and many more reasons.
◉ As we are making progress with solving Computer Use more effectively, interesting use cases are cropping up. One of those is a classic problem, the ability to do UI tests with natural language. read more here.
◉ Besides Anthropic, now Google is planning to ship a Computer Use project
◉ While incredibly powerful, computer use bots may not be a panacea for web related workflows. Instead of clicking and navigating to automate workflows, it is easier to simply call backend APIs to fulfil the task. The issue is that many websites may not expose APIs directly and we may have to recover them. So, which one would you prefer: computer use agents vs API agents? Read more here
◉ New Agent frameworks keep coming up, for example AWS released their multi-agent orchestrator (MAO) framework. The architecture diagrams look pretty similar across the existing frameworks, whether it be Crew, or Autogen or Swarm or Letta or MAO, so one has to look deeper to distinguish between them. What differs?
the architecture of long- and short- term memory blocks, explicit or implicit
how agents exchange information among them: shared memory vs message passing.
how agents call tools and orchestrate among each other / handoff to each other
◉ While many frameworks highlight message passing between agents, I think blackboard-like shared memory is an elegant abstraction for understanding operational semantics of agents. Read more here.
◉ The MemGPT project was foundational piece of work, making an analogy between LLM application memory (e.g, chat history) and OS virtual memory. They've now extended / reinvented the framework to build Agents with Memory. Read more about how to design Letta agents with memory blocks here.
◉ Bootstrapping, in the context of software development, refers to a process where a simple system is used to create a more complex one. Originating from early computing, bootstrapping involved using a simple machine language program to create more complex ones. This technique, essential in the 1950s and 60s, bypassed the tedious and error-prone process of writing binary code directly. A bootstrap loader, this initial program, loaded crucial OS components into memory, enabling the computer's operation.
The idea manifested again with bootstrapping compilers: a compiler is written in a language that it itself can compile. The compiler can compile its own source code, leading to a more efficient and reliable compilation process. Bootstrapping compilers was a crucial step in the development of programming languages.
We're witnessing a similar seismic shift in the evolution of coder agents. As coder agents (OpenHands, Aider, Bolt, ...) are making progress and beconing more capable, they are bootstrapping now — using their own code to build and improve them. Read more here
Research and Industry
◉ The AlphaChip project (Google DeepMind, 2020) proposed a deep RL method capable to generating super-human chip layouts. The project's authenticity was challenged by some academics and led to some controversy. A paper from the AlphaChip team, co-authored by Jeff Dean was put out on arxiv, with a scathing criticism of unfounded academic criticism. The paper is a delightfully tongue-in-cheek and humorous read. here
◉ Top researchers and institutions often fall prey to local minima of ideas. They pursue and work on methods which everyone else is working on (hype + intellectual persuasion). A different path may bring greater rewards, but it requires the boldness to stray from the conventional wisdom. Andrey Karpathy writes that one of these digressions in the past decade was Reinforcement Learning (RL) and he recalls Yann LeCun's contrarian position that RL was only a cherry on the self-supervised learning cake ( link). Ironically, RL is making a comeback again with post-training, so dust isn't settled on that yet.
◉ Chris Molnar writes about how the tone and language of scholarly writing is way too opaque link. This is one of my pet peeves: while many of us would love to solve this problem, I think that it's a challenge that's deeply entrenched in the very fabric of ... (I'm not sure) academia? or language itself. Ironically, the academic culture has skillfully maintained a contradictory stance, promoting clarity and accessibility in theory, while perpetuating opacity in practice.
That’s a quick snapshot of the AI news from the last month. Stay tuned for the next edition of the newsletter… Don’t forget to subscribe to stay in touch!
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.