Gen AI Summit 2024 | Panel Discussion: Chatbots, Copilots, Agents, Robots: The Future of AI Agents

• Tamar Yehoshua, President of Product and Technology, Glean

• Denis Yarats, Co-Founder and CTO, Perplexity

• Laurie Voss, VP, Developer Relations, LlamaIndex

• Tri Dao, Chief Scientist, Together.ai

• JC Mao, Ex-VP of Google, Partner of Fellows Fund (moderator)

The panel on "The Future of AI Agents," featuring experts Tamar Yehoshua (President of Product and Technology of Glean), Denis Yarats (Co-Founder and CTO of Perplexity), Laurie Voss (VP of Developer Relations at LlamaIndex), and Tri Dao (Chief Scientist at Together.ai), moderated by JC Mao, discussed the rapid evolution of AI agents and their capabilities in autonomously taking input, making decisions, and executing actions. They shared insights on building reliable and secure AI agents, highlighting challenges in data retrieval and evaluation within enterprise settings. Tamar emphasized the complexity of handling enterprise data with robust security, while Denis focused on the importance of speed and accuracy in providing answers. Laurie and Tri discussed the necessity of solid infrastructure, fast inference, and memory management for developing effective agents. They highlighted the competitive advantage of agility and niche focus for startups against larger tech companies. The panelists anticipated future improvements in planning, reasoning, reliability, and efficiency, such as enhanced SQL query capabilities and longer context windows for more comprehensive reasoning. They discussed the importance of fine-tuning models for specific tasks and the complementary roles of RAG and fine-tuning in enhancing agent performance. The conversation concluded with strategies for mitigating risks, ensuring accurate outputs, and the transformative potential of AI agents to significantly enhance productivity and daily life.

JC Mao: Welcome to our panel discussion on AI agents. The field of AI agents has been rapidly evolving, encompassing everything from chatbots and copilots that we interact with daily to the emerging agents and robots that will transform how we live and work. This is a very exciting week. On Monday, OpenAI announced GPT-4o, and they also showcased their amazing voice assistant demo. On Tuesday, Google kicked off their annual Google I/O, where they unveiled a slew of products and services powered by Gemini. Google DeepMind also showed off Project Astra, a personal AI assistant. So, this is really exciting. Here we are at our summit, talking about AI agents. Today, we have the pleasure of hearing from a diverse set of panelists at the forefront of the Gen AI field. With such a diverse panel, I'm sure we will have a very engaging conversation. So, without further ado, let's get started. 

As I mentioned, AI is at the forefront of everything we are doing now, and it has been evolving very quickly from traditional AI to conversational AI to agentic AI. My first question is to get everyone on the same page: What is an AI Agent, and what are the defining capabilities of AI agents?

Tamar Yehoshua: Everybody's using this term "agent," and I feel like as we hear about more agents, the definition gets blurrier and blurrier. People thought they knew what it meant, but now everyone is calling everything an agent. So, I turned to Perplexity and asked, "How would you define an agent?" It said, "Something that takes input, has sensors, gathers information, makes decisions, and takes actions," which I thought was a pretty good definition. If you want, it's autonomous, decisive, and actually takes action.

JC Mao: That's a great definition. I hope everybody agrees with this definition. So, what is your favorite AI agent?

Tamar Yehoshua: You know, agents are actually not a new thing. As we've talked about, agents are something that takes input. One of the things I use every day in the car when I'm driving is Google Assistant, which I know you [JC] worked on as well. When you get a text message, it will read it and ask if you want to reply. This wasn't using new LLM; it was using previous technology, but it's an agent, and it does multi-turn tasks.

Denis Yarats: I think my favorite one is the one we built in Perplexity. I'm biased, obviously, but I think it's something that will just do a much better search for you, answer much better questions, and tackle much harder questions. Even last year, we introduced something that does multi-step reasoning, can handle multiple steps, and even ask follow-ups like, "Can you clarify your question?" It goes much beyond that, and I'm super excited about it.

Laurie Voss: Working at Llama Index, I see people working on agents every day. I attend a lot of hackathons. Recently, my favorite agent was quite simple in execution but powerful in concept: an accessibility agent. It was a Chrome extension that reads your webpages for you and summarizes them, providing links and actions you can take on the page. One defining characteristic of an agent for me is tool use, and this one could fill in forms and follow links via a voice interface, which I found extremely impressive.

Tri Dao: I'll mention two examples. One is Devin from the Cognition Lab. Their demos capture everyone's imagination of an automatic software engineer, where you talk to Devin as if you’re talking to a junior software developer on Slack. Devin can pull up documentation, install software, and do tasks you ask it to. It’s a demo for now, but it’s impressive. Another example is a SWE agent from a team at Princeton, which does similar things using fully open-source models. These examples capture the potential economic value of what agents can do.

JC Mao: Thanks for sharing your favorite agents. I asked the same question to ChatGPT, Google Gemini, and Microsoft Copilot, and they all refused to answer. So, I’m happy our panelists shared their honest opinions. Perplexity was the only chatbot or search engine that gave me an answer. Thank you, Denis, for that. My next question is: today, most AI agents are built on top of large language models augmented with agentic workflows that support reasoning, tool use, execution, and interaction with other agents. They perform better than most baseline large language models on many tasks. I know Glean and Perplexity have been very successful in building applications or agents. Tamar and Denis, could you share your experiences and lessons learned in building Glean and Perplexity?

Denis Yarats: I think we started very simply, just asking if we could do the most basic things well. The most basic thing is just to provide an accurate answer, and can you do it fast? Usually, those two things are kind of at odds with each other; the faster you are, the lower the quality. So we really had to spend a lot of time on evaluations and building tools to understand where the answers were not great or where we had blind spots. Having a very detailed overview of your system and knowing where things are working or not is crucial, especially with LLMs, because of their stochastic nature. It's hard to reproduce results, but this is perhaps the best thing we've done because it's a very fundamental one and unlocks everything else that you do beyond.

Tamar Yehoshua: For people who aren't familiar with Glean, Glean is an enterprise AI platform that crawls all of your enterprise data, whether it's in Google Docs, Microsoft, Slack, Salesforce, JIRA, or Confluence. You can ask questions on it, search on it, or use it as an assistant to ask questions across all your enterprise data. The hardest thing about building Glean is that this is enterprise data, and enterprise data security and privacy are the most important things. We use an LLM to generate the answer, but it's the retrieval engine, the RAG, that was built over the last four years to model the enterprise data and build a knowledge graph from it. What signals do you use to know what's relevant? How do you preserve permissions and security across all these different tools? That's the foundation. We had to spend a lot of time getting that right because enterprise data is complicated. Also, to what Denis said, how do you evaluate it? How do you make sure that with the non-deterministic aspect of LLMs, do you have a way of evaluating it? In the consumer setting, people using ChatGPT understand they might get different answers or use it to generate a draft of a blog post. In the enterprise setting, people don't want something non-deterministic; you really have to get the answer right because the expectations in a business are quite different.

JC Mao: Thank you for sharing. So AI agent developers also need solid infrastructure and framework to build the agentic workflows and agents. LlamaIndex and Together.AI are in the infrastructure business. Laurie and Tri, can you share how LlamaIndex and Together.AI can help agent developers?

Tri Dao: Yes, I agree, the infrastructure layer is very important. Right now, we've seen a bunch of smaller companies and startups experimenting with this. I don't think agents are production-ready yet, but one thing we need is very fast and robust model serving. That's what we've been building at Together. Usually, agents need to process problems, retrieve specific information with RAG, deal with long contexts, use tools, and all that is in the inner loop of an application. You need very low latency and high throughput. So, having a robust inference or model serving system is necessary. Another thing is dealing with memory. Agents need to access internal documents through RAG, read and write to their scratchpad, and handle long contexts better. We've been building alternative architectures to transformers that can handle long contexts much better. This is a rapidly evolving area, and I don't think people have figured out the right abstraction to build for agents yet. But having fast inference and memory for long contexts are necessary components for people building agents.

Laurie Voss: I agree with your points. For those who aren't familiar, LlamaIndex is a framework for connecting your data to LLMs. Initially, we focused on retrieval-augmented generation. As the product matured, we discovered that the more complex your retrieval gets, the closer it becomes to being an agent. Your retrieval strategy becomes agentic in nature. Now the framework is fully focused on agents. Finding the right abstractions is a challenge. Long contexts and memory are things we are building into the framework. We absorb best practices from the industry and the latest research, and as soon as a technique proves effective, it gets integrated into the library. This becomes another strategy you can deploy from the framework to build an agent, giving you a menu of options.

JC Mao: That's great. So you are a customer-/developer-first platform. Can you also share what type of AI agents your developers are building on your platform?

Laurie Voss: We are a framework for every kind of agent. There isn't any specific type of agent we're focused on, but we are very focused on RAG. We see a lot of retrieval-based agents attempting to act as research assistants, whether in financial data, healthcare data, or other industries with a lot of unstructured data. Legal documents, for example, need to be turned into answers faster than a human can do it. One-shot doesn't work well for that stuff. An agentic approach is much more effective for answering complex questions.

JC Mao: Got it. Thanks. I know many large language models are not very good at taking actions. Tri, do you have any observations about developers building “action models”?

Tri Dao: Agents need to take actions, often called tool use. OpenAI does a reasonably good job now; for example, it can write code and start executing it. But that's still in an early stage. One trend we've seen is people fine-tuning models, especially open-source ones, for specific workflows. For example, with RAG, the model might generate relevant queries which then call a retrieval system to fetch documents. Out of the box, you can do that with prompting, but if you generate enough data, you can fine-tune these models to make function calls, use tools, etc. LLMs are intelligence engines that have ingested a lot of data, but they might not be particularly good at the tasks you care about. So, sometimes it's necessary to fine-tune for the workflow you want.

JC Mao: Got it. Thanks. So I know all the hyperscalers are also interested in building AI agents. That's been their main focus, so we would expect major improvements, if not breakthroughs, in their future releases. We've seen some from the announcements by OpenAI on Monday and Google's announcement on Tuesday, but I believe those are just teasers for what's coming next. What major improvements or breakthroughs do you anticipate from their future releases? Each of you, please name one major improvement or breakthrough you would like to see.

Denis Yarats: The very obvious one, I think, is replacing autoregressive generation with planning and reasoning over search. I think that's going to unlock a lot of potential.

Tamar Yehoshua: I would say reliability is probably the most important thing, especially for agents that are multi-turn. How do you implement some kind of self-reflection so that the model can validate itself?

Laurie Voss: The one I'm personally looking forward to involves data. As a former DBA, it frustrates me that LLMs are almost there with writing SQL queries across large databases to answer questions. In building retrieval systems in an LLM context, we usually think of vector search, but we have vast amounts of data in structured databases locked behind SQL interfaces. We're very close to LLMs being able to write flawless SQL, including complex joins, to access this data. This breakthrough is imminent, as context windows get larger and prompting techniques improve. Unlocking this richer information in a structured form will be a game-changer.

Tri Dao: I'm most excited about long context windows, especially with multi-modality where you deal with audio, images, or video. GPT-4 has a context window of 128k, and Google just announced support for a 2 million token context window. This unlocks new applications, allowing us to handle very large code bases, long documents, and more. The ability to reason over long contexts is the breakthrough I'm most excited about.

JC Mao: Long-term context windows are definitely important. Do you think 2 million tokens is long enough, or do we need to go higher?

Tri Dao: Internally, Google has had success with 10 million tokens. Developers are very creative and will push this limit. Instead of fine-tuning, people are starting to put thousands of documents in the context window and let the models reason over that. So, I think we'll keep pushing that frontier.

JC Mao: Got it. I totally agree. My next question is: As foundational models become more capable and reliable, do you think the value addition on top of these models with agentic workflows will diminish?

Tamar Yehoshua: I was at a dinner with startup founders recently, and this was a topic of discussion. As models get smarter, we have to assume that a lot of the behavior we're coding now won't be necessary in the future. I'd be very excited if models could do better reasoning and take actions without needing explicit instructions, making much of our current work obsolete.

Denis Yarats: It's all about the product and how you can leverage better models that can reason. There are unlimited ideas, only limited by your imagination. If you build a good product and enable certain features, everyone will benefit from it.

Laurie Voss: As a framework, every time the models get better, things get a lot better for us. Tasks that we previously struggled to get our software to do suddenly become easy. But in addition to the models getting better, we shouldn't sleep on the models getting faster. For instance, GPT-4o is twice as fast as GPT-4 was. In an agentic context, that means it can take twice as many turns, think twice as much, and take twice as many actions for the same latency. This can be a step change in how capable the agent appears to be because it has more opportunities before the user gets bored.

Tri Dao: I think that better and better foundation models will make agentic workflows more effective. For example, right now, a software engineer agent might need explicit prompting to plan and execute steps. Stronger models might be able to generate and write code right away without explicit instructions. However, for most applications, agents are about taking action, and foundation model providers may not cater to every single context. The general ability of the models will improve, making it easier to fine-tune or demonstrate specific workflows. Vertically integrated companies will use these models as inference engines, fine-tuning them for specific actions, and will continue to provide value in the agent workflow.

JC Mao: Yeah, I agree. As the foundational models continue to improve, the impact on the startup ecosystem won't be even. Some companies will flourish if they have deep integration and a unique value proposition, while others may struggle to differentiate themselves. The impact will definitely not be uniform. My next question is about competing with the big tech companies that are building both foundational models and AI applications. As AI startups, you need to partner with these big companies while also competing with them. Can you share your best practices on how to position your company and compete in this highly competitive marketplace?

Tamar Yehoshua: It's a very competitive marketplace. From my experience before Glean, when I was at Slack, you always have to do something that goes against the grain of what the larger companies can do. For example, Glean is a horizontal platform that integrates with everyone's products. Big companies may say they will integrate with everyone, but they are not going to be good at integrating with products outside of their ecosystem because it's not in their interest. You have to find where the incentives of larger companies are not aligned with what you are doing and double down on that.

Denis Yarats: Yeah, it's very similar. You need to build on your mission, whatever you think will work in the future, and make sure your mission takes advantage of the inefficiencies of big companies. We don't pay much attention to competition; we just try to do our best and see where we end up. It's important to think about the future, understand how models will evolve, what capabilities will be enabled, and plan your strategy around that. There’s so much to do, and the better the models, the more there is to do.

JC Mao: Google also announced they will roll out “AI Overview” in their search. What’s the impact on Perplexity?

Denis Yarats: It was great to see it; it validates our idea that it’s the right thing to do. I’ve been in A/B tests for over half a year, and while it has some advantages, Google's page is overloaded with elements vying for attention. Our approach is to minimize perplexity, making things cleaner and easier. There’s opportunity in simplicity. More competition is ultimately better for the end user.

JC Mao: Laurie and Tri, you also compete against big infrastructure players like AWS, Azure, and GCP. What is your strategy, and can you share any insights with the audience?

Laurie Voss: As an open-source framework, we embrace and extend new techniques and technologies. When new technology comes out, we integrate or absorb it into our framework. The reason startups exist is that big companies can't pivot quickly. We pride ourselves on our agility, supporting new models and technologies almost immediately, often within minutes or hours rather than days. We compete with agility.

Tri Dao: I agree. Velocity is crucial. Being a startup means moving fast and creating a tight feedback loop with customers. General AI is incredibly fast-moving, which favors pure-play startups. We saw this with the big data era, where companies like Databricks and Snowflake competed with hyperscalers. The general AI era will likely see similar dynamics as long as you move fast.

JC Mao: Thanks for sharing your strategies. It's fascinating to see the emergence of agentic capabilities as foundational models scale up. However, there are many risks that developers and society will need to address. Could you share your thoughts on how to mitigate these risks and what safeguards need to be put in place?

Denis Yarats: Yeah, I think the main thing we care about is making sure we provide accurate and truthful answers. We build everything to support that, from the way we train our models to how we verify and evaluate them. It's about ensuring we don't provide misinformation and that we deliver useful information. That's at the core of our mission, and we invest heavily in it. So, I would say that's our main safeguard: providing accurate information.

Tamar Yehoshua: For Glean, it's very similar. We want to ensure that everything we provide to an enterprise is permission-aware and secure. You don't want someone asking about the CEO's salary if they're not supposed to know. It's critical to respect the permissions of all the information. And, as Denis mentioned about hallucinations, we must ensure the data we provide is accurate and has the necessary guardrails. That's our big focus.

Tri Dao: Yeah, it's a difficult question. Using LLMs certainly opens you up to many new exploits. What I've seen work well so far is having a separate guardrail model. For example, Meta has done an incredible job with releasing Llama and Llama Guard, which checks the output of the model you're generating. You can fine-tune this guardrail model to filter out unwanted content. But it's still an open research question. LLMs give us much more power but are also more challenging to use because we haven't nailed down the abstraction.

JC Mao: Got it. We're running out of time, so I'd like each panelist to share their final thoughts on the future of AI agents.

Tamar Yehoshua: I'm really excited to see what's going to happen and how much more productive we can all be, both in our jobs and personal lives. AI will take a lot of the drudgery and friction out of things we do today. We have a document of all the workflows we want to automate with Glean, and it's exciting to think about the use cases. I believe they are all realistic and will happen within the next year.

Denis Yarats: We're just getting started. Right now, we basically have horses; in a couple of years, we'll have rocket ships. There's so much more to come. In a few years, we'll look back and see how far we've come. It's pretty cool, and there's a lot of product to build around these advancements.

Laurie Voss: Yeah, roll on GPT-5 is what I say. Much of the agentic stuff right now is at the boundary of what current models can do. Even modest improvements can make agents more reliable and capable. I'm really looking forward to that.

Tri Dao: The barrier to entry is lower than ever. It's now really easy to try this stuff out, whether you want to use closed models from OpenAI, Anthropic, or Google, or open models. Many service providers like Together, Vercel, LangChain, and LlamaIndex make it easy to build a demo in a day without being super technical. It’s getting easier to use these tools, so my message is to try them out.

JC Mao: Thank you. I'd like to thank all the panelists for sharing their insights and experiences. It's an exciting time for AI. Thank you.