Chatting with a Podcast Episode: Engineering Interactive Conversations

In previous posts, we've detailed our methods for generating and curating high-quality podcast content, covering aspects such as content filtering, transcript generation, and audio synthesis. Today, we're focusing on another facet of our product: allowing users to interact with the content of their podcast via chat. We'll explain the design of this system, especially how tool calling helps ensure good quality and low latency.

From Passive Consumption to Active Interaction

Traditionally, podcasts have been a passive listening experience. Our chat feature lets listeners interact with the content. Users can ask questions or dive deeper into topics that interest them.

Listeners often come across complicated topics or unfamiliar terms that prompt immediate questions. With interactive chat, users can directly reference specific points from episode transcripts. Our system quickly finds relevant sections, allowing for precise responses and making the podcast experience more engaging and dynamic.

Technical Challenges and Solutions

When using a language model, we face the practical constraint of token limits. Optimizing token efficiency is essential because large-context language models suffer performance degradation when overloaded. To solve this, we selectively reduce context, balancing detail and speed. We also chain language models, using smaller, cheaper models to refine context before passing it to larger, more powerful models. Pushing more tokens to the smaller models also helps us save money.

Managing context, especially in long conversations, presents a significant challenge. The context is what we provide as inputs to a language model during each request. It is impossible to use an append-only context because there are constraints corresponding to each LLM. We employ truncation and prompt management to maintain coherent interactions.

Constantly changing the input context window on an LLM can result in fragmented conversations where continuity breaks down. For example, if the LLM generates a message based on facts from a specific article, but when generating the next message it doesn’t have access to that same article, it will reasonably infer from that context that it should output unsupported facts. To counter this, we implement a "working memory" system that tracks tool-calling references, which tracks what information was used when, giving the model the sufficient context to understand the conversation.

Flexible and Reliable LLM Infrastructure

Another important part of our approach is maintaining an infrastructure that can switch dynamically between multiple LLM providers, enabling us to respond quickly to service disruptions, outages, performance issues, or pricing changes and ensuring reliability and consistent availability. This setup also makes experimentation easy. By being able to test and compare factors like latency, accuracy, and cost-efficiency across different providers, we are able to quickly iterate and improve system performance and efficiency. For example, we discovered limitations with the Claude 3.5 model regarding verbosity. As newer models became available, our flexible system allowed easy upgrades to improve user experience.

Global Rate Limit Management

Managing global rate limits imposed by external providers is another key challenge, as we often use the same LLM across many services. We built an internal tracking system to maintain service continuity and performance. By monitoring API usage and internally queuing requests, we prevent rate limit issues and prioritize critical platform features like agent chat.

Conversation Initiation

Nothing feels worse than bad latency, so as soon as a user opens the chat window, we start fetching all the relevant information about that episode, and send the first message.

We also generate several suggested questions for the user, based on the content of the episode. In our experience, users can feel intimidated when presented with a blank text box, and suggestions make using it lower friction, as well as giving the user context for what sorts of inputs are expected.

Integration and Application of Tool Calling

One important feature of interactive conversations is tool calling, which lets the chat system use external resources like detailed show notes, linked webpages, or emails mentioned during an episode. These resources are pulled in automatically based on user questions or conversation context, adding depth to the interaction. For example, if listeners want more detail on an article mentioned in an episode, the chat system calls up the relevant information. This seamless integration allows listeners to move smoothly between general conversations and deeper explorations without interrupting the flow.

Beyond Text Interaction

Text-based chat is practical and effective, but there are inherent trade-offs compared to speech interaction. People typically read faster than they can listen and speak faster than they can type. Additionally, spoken content often includes nuances and subtleties absent in text. However, there are times when speaking isn't practical—such as in public places or noisy environments. Because each mode of interaction has its strengths and limitations, it's important to expand our interactivity to include speech-based interactions.

Real-time speech interactions introduce additional technical challenges. They tend to have higher latency, increased infrastructure costs, and reduced control over model behavior compared to text interactions. These factors make real-time speech systems more complex and expensive. But in sum, we find real-time, natural speech interaction so compelling that we are actively designing innovative and cost-effective solutions. Among other things, this technical foundation will enable the world's best Interactive Podcasts.

We at Continua aim to build the best AI chat experience across modalities and surfaces. By improving integration with external tools, reducing latency, and maintaining flexible infrastructure, we're enhancing listener engagement. Stay tuned as we continue exploring new interactive features and technologies, transforming passive podcast listening into dynamic conversations.