Insights from ai-PULSE 2026: Building Voice AI for Real Conversation

Garbage In, Garbage Out: Why Speaker Intelligence Matters When Training Voice AI

Vincent Molina, CEO & Co-Founder, pyannoteAI

The qualities that make a conversation feel natural also make it difficult to use as training data. People interrupt one another, speak over each other, change their tone, and switch between languages. A voice model needs exposure to this variety, but first, developers must make sense of what is happening in each recording.

That is where speaker diarization comes in. pyannoteAI’s technology identifies who speaks when, including moments of overlap. For Vincent Molina, this work is foundational to model quality: if developers misunderstand the structure of their audio, those errors can carry through into the model they train.

He gave the example of two speakers mistakenly labelled as one. A model trained to reproduce the first person’s voice might unexpectedly pick up characteristics of the second. Missed overlap can also create errors in transcription or give a speech-to-speech model a distorted picture of how people take turns.

The composition of a dataset matters as much as its accuracy. Real recordings capture spontaneous speech, accents, and emotion, while synthetic conversations can be produced more easily at scale. Both can play a role, provided they reflect the voices and languages the model is intended to handle. Identifying speakers across recordings can also reveal when a single voice is disproportionately represented.

Molina’s message was that building a convincing voice model begins long before training. The decisions made when selecting, segmenting, and labelling audio shape what the model can learn about both individual voices and the way people interact.

▶️ Watch session in full

Vincent Molina, pyannoteAI's CEO & Co-Founder, explains why identifying speakers and preparing audio data are essential steps in training reliable voice AI models.

Building the Voice Operating System

Neil Zeghidour, CEO, Gradium

For decades, interacting with computers has meant adapting to their language: typing commands, writing code, or navigating menus. Gradium CEO Neil Zeghidour argued that as AI becomes better at understanding everyday language, voice can become a more natural way to express what we want machines to do.

He traced that shift through successive generations of voice interfaces. Early assistants such as Siri could retrieve information or perform predefined tasks, but their workflows were limited to requests developers had anticipated. Later, large language models enabled more open-ended exchanges, even if early voice modes could not reliably take actions or retrieve current information.

Voice agents bring conversation and action together. Zeghidour demonstrated a drive-through ordering system that took a customer’s request, navigated a menu, and added items to a basket using tool calls. The agent could follow a conversation while interacting with an external system to complete a task.

Yet many voice agents still rely on a succession of separate components: speech recognition, a language model, and speech generation. Zeghidour argued that these cascades introduce latency and can lose information that isn’t captured in text, such as the speaker’s tone. They can also struggle with conversational signals: a brief “mm-hmm” might be mistaken for an interruption.

An alternative is speech-to-speech modelling, in which a single model takes audio as input and produces audio as output. Zeghidour demonstrated Moshi, an open-source model developed at Kyutai that can handle overlapping speech in real time. Whereas Moshi can produce a more fluid interaction but not use tools, more recent systems combine speech-to-speech interaction with tool calling.

Looking ahead, Zeghidour pointed to robotics as a further challenge. A robot in a home or factory may need to understand someone speaking from several metres away while distinguishing instructions addressed to it from nearby conversations. For voice to become an interface used across devices and environments, systems will need to navigate those complex situations as well as the words themselves.

▶️ Watch session in full

Gradium CEO Neil Zeghidour explores the architecture and conversational timing needed to build a voice operating system.

Bringing Ulysses Home: From Voice Cloning to Real-Time Conversation

Jean-Denis Zafar, Forward Deployed Engineer, ElevenLabs

A voice can do more than convey words. It can express enthusiasm, offer reassurance, or give a character a recognizable identity. For ElevenLabs' Jean-Denis Zafar, expressiveness is central to making voice AI useful across both content creation and everyday interactions.

He illustrated the possibilities with The Odyssey, narrated using a licensed clone of Sir Michael Caine’s voice. As voice cloning becomes more accessible, Zafar emphasized the importance of safeguards such as verification, watermarking, and legal agreements, alongside a marketplace where people can license their voices. Advances in voice generation, he explained, have also made it possible to combine expressiveness with a stable vocal identity.

That expressiveness matters in conversation, too. An agent responding to someone whose apartment has flooded should sound different from one helping a customer plan a holiday. Zafar demonstrated a model adapting its tone to the content of a response without being explicitly instructed to sound empathetic.
But a natural-sounding voice is only one ingredient of a useful agent. To help someone accomplish a task, it must understand their request, navigate the conversation, and access the systems needed to act. Zafar brought those elements together in a fictional demonstration, using Caine’s licensed voice for an agent helping Odysseus arrange his journey from Troy to Ithaca.

Making the exchange feel fluid also requires attention to timing. An agent needs to handle interruptions and avoid leaving the user wondering whether it is still working. When an external system is slow to respond, a simple acknowledgement such as “let me look that up” can make the delay easier to navigate.

▶️ Watch session in full

Jean-Denis Zafar, Forward Deployed Engineer, France at ElevenLabs, demonstrates how voice cloning and agentic tools can come together in a real-time conversation.

Want more insights like these?

Voice AI was just one of the many topics we covered at ai-PULSE 2026.

Explore the latest developments in AI and their applications across media, retail, defence, and more. Watch the ai-PULSE 2026 replays.


Recommended articles