Voice calls

Live voice is where the architecture earns its keep: a call needs sub-second conversational reflexes and deep reasoning, and no single loop provides both. The runtime splits the job between two brains that share one conversation.

​

The two brains

The fast brain is Assistant(Agent) in unify/conversation_manager/medium_scripts/call.py — a LiveKit Agents worker running in its own subprocess. It owns the real-time loop: Deepgram STT (with diarization), Silero VAD and turn detection, a lightweight LLM for conversational responses, and TTS through Cartesia or ElevenLabs depending on the assistant’s voice_provider. Its job is to keep the conversation feeling human right now. The slow brain is the ordinary ConversationManager turn loop — same code as for text, now fed per-utterance events. It owns judgment: what to actually say, when to start work, when to end the call. They communicate over a Unix domain socket (domains/ipc_socket.py):

• Upward: each completed user turn publishes an inbound utterance event (InboundPhoneUtterance, InboundUnifyMeetUtterance, …), which schedules a slow-brain turn like any other message.
• Downward: the slow brain speaks through its guide_voice_agent tool, which publishes a FastBrainNotification — with should_speak=True the fast brain delivers the message verbatim over TTS, and an optional fast_brain_guidance field steers how it handles the next few turns on its own.

​

Call lifecycle

LivekitCallManager (domains/call_manager.py) owns session mechanics: it names rooms (unity_{assistant_id}_{medium}), prewarms a persistent worker subprocess, dispatches the agent into a room per call, and bridges broker events into the subprocess. On the gateway side, inbound phone calls arrive via twilio_call_webhook (adapters/twilio.py), which sets up the Twilio↔LiveKit SIP bridge; outbound calls go through the phone channel (send_call, dispatch_livekit_agent — SIP trunks are created per provisioned number at purchase time). Unify Meet sessions skip telephony entirely: the Console joins the LiveKit room directly, rung via ring_unify_meet() on the ConversationManager (with its ~25-second no-answer fallback to text). Google Meet and Teams meetings are joined via browser automation rather than SIP.

​

The feel of a call, mechanically

• Fillers. When a user turn needs the slow brain and it hasn’t answered yet, the fast brain schedules a short buffer phrase (“one moment…”) — _schedule_buffer_filler in call.py — suppressed if the slow brain already responded.
• Turn classification. Between full slow-brain turns, the fast brain classifies each user turn (select_fast_brain_turn in domains/fast_brain_turn.py): smalltalk it can answer itself, silence, deferral to the slow brain, or — when the hang-up gate is armed — a natural close.
• Barge-in. If the user talks over TTS, a VoiceInterrupt event records the spoken_prefix actually delivered and the unheard_remainder, so the slow brain knows exactly what the user did and didn’t hear and can re-weave the rest.
• No turn preemption. Speaking mid-turn does not cancel the running slow-brain turn. The new input queues as the single pending turn (see the conversation runtime), and the fast brain covers the gap. Urgency-based preemption was tried and removed.
• Proactive speech. During long silences while work runs, ProactiveSpeech (domains/proactive_speech.py) decides whether the assistant should say something unprompted.

​

Who said what

Attribution comes from the meeting roster and the platform’s own participant signals — never from matching voices. In a browser meeting the two halves arrive separately: the transcriber tags each finalised utterance with an anonymous diarization id (S0, S1), while the meeting platform reports spans of who was speaking over the Recall relay. meet_speaker_map.py pairs them by overlap of time spans rather than by sampling who is speaking when a final lands — by then the speaker has stopped and the platform has already sent speech_off, so instantaneous sampling fires only when someone talks over the previous speaker’s tail, exactly when it’s most likely to name the wrong person. Votes accumulate rather than binding on first sight, because diarization ids are per-call and drift. Elsewhere, an unresolved turn is attributed to the call contact and the slow brain infers the real speaker from the conversation. Enrolled voice embeddings are not compared against live audio to decide who is speaking. That path was removed after production audio showed the cosines conflated distinct speakers — including the assistant’s own TTS voice — about as often as they separated them, so a matched label was as likely to be wrong as right. SpeakerTracker (speaker_id.py) remains as capture-only machinery: clustering keeps auto-enrollment single-voice pure and counts distinct voices behind VoiceEnrollmentCaptured / VoiceEnrollmentSuggested.

​

Hang-up semantics

Ending a call is a two-brain negotiation with three tools: This split is why short scripted calls (like onboarding channel tests) end gracefully on their own, while open-ended calls never end abruptly unless the slow brain explicitly decides to.

​

Screen share during Meet

Screen-share state flows as events (AssistantScreenShareStarted/Stopped, UserScreenShareStarted/Stopped, webcam variants) from the Console through the internal adapter. Frames from shared user tracks are captured (UserTrackCaptureManager in call.py), buffered on the ConversationManager, and attached to the next slow-brain turn as vision input — which is how “look at my screen and tell me what’s wrong” actually works. Screenshots also land on disk under Screenshots/ for the Actor to reference during act work.