Séance

  • Voice AI and telephony
  • AI Experiments
  • 2026
  • Open source

Next.js, TypeScript, WebRTC, D-ID and 1 more

Source code

Séance is a webcam parlor game. You hold up an object, a chipped mug or somebody else’s hairbrush. Madame Esmé Voss, a Victorian medium dead since 1881, channels its haunted history out loud. I built it for the D-ID and ElevenLabs hackathon, and the whole point was to make vision load-bearing. Take the camera away and there is no game left.

Most avatar demos bolt vision on afterward, so the character narrates in general terms and never names the chip on the rim. That is the moment the illusion dies. I wanted the character to react to the object, and then to the face the player makes when she gets it right. That means the seeing and the speaking have to sit in one conversation, not two.

calltranscript

# held up to the camera: a chipped enamel mugEsme Oh. That crack on the rim. It was dropped in anger.Player It was my grandmother's.Esme And she kept it after the argument. That says plenty.Player How could you possibly know that?Esme I am dead, dear. I am not unobservant.Player Please do not tell my mother.prophecy card: "the mug that outlived the quarrel"

6 turnsone call

A webcam parlor game powered by expressive AI avatars and real-time ElevenLabs agents.
Chapter
Voice AI and telephony
Category
AI Experiments
Year
Availability
Open source
Last commit
stack5 parts
  • Next.js
  • TypeScript
  • WebRTC
  • D-ID
  • ElevenLabs

AI, WebRTC, ElevenLabs, Avatars, Game

Why the native bridge

Three ways exist to put an ElevenLabs voice on a D-ID face. I worked through all of them before writing the connection.

ApproachVerdict
Relay ElevenLabs audio per turnRejected. speak() takes only a static audio_url, so every turn pays an upload
D-ID runs the model, ElevenLabs supplies the voiceRejected. That uses ElevenLabs as a voice, not as the agent
A D-ID agent bound natively to an ElevenLabs AgentChosen. Audio bridges server side and vision arrives with it

The deciding fact is that the SDK exposes only {type:'text'} and {type:'audio', audio_url} script shapes. No streaming audio ingestion exists, so a live ElevenLabs track cannot be piped in from the page.

How she sees

scripts/create-agent.mjs runs once. It stores the ElevenLabs key in D-ID’s secret vault. Then it posts to /v2/agents/integrations/elevenlabs with an expressive presenter, the external agent id and vision: {enabled: true}, and polls until the agent is ready. At runtime src/lib/did.ts wraps @d-id/client-sdk, imported dynamically because it touches window. The browser captures microphone and camera itself, then calls publishMicrophoneStream and publishCameraStream. D-ID forwards the microphone to ElevenLabs and samples camera frames into that same conversation as system updates. The ElevenLabs key never reaches the page. The D-ID client key is domain scoped and served from /api/did-credentials, which keeps it out of the static bundle and leaves it rotatable.

Latency in costume

The phase machine runs dormant, summoning, present, channeling, revealing and prophecy, with a lost state on failure. Most transitions come from onAgentActivityStateChange, which reports IDLE, LOADING, TALKING or TOOL_ACTIVE. Channeling is the exception, because it fires locally. src/lib/motion.ts downsamples the self-view to a 32 by 24 grayscale buffer six times a second and watches the mean frame difference. A burst above 0.06 that then settles below 0.025 for three ticks means an object was presented and held still. The candle flares and the smoke rises before Esmé has said anything. That gap is the round trip to the vision model, wearing a costume.

Decisions and costs

  • Let the platform own the conversation. ElevenLabs Agents runs speech to text, the persona, the voice and turn-taking, so the browser holds no conversation state. Tuning any of it then happens in a dashboard rather than in this repository.
  • Keep vision ambient rather than structured. D-ID samples frames on its own schedule, and chat(text) sends a full user turn, so there is no quiet context channel. The game cannot hand her a clean object description without spending the player’s turn.
  • Make the Gemini route optional. /api/vision returns 501 when GEMINI_API_KEY is unset and the client skips it, so the prophecy card loses its label and nothing else breaks. One analyze() function holds the provider.
  • Treat manual interrupt as best effort. interrupt() sits inside a try block because it may be inert on the expressive path, which is the piece I would revisit.

Setup and limits

Setup is npm install, an .env.local carrying DID_API_KEY, ELEVENLABS_API_KEY and ELEVENLABS_AGENT_ID, then npm run setup:agent, which prints DID_AGENT_ID and DID_CLIENT_KEY to paste back. A paid D-ID plan is required, because the native ElevenLabs integration needs one. Deploying also means adding the production domain to the client key’s allowed domains in D-ID Studio. The architecture notes list what still wants checking against a live account, including the /secrets body shape, which is why the setup script tries two variants. The source is MIT licensed, and D-ID, ElevenLabs and Gemini remain under their own terms.

All 43 projects