← All posts
Changelog

Designing a game's soundscape with AI-generated sound effects

For the first couple of weeks of trivai's life, the sound button was a lie. It sat in the header looking clickable, and if you hovered it, a tooltip gently admitted "coming soon." We shipped the button before the sounds the way you buy running shoes before you start running: mostly as a threat to your future self.

The sounds exist now. Twenty-nine of them, covering everything from the first tile tap to the endgame podium, and every single one was AI-generated from a text prompt. This post is about what that process was actually like, because almost none of it worked the way we expected.

The prompt is the instrument

We treated each generation prompt as a design artifact: written, argued over, versioned, and kept in the spec next to the sound it produced. Getting good audio out of a text-to-SFX model (we used ElevenLabs) turned out to have its own grammar, learned one failed render at a time:

  • Onomatopoeia fails. Ask for a "ker-chunk" and you get a single thud. The model doesn't hear the word; it reads it. What works is describing sequenced physical events: "first a wooden sliding scrape, then immediately a deep muted impact, two distinct beats."
  • Genre words drag their whole aesthetic along. We asked for a "dubstep LFO" wobble and got festival aggression. Naming gear instead ("vintage Rhodes through a gentle auto-filter") got warmth. The model associates, so you have to choose associations on purpose.
  • Reverb is a place, not an effect. "Add reverb" does little. "As if in a quiet temple hall" builds you a room.
  • Never ask for "soft." Loudness words are a trap: request a soft sound and the model renders nearly inaudible audio. Loudness belongs to the gain slider at playback. In the prompt you describe restrained timbre, then ask for it "recorded at a clear strong level."
  • Negatives are fences. "No reverb tail, no buzzer, no metallic ring" keeps the model from wandering into its favorite clichés.

The best prompt of the whole project didn't come from any of that method. The board transition needed a whoosh that was big and soft at the same time, which is nearly a contradiction; every attempt was either a jet or a whisper. Dylan suggested "like an owl landing," owls being nature's engineers of exactly this problem. It worked on the first render, and it's the voice of the board to this day. The lesson we took: the model has heard more of the world than any effects menu has words for. Describe the world.

Tuned against the real thing

A sound that's lovely in a file browser can be wrong in a game. Every sound was auditioned inside dev sandboxes that drive the actual components: the real board shuffle, the real question modal, the real steal panel, the real trap reveal, with live sliders for gain and pacing. When a value finally felt right, the slider setting was baked in as the shipped default. Nothing was tuned in the abstract; everything was settled by ear, against motion.

The great latency hunt

The hardest week was the one where every sound felt late. Not broken, just... late, like a dubbed film. It turned out to be four separate problems stacked on top of each other, and we had to find them in order:

  1. The browser was slow to start. Spinning up an HTML audio element costs tens of milliseconds. The fix was the Web Audio API: decode every file into a buffer up front, then schedule playback on the audio clock.
  2. The files themselves were late. Generated audio arrives with its own padding and slow attacks. So we analyze each file for its perceptual anchor, the moment your ear actually registers it, and align that to the visual beat instead of the file's first sample. A slide is anchored to its onset; an impact is anchored to its loudest instant.
  3. The hardware was late. Audio output has real device latency, around 38ms on wired outputs and 150ms or more on Bluetooth. The scheduler measures it and compensates.
  4. And underneath all three: the first sound was scheduled at the animation's start, when the ear expected it at the board's first visible movement. Even a perfectly synced sound is late if it's synced to the wrong moment.
visual beatlead-in the file arrived withperceptual anchor
Sounds are scheduled so their perceptual anchor, not their file start, lands on the visual beat.

The sounds that didn't make it

Some of our favorite decisions were deletions.

The original tile-select sound was a pleasant little marimba note, and it lasted right up until Dylan pointed out it sounded exactly like an airplane seatbelt chime. It did. It could not be unheard. The deeper problem is that almost any single pitched mallet note drifts cabin-ward, so the replacement isn't a nicer note; it's not a note at all, just a small non-tonal contact sound, like a good keycap.

A hot-streak fanfare was drafted, and cut. When you chain correct answers, bonus rings bloom on the tiles around you, and giving that the sound turned out to say more than a trumpet ever did. The streak's audio identity is now its mechanical consequence.

A looping background ambience was fully prototyped, sounded rather nice, and was rejected anyway, because the soundscape has one absolute rule: nothing loops. Every sound is caused by something. A hum that plays forever is caused by nothing, and your brain files it under fridge.

The trap-sprung sound went the other way and escalated. It started life as a latch click, which lasted until we heard it against the prismatic confetti of the actual trap reveal, where it read as someone closing a cupboard during a fireworks show. The shipped version is a deep seal bursting open with a full resonant boom. When the visual is magic, the sound has to commit.

Then the sound started designing the game

The strangest part came last: the audio began sending changes back into the game. A mid-game board compaction had a lovely warble with nothing to point at, so the category headers gained a glow pulse for the sound to anchor to. The tile-tap felt stepped on until the spec added 350ms between the click and the visual transition; the tap gets its moment, then everything moves. And on the loading screen, every star in the constellation plays the same file pitch-stepped one semitone higher than the last, so each constellation quietly builds its own ascending scale. None of that was in any plan. All of it exists because a sound asked for it.

Roughly thirty rounds of generation, five audition sandboxes, a stack of pull requests, and one chime lost to aviation. The mute button in the header does something now, and we're a little proud that most players will never notice any of this, which is the point. Good game audio isn't heard. It's believed.

Type six topics and listen for the owl.

Put it to the test.

Type six topics you actually care about and see what a researched board feels like — every question checked against the bar above.

12 free tokens on signup · No card required