Adaptive Soundtracks in Games

Most video games these days have some kind of dynamic soundtrack, where the music reacts to what's happening on screen. Think of the way that music crossfades from calm to tense when you enter a battle... then drops back down to calm when you kill the final enemy in a room. For most games, that's about as good as you're gonna get. You might get something a little bit more nuanced like Red Dead Redemption's interweaving stems, or the chopped up orchestral score of Dead Space, or the way that different layers of Amon Tobin's Chaos Theory score weaves into one another as you get spotted and then enter combat.

But I want to talk about some of the more inventive ways that game makers have used interactive audio to make games feel even more dynamic and let games provide feedback, or create a different atmosphere just by subtly tweaking the soundtrack. Nintendo is obsessed with this stuff, especially in the Mario and Zelda games. Check out this map screen from New Super Mario Bros U, where the music in each world uses the same melody but the game seamlessly switches the instruments as you move between screens.

Nintendo did the same thing in Skyward Sword. In the shop on Skyloft, the music shifts tone as you move from store to store.

And in certain dungeons, like Sky View Temple, the music is the same throughout but different elements are added as you enter different rooms.

DS game Spirit Tracks has lots of good examples. Listen to how the train section music dynamically changes as you speed up.

And in the Spirit Tower, more instruments are added to the song as you ascend this central staircase - which gets taller and taller as you complete the game - giving you a truly epic feel when you reach the final floor towards the end of your adventure.

There are loads of good examples in the Super Mario Galaxy games, like these collectable notes that play a song as you pick them up... And this ball rolling level where the tempo of the soundtrack shifts as you speed up and slow down. In other Mario games, you have the bongos that enter the mix when Mario jumps on Yoshi… Super Mario 64's clever auditory illusion where the pitch is always rising as you run up Bowser's endless staircase… Or Luigi humming… Or whistling… Over the Luigi's Mansion soundtrack depending on the level of danger. Or the Mickey Mousing in the New Super Mario Bros games, where the enemies dance to the music.

Oh, and have you ever noticed how the music on Mount Wario in Mario Kart 8 just feels so perfect, like a live orchestra is playing along to your every move? That's actually a super simple trick - the race has four different soundtracks that cross fade into one another as you cross invisible trip wires on the track. So you've got this song when you're first descending down the mountain… This song when you enter the cave… This crazy violin song when you're avoiding the trees… And this, when head towards the finish line… Which all results in something wonderfully cinematic, and it works perfectly no matter how fast or slow you go. That's the magic of an adaptive score and Nintendo uses them well. Pretty much all of these examples have no real effect on gameplay but they add character and texture to the world. They often make Zelda games feel more epic, and they make Mario games feel more playful and toylike.

It's not just Nintendo, of course. Adaptive soundtracks like this been around since Monkey Island 2 when LucasArts used something called iMUSE to seamlessly transition between songs for drama, jokes, and exploration. It's most evident in the area Woodtick where entering different rooms causes the song to transition into different themes.

And when you wake up these sleepy pirates, an accordion is added to the mix. Like all of Monkey Island, it was inspired by Disneyland rides. The ones that fade naturally from one soundtrack to the next. Nintendo's old pal Rare did this a lot, too. Banjo Kazooie, for example, has music that fades from area to area… Or changes tone when you dive underwater. A more recent example is Portal 2. Not only does it apply nifty effects to the music when you jump on the blue gel… And run on the orange gel… But in some test chambers the different puzzle bits sing out as you activate them. Watch how directing lasers into these sensors produces a new soundtrack.

It's not just a nice auditory gimmick, but it tells you that you're on the right path and if you mess up your current progress, you'll know because you'll hear it. It might have been inspired by Auditorium which does something pretty similar…

But all goes to show how audio is an often forgotten way of providing negative and positive feedback, during the puzzle solving process of a game. I also like how Luftrausers changes the instrumentation of its song depending on the elements your ship to reinforce the sense of customisation... The way Wipeout HD applies a lowpass filter to the background music while your craft is using a shield to amplify the effect of being in a cocoon...

And the way LA Noire's soundtrack fades out when you wander outside the crime scene, to let you know that you're not gonna find any more clues. The thing about a soundtrack is that it's inherently artificial. I mean, if you went to a tropical island and starting murdering dudes, you wouldn't hear a crazy war drum soundtrack booming from the sky. So game makers should be free to experiment with audio without sacrificing the authenticity of their murder simulators. And they can make adaptive soundtracks that are bit more inventive than just going a bit noisy during fights.

Perhaps in an action game, each enemy could represent one instrument in the fight scene soundtrack which fades out each time you kill an enemy. I don't know, I'm not a composer, but adaptive and interactive audio is something that is exclusive to games and that's exciting, so here's hoping more game makers experiment with using contextual soundtracks like this, in new, creative ways in upcoming games.