Interactive Mixing Project

Over the summer I have been working on my final masters project which is an investigation into interactive real-time mixing in games. The two main issues I have been looking at are;
-the potential for using film style mix techniques in games,
-what extent can currently available middleware tools implement interactive mixing.

There has already been a certain degree of interactive mixing done in games, even back on PS2 games such as Scarface and MGS3, and especially more recently with the sophisticated mixing systems of 'HDR audio' in DICE's 'Frostbite' engine and Radical's 'AudioBuilder'.
Yet it seems there is still a lot for games to achieve in terms of fully utilising the mix ideas and techniques that have been proven to be effective in film sound, i.e. use of focus, contrast, silence to express emotions and subjectivity of the character/s [1].
It can be very important and useful to develop intelligent mixing systems in games like 'HDR audio' that ensure a mix doesn't become muddy. Perhaps though, games could also benefit from developing the use of narrative / event related mix moments that employ mix decisions based on emotional subjective reasons[2]...
There are many great films that have unique mix moments that can enhance the narrative, story, and dramatic effect - but I think part of the reason the mix moments are so effective is because they are unique within the film. Always a great example is the tiger scene in Apocalypse Now.
So maybe one way games can learn from this is to have some mix moments that are unique to particular points in the game so they can take advantage of the narrative, story, and character development. The mixes would still be interactive because they would be triggered by the player's actions and also feed off of game variables.
This could not only help to diversify the audio landscape, but also add another string to the bow of audio in games helping to enhance the overall experience.

So from this notion of working mix moments into games I have been putting together some examples.
I first identified mix techniques used in films then looked for scenarios in games where similar mix techniques could be applied. I then captured game footage with which to create a guideline mix. This involved spotting the footage and adding the different sound stems to re-create how it would sound in game. From this point I could then play with the mix and add automation to set out how I wanted it to sound in game.
I then tried to implement these mix moments in game using audio middleware tools.

Here are some of the guideline mixes I have made. I have included the version without the mix moment first for each example to show the difference.

This first example is a scenario from Doom3, it shows the player going into a room with enemies lurking in the darkness. The mix makes use of delay and reverb to characterise the enemy sounds through the character's POV, helping to emphasise emotions of fear/paranoia etc.

Mix features OFF...


Mix features ON...


----------------------------------------------
The next example is also a scenario from Doom3, the player has to drop down into the room and at some point a monster might make an appearance. The mix makes use of (relative)silence leading up to the event to provide a dramatic emphasis through the contrast created. Again filtering the sound through the character's subjective POV.

Mix features OFF...


Mix features ON...


An interesting issue raised by Simon Ashby, product director of Wwise at Audiokinetic, in Rob Bridgett's article, is that difference in styles of gameplay between a cautious player and an extremely bold player for example will produce different results in the soundtrack. So this needs to be taken into consideration when creating an interactive mix, especially if unique mix moments based around the narrative are being developed. There is a balance that would hopefully be struck, some mix moments could be tweaked so that they work for nearly all types of player. Perhaps it may be a fact though that the intended mix may not be realised in game 100% of the time, on the flip side however, this could work as an advantage because if the player plays the game a second or third time the difference in the mix may help give a varied experience. One thing not to forget of course is that the player always needs to hear the required information, i.e. voice instructions, and also that the mix never becomes a sonic mess - unless it's for a desired effect that is.

So here I have an example of the silence scenario again but with footage of faster gameplay around when the mix moment occurs. The previous example showed the player exploring the dark area that had been dropped into. This next example shows the player running straight through into the lit room.
Hopefully it demonstrates how this mix moment would work for different styles of play.

Mix features OFF...



Mix features ON...


----------------------------------------------
Some other links to interactive mixing and subjective game audio type stuff....
Recreating Reality
The Future Of Game Audio
Game Audio Pro
Subtlety and Silence
Ducking

Dynamic ambience in 'Prototype'

I have yet to play the game, but from what I've seen and read I can't wait to play it!

The Gamautra feature explaining how the sound team at Radical went about creating and implementing the ambience in Prototype describes some useful interactive real-time mixing techniques and offers insight into the workings of their proprietary audio tool AudioBuilder.

One section that particularly interested me is how they use 18 channels of ambience simultaneously streaming from disc and dynamically mix the levels of individual tracks or groups depending on game variables - more specifically the density of pedestrians, infected hordes, and traffic building up around the player.
The system also uses these density values along with the position of the sounds relative to the player to dynamically mix the ambience among the quadrophonic channel set-up to help provide a sense of orientation.

This interactive mix of the individual ambience elements seems to be entirely reactionary to real-time variables. So, though I'm sure it provides an immersive ambience that feels alive, I can't help but think it would be nice to override this for events or moments in the game. For example, it could be effective to be able to duck all ambience apart from the infected hordes at certain points in the game to achieve a dramatic subjective type moment.

The overall ambience is bussed to the main mixer system and does allow for control of the ambience bus within the overall mix to apply filtering for cinematics and special game modes. However it would have been pretty cool to have the control to exclude a certain element of the ambience that is filtered, giving the ability to inject that extra bit of emotional delivery when neccessary.
Perhaps the question remains, would this be apporopriate to the gameplay or the character and story? I'm not exactly sure as I haven't played it but it would be interesting to see if that extra level of flexibilty in the mix could intensify the cinematic qualities in the gameplay and possibly even help provide more variation in the soundscape...

All things considered though it's great to see a complex and intelligent mix being implemented and also described so openly to the world!

SoundSeed - A New Hope

My blog this week is not quite a new innovative use of interactive audio, more a new innovative tool for interactive audio.

SoundSeed has very recently been announced by Audiokinetic and it certainly seems to be an innovative way get more varied sound from very little memory useage. Thus giving the ability to creatively push the limits of game audio.

SoundSeed is a family of sound generators, developed by Audiokinetic, that can generate unlimited variations of a single sample using DSP technology.
There will be different modules for SoundSeed based on different types of sounds, the first to be released is Impact. This module deals with impact sounds such as footsteps, sword clangs etc.
It works as a plug-in for Wwise although is actually split into two parts- a stand alone modeler and a plug-in.
This is because the workings of SoundSeed Impact is based around analyzing the sound file and splitting it into two parts- a residual sound and a parametric model.

The modeler analyzes the 'resonant modes' of the sound and extracts this data in order to output a parametric model that can recreate the original and also be modified to create variations of the original sound.
The modeler also outputs the residual sound which is the original sound but with all the resonant values removed.

The Wwise plug-in part of SoundSeed Impact uses the modified parametric model in conjunction with the residual sound to produce a completely different sound!

What's also exciting is that SoundSeed can control parameters of the sound based on game elements such as physics or artificial intelligence!!
It also has a function to control the quality of a sound meaning you can modify the runtime CPU usage based on the importance of a sound or the distance of the source from the listener.

Here is one of the only examples available from Audiokinetic of what SoundSeed Impact can do;



Not quite as impressive as one might of thought but I'm sure the actual implementation of it will produce some great results and game developers Realtime Worlds are using the system in their most recent game APB.
Realtime Worlds report that they are really happy with SoundSeed Impact so far and it is delivering on the promise of reducing the memory footprint.

Info from;
http://www.audiokinetic.com/
http://www.developmag.com/news/30708/Audiokinetic-unveils-SoundSeed

Next-Gen Audio

A nice thorough look at next-gen audio from Alexander Brandon!

http://www.gamasutra.com/view/feature/3814/nextgen_audio_squareoff_.php?page=1

Non-repetitive sound design - Crysis

Crysis is a great example of non-repetitive sound design, this is true for the ambience/ sound effects as well as the music.
For a start there were 3500 sound events used, nearly 15000 individual sound files, all in all taking up 1.7GB on the DVD disc!
To create a non repetitive ambience the sound designers have done much more than randomise timing, pitch and velocity of sounds. They have marked across the map for sounds, overlapping areas and placing them inside each other as well as prioritising sounds based upon the location of the player. To add realism on top of fighting repetition the ambience has been given dynamic behaviour, for example a bird sound ends when gunshots are fired. Pretty sweet!
The music is very cinematic and succeeds in being non-repetitive. The sound designers apparently used a system of nodes. I assume this links in with the varying levels of intensity in the score to produce arrangements of music that will be slightly different every time you are in action of a defined intensity.
Also worth noting is that even though the music was designed to crossfade seamlessly, they still covered the transitions with sound effects wherever possible so not to jar the player. I find this to be a really cool idea, I think it's an inventive way of dealing with transitions.

The Gamasutra post about the Crysis audio team speaking at the GDC this year is definitely worth a look http://www.gamasutra.com/php-bin/news_index.php?story=17530