Interactive Mixing Project

Over the summer I have been working on my final masters project which is an investigation into interactive real-time mixing in games. The two main issues I have been looking at are;
-the potential for using film style mix techniques in games,
-what extent can currently available middleware tools implement interactive mixing.

There has already been a certain degree of interactive mixing done in games, even back on PS2 games such as Scarface and MGS3, and especially more recently with the sophisticated mixing systems of 'HDR audio' in DICE's 'Frostbite' engine and Radical's 'AudioBuilder'.
Yet it seems there is still a lot for games to achieve in terms of fully utilising the mix ideas and techniques that have been proven to be effective in film sound, i.e. use of focus, contrast, silence to express emotions and subjectivity of the character/s [1].
It can be very important and useful to develop intelligent mixing systems in games like 'HDR audio' that ensure a mix doesn't become muddy. Perhaps though, games could also benefit from developing the use of narrative / event related mix moments that employ mix decisions based on emotional subjective reasons[2]...
There are many great films that have unique mix moments that can enhance the narrative, story, and dramatic effect - but I think part of the reason the mix moments are so effective is because they are unique within the film. Always a great example is the tiger scene in Apocalypse Now.
So maybe one way games can learn from this is to have some mix moments that are unique to particular points in the game so they can take advantage of the narrative, story, and character development. The mixes would still be interactive because they would be triggered by the player's actions and also feed off of game variables.
This could not only help to diversify the audio landscape, but also add another string to the bow of audio in games helping to enhance the overall experience.

So from this notion of working mix moments into games I have been putting together some examples.
I first identified mix techniques used in films then looked for scenarios in games where similar mix techniques could be applied. I then captured game footage with which to create a guideline mix. This involved spotting the footage and adding the different sound stems to re-create how it would sound in game. From this point I could then play with the mix and add automation to set out how I wanted it to sound in game.
I then tried to implement these mix moments in game using audio middleware tools.

Here are some of the guideline mixes I have made. I have included the version without the mix moment first for each example to show the difference.

This first example is a scenario from Doom3, it shows the player going into a room with enemies lurking in the darkness. The mix makes use of delay and reverb to characterise the enemy sounds through the character's POV, helping to emphasise emotions of fear/paranoia etc.

Mix features OFF...


Mix features ON...


----------------------------------------------
The next example is also a scenario from Doom3, the player has to drop down into the room and at some point a monster might make an appearance. The mix makes use of (relative)silence leading up to the event to provide a dramatic emphasis through the contrast created. Again filtering the sound through the character's subjective POV.

Mix features OFF...


Mix features ON...


An interesting issue raised by Simon Ashby, product director of Wwise at Audiokinetic, in Rob Bridgett's article, is that difference in styles of gameplay between a cautious player and an extremely bold player for example will produce different results in the soundtrack. So this needs to be taken into consideration when creating an interactive mix, especially if unique mix moments based around the narrative are being developed. There is a balance that would hopefully be struck, some mix moments could be tweaked so that they work for nearly all types of player. Perhaps it may be a fact though that the intended mix may not be realised in game 100% of the time, on the flip side however, this could work as an advantage because if the player plays the game a second or third time the difference in the mix may help give a varied experience. One thing not to forget of course is that the player always needs to hear the required information, i.e. voice instructions, and also that the mix never becomes a sonic mess - unless it's for a desired effect that is.

So here I have an example of the silence scenario again but with footage of faster gameplay around when the mix moment occurs. The previous example showed the player exploring the dark area that had been dropped into. This next example shows the player running straight through into the lit room.
Hopefully it demonstrates how this mix moment would work for different styles of play.

Mix features OFF...



Mix features ON...


----------------------------------------------
Some other links to interactive mixing and subjective game audio type stuff....
Recreating Reality
The Future Of Game Audio
Game Audio Pro
Subtlety and Silence
Ducking