Interactive Mixing Project

Over the summer I have been working on my final masters project which is an investigation into interactive real-time mixing in games. The two main issues I have been looking at are;
-the potential for using film style mix techniques in games,
-what extent can currently available middleware tools implement interactive mixing.

There has already been a certain degree of interactive mixing done in games, even back on PS2 games such as Scarface and MGS3, and especially more recently with the sophisticated mixing systems of 'HDR audio' in DICE's 'Frostbite' engine and Radical's 'AudioBuilder'.
Yet it seems there is still a lot for games to achieve in terms of fully utilising the mix ideas and techniques that have been proven to be effective in film sound, i.e. use of focus, contrast, silence to express emotions and subjectivity of the character/s [1].
It can be very important and useful to develop intelligent mixing systems in games like 'HDR audio' that ensure a mix doesn't become muddy. Perhaps though, games could also benefit from developing the use of narrative / event related mix moments that employ mix decisions based on emotional subjective reasons[2]...
There are many great films that have unique mix moments that can enhance the narrative, story, and dramatic effect - but I think part of the reason the mix moments are so effective is because they are unique within the film. Always a great example is the tiger scene in Apocalypse Now.
So maybe one way games can learn from this is to have some mix moments that are unique to particular points in the game so they can take advantage of the narrative, story, and character development. The mixes would still be interactive because they would be triggered by the player's actions and also feed off of game variables.
This could not only help to diversify the audio landscape, but also add another string to the bow of audio in games helping to enhance the overall experience.

So from this notion of working mix moments into games I have been putting together some examples.
I first identified mix techniques used in films then looked for scenarios in games where similar mix techniques could be applied. I then captured game footage with which to create a guideline mix. This involved spotting the footage and adding the different sound stems to re-create how it would sound in game. From this point I could then play with the mix and add automation to set out how I wanted it to sound in game.
I then tried to implement these mix moments in game using audio middleware tools.

Here are some of the guideline mixes I have made. I have included the version without the mix moment first for each example to show the difference.

This first example is a scenario from Doom3, it shows the player going into a room with enemies lurking in the darkness. The mix makes use of delay and reverb to characterise the enemy sounds through the character's POV, helping to emphasise emotions of fear/paranoia etc.

Mix features OFF...


Mix features ON...


----------------------------------------------
The next example is also a scenario from Doom3, the player has to drop down into the room and at some point a monster might make an appearance. The mix makes use of (relative)silence leading up to the event to provide a dramatic emphasis through the contrast created. Again filtering the sound through the character's subjective POV.

Mix features OFF...


Mix features ON...


An interesting issue raised by Simon Ashby, product director of Wwise at Audiokinetic, in Rob Bridgett's article, is that difference in styles of gameplay between a cautious player and an extremely bold player for example will produce different results in the soundtrack. So this needs to be taken into consideration when creating an interactive mix, especially if unique mix moments based around the narrative are being developed. There is a balance that would hopefully be struck, some mix moments could be tweaked so that they work for nearly all types of player. Perhaps it may be a fact though that the intended mix may not be realised in game 100% of the time, on the flip side however, this could work as an advantage because if the player plays the game a second or third time the difference in the mix may help give a varied experience. One thing not to forget of course is that the player always needs to hear the required information, i.e. voice instructions, and also that the mix never becomes a sonic mess - unless it's for a desired effect that is.

So here I have an example of the silence scenario again but with footage of faster gameplay around when the mix moment occurs. The previous example showed the player exploring the dark area that had been dropped into. This next example shows the player running straight through into the lit room.
Hopefully it demonstrates how this mix moment would work for different styles of play.

Mix features OFF...



Mix features ON...


----------------------------------------------
Some other links to interactive mixing and subjective game audio type stuff....
Recreating Reality
The Future Of Game Audio
Game Audio Pro
Subtlety and Silence
Ducking

Dynamic ambience in 'Prototype'

I have yet to play the game, but from what I've seen and read I can't wait to play it!

The Gamautra feature explaining how the sound team at Radical went about creating and implementing the ambience in Prototype describes some useful interactive real-time mixing techniques and offers insight into the workings of their proprietary audio tool AudioBuilder.

One section that particularly interested me is how they use 18 channels of ambience simultaneously streaming from disc and dynamically mix the levels of individual tracks or groups depending on game variables - more specifically the density of pedestrians, infected hordes, and traffic building up around the player.
The system also uses these density values along with the position of the sounds relative to the player to dynamically mix the ambience among the quadrophonic channel set-up to help provide a sense of orientation.

This interactive mix of the individual ambience elements seems to be entirely reactionary to real-time variables. So, though I'm sure it provides an immersive ambience that feels alive, I can't help but think it would be nice to override this for events or moments in the game. For example, it could be effective to be able to duck all ambience apart from the infected hordes at certain points in the game to achieve a dramatic subjective type moment.

The overall ambience is bussed to the main mixer system and does allow for control of the ambience bus within the overall mix to apply filtering for cinematics and special game modes. However it would have been pretty cool to have the control to exclude a certain element of the ambience that is filtered, giving the ability to inject that extra bit of emotional delivery when neccessary.
Perhaps the question remains, would this be apporopriate to the gameplay or the character and story? I'm not exactly sure as I haven't played it but it would be interesting to see if that extra level of flexibilty in the mix could intensify the cinematic qualities in the gameplay and possibly even help provide more variation in the soundscape...

All things considered though it's great to see a complex and intelligent mix being implemented and also described so openly to the world!