Let the Sonnenschein In

I first read David Sonnenschein's book "Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema" later in my sound mixing career. I wish I had read it earlier on. Not only did it inspire me to think about sound in a new way, it inspired me to think about filmmaking and sound's relationship to story telling in a new way.

I've always encouraged my sons (aged 8 and 12) to listen to sound and music in the world around them with care and fidelity, and so when I saw David's videogame "3 Deaf Mice" on Kickstarter, giving my support was a no-brainer.

- Peter Schneider

David Sonnenschein is a sound designer, musician, filmmaker, and neuroscientist, as well as the author of the seminal Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema. His latest project is 3 Deaf Mice, a video game designed to teach players new listening and perception skills in a fun, intuitive way.

Gotham Sound and Communications: Tell us about your background. You’ve got a very interesting, very varied background. How did you get here?

David Sonnenschein: I started really from two sources. My parents represented very specifically science and art. My father was a professor of physiology at UCLA medical school and my mother was very accomplished in many different art forms. I was stimulated by both of them to really excel in school and in my creative activities. I began playing in symphony orchestras when I was eight years old, got very classically trained and really helped train my ear to listen to the world in very, very specific ways. It was a very natural thing for me to do.

I was stuck in a way because of the traditions of classical music to play what other people thought was the right way. In fact, I had to break out of that listening into a different form through jazz, which was about improvisation and being able to sound read instead of sight read, and I used my ears in a whole new way. So that was a very important evolution in my ability as a creator of sound, to be able to express without somebody telling me what was right or wrong.

GSC: Where does the neuroscience come in?

DS: Well, being interested in the mind and body from a young age, influenced by my father’s studies in physiology and my interest also in philosophy and the experience of our dreams, it led me to study neuroscience and ultimately work in a sleep laboratory when I studied brain waves and different psychological states and the interface between our mind and our body. With this knowledge and skill set, I also brought that into my sound and filmmaking role because it’s so important to utilize our natural makeup as human beings. I’m talking about our eyes and our ears and our brain and ultimately how we perceive the world.

Now the entrance into the film world really opens up into an area of communication and story and so what I’ve done is taken these three threads of music, neuroscience and storytelling in film and woven them together into a place of inquiry and art of sound design.

GSC: So the next logical stop was taking it to a game where it can be interactive.

DS: Absolutely. I have been interactive in my music as an improvisational jazz musician and love the idea of interacting with audience as well. I perform with dancers and go out on the dance floor with them as I’m playing music.

I had felt over the years rather constrained in film sound because we work in isolation, we put it out there, the audience experiences it, and that’s it. So entering into the gamer world is such a delight for me because what I’m experiencing is the interactivity - not just of the player with the game, but in the actual creation of the game - I am interacting with the community to create it. Specifically, we’re running a Kickstarter project right now that is about crowdfunding - and people are helping fund the actual game - but more than that they’re actually helping create the game with me with feedback as we are working on it. Games, as opposed to films, work with a tremendous amount of testing and we will be testing various aspects of the game to see how it applies to different demographics. For example, children without any kind of training compared to professionals who work in film or games or music have a very different profile, and this game is going to reach out to all of these communities.

We are actually creating a game right now with the Kickstarter project where people will watch the Kickstarter video and get clues, listen to the sounds and enter a contest to win prizes - just from the video. This is a very different way of collaborating with the audience and the community than just putting a film out there, then seeing how they like it or not.

I’d say that it’s also a very different aspect of our world when we learn a skill to be rewarded The gameplay is asking the player to get really involved in their world of sound and in this case they’re going to be developing listening skills that will enhance their ability to communicate. I’d say a general purpose of this game is to develop mindfulness, which is a broader awareness of the environment and how we can function in a balanced and effective way with the people around us.

GSC: One of your recent credits was as audio consultant for Mass Effect 3. How did you get involved? How did your principles of sound design play out in putting that together?

DS: I was contacted by one of the audio leads at Bioware, which produced the Mass Effect franchise. He has been using my book, Sound Design, in his own work for many years and felt that I could contribute to their team, and so I spent several days with them.

The exchanges that we had were fundamentally about developing more engagement by the player through audio and this engagement had to do with looking at the microlevel of what is happening in any one given scene - how does the player interact with the level - to more of an overview - how do that level and parts of the storyworld contrast from one to another? When the player would enter into a certain world, what was the essence of that experience? Was it seeking a specific goal? Was it in a lost or fragmented and scary world? Was it to communicate something specifically so you could get to the next level and make the proper choices so that you could experience a specific success in managing to accomplish the mission?

The audio is often neglected in film until the end of the production, while in games the audio team is working from the very, very beginning with the other team members on the story and the graphics and the interface, so they are contributing creatively at a very fundamental game experience level.

GSC: Yeah, that sounds very handy.

DS: It was an exciting project because I knew it was going to be seen and experienced by many, many people being an already successful franchise with the first two parts and I was helping them develop the third installation.

GSC: Can you tell us a little bit about how your game will work?

DS: This game is combining storytelling and character with specific goals of creating a song. This song is 3 Deaf MIce’s next big hit, but they need the player to help them because they have gone somewhat deaf - hence the name - from listening to too loud music.

GSC: And you get a nice audio safety lesson in there too.

DS: Oh, yes. The other underlying message in there is: please protect your ears and listen to the quality of your music. You don’t have to bump it up to damaging volume levels.

In this case, we’re going to be listening in ways that most people are not tuned into their world. It’s developing your brain muscles to start to recognize how sound really affects us in the world, so we’re going to have specific sonic treasure hunts where you have to find sounds within other sounds and where they are in the scene. Once they’re identified, you’ll start to become aware of their physical nature, which I call shape and that refers to the waveform of the sound and what it looks like when you visualize it. For example, we’ll develop the sense of pitch and timbre, which is the frequency spectrum of a sound that distinguishes, say, a flute from a violin, and we’ll look at attack and reverb.

All of these are going to be mostly invisible experiences that we have in our everyday life, but we’re going to bring them into the visible world through the gameplay and challenging the player to develop more sensitive listening skills around that. Ultimately the player will begin creating new forms and structures from the sounds that they find that will lead them towards a musical composition which becomes the song. When the song is completed, the players are going to have the fun opportunity to remix the song by playing around with all the different sounds they’ve been working with in different orders and different structures.

GSC: What do you hope people will get out of the game?

DS: Those who have never played or composed or studied music are going to begin to feel that they too are empowered to create with the sounds that they have in their normal environment. They don’t have to practice an instrument for years and years to become really musical beings The people who have been working professionally with sound will be able to experience it in a new way that is extremely playful, malleable and accessible.

In all these cases, we’re developing community around being able to share each others musical inventions and ultimately open up to sharing their own sounds that can be incorporated into the musical structures. I have a series of games that will build a much larger platform of interactivity and real time audio work once the first game is done.

GSC: Anything else you’d like to share about the game or sound in general?

DS: I have a natural tendency to work with sound and image that comes from a process called synesthesia, which allows me to see sound and hear images in a very spontaneous way. I only discovered as I was growing up that not everybody does this. So my intention with my film work, sound design and now specifically with my game is to bring this experience to many people so they too can begin to hear and see both sound and image. And I hope by doing so, they can become more creative and communicative in their lives.