Gamers logo

Video Game Audio Software, Hardware and an Exploration into its Effect on the Player

An essay on the evolution of gaming audio

By FoxPublished 4 years ago 14 min read
Like

Audio has played an important role in interactive entertainment for the majority of its existence. Unfortunately, early home consoles were completely silent as they lacked the processing power to handle both audio and visuals. The first game with audio, seen on american shores, was Taito’s “Gunfight” which used a microprocessor instead of hardwired circuits and a mono amp for gunshot noises. Taito later follows this innovation with the classic “Space Invaders”. By the early 70s, home computers were being released that plugged directly into the TV but still didn’t play sound. The late 70s saw the release of early mono audio machines, from which the tembre was improved until the early 90s, when stereo consoles began releasing. From here, There’s a more complex turn of events involving many releases of PC sound cards and iterations on similar consoles. At this point we see an interesting battle of software and hardware between the major manufacturers. This is where I will explore the specifics of audio solutions, both hardware and software. As well as this, I look forward to discussing the impact these solutions have on the audience. First I will consider the earliest uses of sound.

The earliest form of sound for video games was synthesized as the game was played by components on the main board of the hardware. Pong used the sync generator, a chip used to regulate timing in other hardware on the board. The creator, Allan Alcorn describes the sound design process as “...In less than an hour (I) poked around and found different tones that existed in the sync generator…” (Alcorn, 2008). The sounds for the ball hitting the wall and paddle are coincidentally an octave apart, making them feel coherent. (Caprani, 2014) This foreshadows a usage of the Computer’s existing hardware to generate simple tones. The IBM PC and Apple II both used a direct interface from the CPU to a simple speaker. (Murray, 2015) The disadvantage was that this used the CPU almost completely on early computers so it struggled to do anything else while generating sound. This included all movement on screen and any background processes. With some complex programming and timing, music could be added to games as the only waveform was a pulse wave which could be timed in a CPU cycle between other processes to create a tone while also running the rest of a program.

The same was not to be said for home consoles. The Atari VCS, which became known as the Atari 2600 later in its life-cycle, Used a Stella chip to output graphics and audio. The Stella or TIA (Television Interface Adapter) chip ran audio through three registers that manipulated code through basic maths. It could only add subtract, multiply and divide using 8-bit numbers. (Davie, 2003) You would program pitch in the second register by dividing a base frequency of 30KHz using 8 bits to create the numbers to divide by. This created something like a Pythagorean tuning for the console that meant it didn’t lend itself very well to western understandings of composition. Because of how out of tune this system was in comparison to equal temperament, it wasn’t used very much for music. (Weske, 2000) Following this implementation, other consoles used a similar system of maths to create tones but music wasn’t widespread in gaming until systems implemented separate sound chips that could be programmed to play any frequency on their oscillators.

The Nintendo Entertainment System (NES) used four separate oscillators and a PCM sample channel to create all of its music and sound effects. A common method of achieving music and sound effects was to momentarily replace the accompaniment line with sound effects. Most players wouldn’t notice it leaving in comparison to the lead line stopping. Super Mario Bros. 3 uses this technique throughout the game, sometimes replacing the lead line to force the players attention onto the sound effect. (D’Argenio, 2018) This new found tonality can be used to great effect in early cases of sound design in gaming. Mario games are a fantastic example because the noises are so simple. Many Mario sounds are two notes either an octave or a perfect fifth apart. The sound of picking up a coin is two notes an octave apart, ascending. This creates an effect of pulling up or a positivity like it’s mimicking a cheer or an exclamation. This effect would have been difficult to create on earlier consoles with limited pitch control. (Locke, 2017)

After the NES, Nintendo released the Super Nintendo Entertainment System (SNES) in 1990. This system was limited in it’s audio like the NES but had 8 tracks for music and sound effects. These tracks weren’t limited to single waveforms like the NES but instead used sampling. The audio system had 64Kb of RAM to store any and all of the games samples and music tracker files. This is the second difference between the systems, the use of a music tracker meant that the SNES could generate complex, evolving suites of music without knowledge of code needed from the composer. A music tracker file consists of a list of timed instructions for each track on what sample, note and effect to use. (Puschak 2018)

After the release of the Super Nintendo Entertainment System, which used a similar system as the NES, programmed music played in a set sequence, The Sony Playstation was released in 1994. The Playstation was an outlier in the industry because it relied entirely on CD-ROMs to store games, a much larger but slower medium than the SNES’s cartridges. (Oxford, 2018) The original Playstation used mp3 files stored on their discs as their soundtracks. Some games could even have their soundtracks and sound effects played through a CD player simply by inserting the game disc. For example, Castlevania: Symphony of the Night even includes some pieces that were not used in the final game on the disc, only playable through a CD player. It can be surmised that, because of the CD-ROM’s increased capacity, music tracks and sound effects were simply compressed to mp3 and signalled to play and loop on the system instead of each note being programmed. The Playstation did however contain an envelope filter and reverb in its sound chip. (Oxford 2018) What the system did with these effects is something I can only guess as finding sources that confidently states their usage has been close to impossible. I would speculate these effects would be used on sound effects to change their sounds in different environments. For example, a combination of a longer tail and a different reverb could create indoor and outdoor versions of a sound in real time.

After these consoles, the technology for creating sound largely stayed the same until a shift from using hardware to create sound to software. This is largely because of Microsoft’s Xbox. The Xbox’s first prototype was made of laptop components and was designed to run entirely on DirectX, initially slated to be released as the “DirectX Box.” (Marshall 2019) DirectX is an API or Application Programming Interface. An API is a piece of software that sits between, in this example, the game and the operating system of the hardware in question. DirectX is actually a collection of smaller pieces of software bundled together like Direct3D, DirectDraw and DirectMusic. The important thing about this standardised API is that it enables bespoke software to be coded for it and it’s guaranteed to work on systems that use it. Meaning, a development company can create their own game “engine” that they can do with as they please. Examples of different game engines are Source, an engine created by Valve and CryEngine, created by Crytek. These two engines are compatible with DirectX but are completely different. Source has a distinct focus on physics simulations whereas CryEngine is notorious for being focused on extreme graphics fidelity at the cost of performance. So much so that the internet meme of “But can it run Crysis?” still prevails today even though Crysis was released in 2007. This large disparity in engine design leads to breadth and depth never before seen in games. The same company can create completely different games relatively easily just by switching engines or designing their own.

All these differences in engine and design lead to very different audio solutions, from basic solutions to play music to dynamic score and sound effects seen in Alien: Isolation. This game uses a dynamic sound environment based entirely on how close the Xenomorph is to the player, lowering environmental sounds and increasing music volume when it’s near. Simplistic on paper but difficult to implement. The development team also tried adding random sounds of the alien into the environment as the player moved but this didn’t feel correct and instead, devised a system to have the alien still move around even when not on screen, still generating sound. (McGregor 2015) This technology brings us up to present day and the reliance on software to create convincing environments for the player.

To assist me with this section of the essay, I created a survey about the aural experience of interactive media and what people thought of it. In this survey I asked what platforms the participants frequently used and what genres of games they played for context. I then asked them to rank the importance of sound design from 1 to 10 and the same for soundtrack. I ended the survey with 15 participants but I didn’t get the breadth of participants that I would have liked. I received zero responses from people who play VR games regularly. However, I did capture what I believe to be an accurate cross section of audio equipment. My results show that 86% of the participants have used headphones for gaming. 40% have used TV speakers, 46% standalone stereo speakers and 20% small surround setups. I believe these results to be relatively close to my own anecdotal evidence from talking to friends and people online about their experiences. Something I found interesting about the results is that the ratings for sound design are slightly higher than soundtrack but, when asked for comment about audio in games, none of the participants mentioned sound design. I would speculate that this is due to a lack of vocabulary or general discussion around sound design in gaming circles. What can be said from these results is that the general consensus on the importance of audio is high. With this knowledge, what is being done to support this appreciation for audio? (Jones, 2019)

Nvidia, over the last few years, has been supporting VR development with their VRWorks program.This includes some technologies that make VR development easier. One of these pieces of technology is simply called VRWorks Audio. This technology “path-traces” sound to create a convincing binaural audio experience for the user. This technology was created specifically for VRWorks but applies to any game that needs realistic sound. (VRWorks - Audio) This kind of technology existed in the 90s and was developed by a company called Aureal. The technology was similar in result but the execution may have been different. Very little exists in way of explanation for this technology other than the companies legal battle with Creative in 1999.

These advancements have led to an almost limitless way of forming expression through audio in gaming. The fact that you can import any sound you want into an engine without worry of file size is an incredible achievement in technology. A sound designer today can take their time creating outside the limitations of what hardware can generate in real time to create incredibly affecting sounds without taxing the processor at all. On top of this, effects can be applied over the top of these sounds in real time to convince the player that it really is in the space they think it is. A sound designer can create any number of bespoke sounds to add to a production, the only limitations now are time and budget. This means a developer can concentrate on making the player feel however they want. For Games like Alien: Isolation, this means discomfort and fear. Achieved through sound design and music being programmed to play at the right times. (McGregor, 2015) For a game like Wipeout 2097, a distinctly 90s vision of future racing, this means the right soundtrack. Wipeout used tracks from the likes of The Prodigy, Orbital and The Chemical Brothers to pull the player into the speed and exhilaration of races combined with the zipping and whooshing of the hovering vehicles reminiscent of jet engines. Early Games like Super Mario Bros. Relied on specific scales like major pentatonics and pleasing intervals like perfect fifths and octaves but modern games place you into an entire world of sound and visual style. (The Psychology of Sound in Video Games, Amplifon)

These enhancements follow a trend from the original sound design of the earliest home console games, an increase in channels. In many respects, not just mono to stereo, then surround but also the number of sounds playable at once. The future of audio appears to be object based systems like Dolby Atmos, which use a system of rendering objects in 3D space, then bouncing it down to output to however many speakers the user has and in whatever location they may be in the room. (Roberts, 2019) Luckily for game developers, games are already in a true 3D space so supporting Atmos is a change in workflow, not a complete re-design of engines and systems already in place. Dice’s Battlefront and Battlefield 1 already support Atmos on PC through their own engine, Frostbite. Ben Minto, Senior audio director/sound designer at Dice believes that all studios will soon be working in Atmos. (Roberts 2019) As previously mentioned, VR is already receiving its own audio solutions from companies like Nvidia and AMD that, when coupled with Atmos, object based positional audio, will be the next step in creating the most realistic audio environment possible.

The path of audio in the world of interactive media appears to be headed to one of realism and unmatched clarity. The idea that a simple API can be added to an existing engine that creates accurate sound regardless of the environment it’s placed in, must be an exciting one for developers. This technology could see even small indie games with accurate positional audio. The sound design itself is still up to them to create themselves however, which can lead a lot to be desired in smaller productions. The advancements in technology and the size of the gaming market today indicate a continuing growth of creativity and exploration of the medium.

Bibliography

Alcorn, (2008) Al Alcorn Interview, interviewed by Cam Shea, IGN, 10th March 2008 Available at: https://uk.ign.com/articles/2008/03/11/al-alcorn-interview

Aureal vs Creative (1999)

https://web.archive.org/web/20071010054918/http://alive.singnet.com.sg/features/aureal-creative.htm

Andersen. A, (2014) 10 Inventions that Changed the History of Game Sound, A Sound Effect, 26th August 2014, Available at: https://www.asoundeffect.com/10-inventions-that-changed-the-history-of-game-sound/

Craddock, R (2018) How Music Was Made on the SNES, Nintendo Life, 16th August 2018. Available at: http://www.nintendolife.com/news/2018/08/video_how_music_was_made_on_the_snes

Caprani, (2014) The PONG Game, Computer Science Department, Aarhus University September 27th 2014 available at: http://cs.au.dk/~dsound/DigitalAudio.dir/Greenfoot/Pong.dir/Pong.html

Davie. A, (2003) 2600 Programing for Newbies, Session 4: The TIA, Atari Age, 22nd May 2003, Available at: http://atariage.com/forums/topic/27189-session-4-the-tia

D’Argenio. A (2018) Gaming Literacy: An Introduction to NES Sound and 8-Bit Sound Illusions, Gamecrate, 15th January 2018, Available at: https://www.gamecrate.com/gaming-literacy-introduction-nes-sound-and-8-bit-sound-illusions/17881

Locke. C, (2017) The Beeping, Gargling History of Gaming’s Most Iconic Sounds, Wired, 6th December 2017, Available at: https://www.wired.com/story/psychology-iconic-video-game-sounds/

Lakawicz. P&S (2017), Episode 20: The SNES. Retro Game Audio, “1 Year ago” Available at: https://soundcloud.com/retrogameaudio/the-snes

Mcdonald, G (2005), A History of Video Game Music, Gamespot, 28th March 2005, Available at: https://www.gamespot.com/articles/a-history-of-video-game-music/1100-6092391/

McGregor. J (2015), Seeing with your ears - the audio of Alien: Isolation, PC Gamer, 13 February 2015, Available at: https://www.pcgamer.com/uk/the-audio-of-alien-isolation/

Marshall. R (2019) The History of the Xbox, Digital Trends, 30th December 2019, Available at: https://www.digitaltrends.com/gaming/the-history-of-the-xbox/

Murray, (2015) How Oldschool Sound/Music worked, The 8-Bit Guy, 15th October 2015, Available at: https://www.youtube.com/watch?v=q_3d1x2VPxk

Oxford. D (2018) Why Cartridges instead of CDs for the Nintendo 64?, Old School Gamer Magazine, 1st February 2018, Available at: https://www.oldschoolgamermagazine.com/why-cartridges-instead-of-cds-in-the-nintendo-64/

Puschak. E (2018) How Music Was Made On Super Nintendo, Nerdwriter1, 15th August 2018, Available at: https://www.youtube.com/watch?v=jvIzIAgRWV0

Roberts. B (2019) Dolby Atmos: What is it? How can you get it? What speakers do you need?, What Hi-Fi?, 15th January 2019, Available at: https://www.whathifi.com/advice/dolby-atmos-what-it-how-can-you-get-it

Sebastian. L (2016), What is an API? (Application Program Interface), Techquickie, 4th October 2016, Available at: https://www.youtube.com/watch?v=6STSHbdXQWI

Tyson. J (2000), How Playstation Works, Howstuffworks, 17th October 2000, Available at: https://electronics.howstuffworks.com/playstation2.htm

The Psychology of Sound in Videogames, Amplifon, Available at:

http://www.amplifon.ie/resources/playing-with-your-mind/

VRWorks - Audio, Nvidia GameWorks,

https://developer.nvidia.com/vrworks/vrworks-audio

Wiki Page of PS1 Games that had music playable through a CD player with the game disc. Available at: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/List_of_PlayStation_games_with_CD_audio_tracks.html

Weske. J, (2000) Digital Sound and Music in Computer Games, December 2000, Available at: http://3daudio.info/gamesound/history.html

list
Like

About the Creator

Fox

I'm Fox,

I write short stories, usually of a science fiction nature. I also lend my expertise in tech and music production to my non-fiction articles.

Reader insights

Be the first to share your insights about this piece.

How does it work?

Add your insights

Comments

There are no comments for this story

Be the first to respond and start the conversation.

Sign in to comment

    Find us on social media

    Miscellaneous links

    • Explore
    • Contact
    • Privacy Policy
    • Terms of Use
    • Support

    © 2024 Creatd, Inc. All Rights Reserved.