Raytraced Audio - Voices

This article describes how sounds are spatialised using the position and velocity of in-game entities like players, projectiles and grenades.

Spatialisation with OpenAL

In OpenAL, listeners and sound sources have position, velocity and orientation attributes, which OpenAL takes into account when mixing the final audio output.

This allows players to hear sounds to their left and right. Punch Deck's track Music To Wear Fingerless Gloves To is used to demonstrate this:

While this works well, there are many situations in Sector's Edge where sounds are playing above, below, infront and behind the player. It's important that the game supports HRTF so that players are able to instinctively tell where sounds are coming from in the full 3D space around them.

HRTF applies subtle filters to sounds based on their direction from the player. These filters mimic how human ears perceive sound from each direction in real life, which helps players hear sounds in the full 3D space around them while still using stereo headphones.

As a test, the two videos below contain a sound source rotating in a circle around the listener, from top to front to bottom to behind. Only one of these videos has HRTF enabled:

Can you tell which one has HRTF enabled?

Sources

Starting with the basics, playing a sound in OpenAL works like this:

• Generate a source (similar to generating a texture/buffer in OpenGL)
• Associate the source with a buffer
• Play the source

By default this will play a sound relative to the listener, meaning the sound follows the player as they move around. This is useful for music and UI sounds.

To instead position a source in 3D space, i.e. 'spatialise' a sound, there are a few attributes we can set on the source:

• position
• velocity - used to simulate the Doppler Effect
• rolloff - affects falloff over distance, e.g. explosions carry over longer distances

Creating a 3D-spatialised source looks like this:

// Create a source
var sourceID = AL.GenSource();

// Store it in a Source object (wrapper class for calling AL functions)
var source = new Source(sourceID);

// Tell the source what sound data to play
source.SetBuffer(bufferID);

// Tell OpenAL we want this sound to be spatialised
source.SetSpatialise(true);
source.SetRelative(false);

// Set falloff variables
source.SetReferenceDistance(8);
source.SetRolloff(1);

// Spatialise
source.SetPosition(position);
source.SetVelocity(velocity);

// Start emitting sound
source.Play();

The above works great for sounds like explosions, footsteps and gunfire, which don't change position once they are created. However for entities that continually emit sound - for example a grenade flying through the air, which continuously beeps until it explodes - the sound should follow the entity:

Voices

The Voice class is used to connect an OpenAL Source with an in-game entity, such as a grenade or player. It contains a Source, IPosition and IVelocity instance:

public class Voice
{
    public Source source;
    public IPosition position;
    public IVelocity velocity;
}

This class is called Voice because it 'produces' sound in 3D space.

These two IPosition and IVelocity interfaces allow a source to be connected to any game entity, for example a grenade:

public class Grenade : IPosition, IVelocity
{
    public Vector3 position;
    public Vector3 velocity;
    
    Vector3 IPosition.GetPosition() => position;
    Vector3 IVelocity.GetVelocity() => velocity;
}

Creating a voice is as follows:

public class Game
{
    void ThrowAGrenade()
    {
        // Create a grenade
        var grenade = player.CreateGrenade();   
        
        
        // Try to create an OpenAL Source with the GrenadeBeep buffer
        if (!AM.TryGenSource(SoundType.GrenadeBeep, out var source))
            return;
            
            
        // Create a Voice and store it in the list of active voices
        AM.Voices.Add(new Voice(source, grenade));
    }
}

Updating Voices

Each frame, the Audio Manager tells OpenAL the position and velocity of every voice, as well as the position, velocity and orientation of the player. Orientation is required so that OpenAL knows which way the player is looking.

public static class AM
{
    public List<Voice> Voices = new();
    
    public void Update(Player player)
    {
        // Update the player's position, velocity and orientation
        AL.Listenerfv(AL10.AL_POSITION, player.position);
        AL.Listenerfv(AL10.AL_VELOCITY, player.velocity);
        AL.Listenerfv(AL10.AL_ORIENTATION, player.orientation);
        

        // Update the position of each voice
        for (int i = Voices.Count - 1; i >= 0; i--)
        {
            var voice = Voices[i];
            voice.Update();
            
            // When the Source finishes playing, remove the Voice
            if (voice.source.IsDisposed())
                Voices.RemoveAt(i);
        }
    }
}

The Update function on a voice is relatively simple and handles disposing the source when it finishes playing.

public class Voice
{    
    public Source source;
    public IPosition position;
    public IVelocity velocity;
    
    public void Update()
    {
        // When the source finishes playing, dispose it
        if (source.Finished())
        {
            source.Dispose();
            return;
        }
        
        // Tell OpenAL the position and velocity of this Source
        source.SetPosition(position.GetPosition());
        source.SetVelocity(velocity.GetVelocity());
    }
}

Sources

The Source class is especially useful here for two reasons:

• it's a thin wrapper on top of the underlying OpenAL alSource* functions
• it manages the transformation of the game's Y-up world space coordinate system to OpenAL's Z-up coordinate system:

public class Source
{
    public uint ID;
    
    public bool Finished()
    {
        var state = AL.GetSourcei(ID, AL10.AL_SOURCE_STATE);        
        return state == AL10.AL_STOPPED;
    }
    
    public void SetPosition(Vector3F v)
    {
        float[] temp = new float[3];
        
        // Transform the game's Y-up coordinates into
        // OpenAL's Z-up coordinates
        temp[0] = v.X;
        temp[1] = v.Y;
        temp[2] = -v.Z;

        AL.Sourcefv(ID, AL10.AL_POSITION, temp);
    }
    
    public bool Dispose()
    {
        AL.DeleteSource(ID);
        ID = 0;
    }
}

Doppler Effect

OpenAL adjusts the pitch and speed of each source based on its velocity, to simulate the Doppler Effect. This helps the player tell if a projectile is moving towards or away from them.

Here's what it sounds like with a super fast velocity:

Next Steps

Now that the sources and voices have been covered, the next articles will cover:

• using the results of raytracing to apply filters to voices
• directional ambience - calculating rolling averages of weighted vectors
• multithreading - ensuring all heavy processing occurs on background threads
• the raycasting function

Playing Sounds

The TryCreateSource(...) function on a SoundStorage object is used to create an OpenAL sound source, which is then sent to the raytracing thread for its initial raytrace. The result of this raytrace is used to determine the strength of the low-pass filter to apply to the sound source, and finally the sound is played using alPlaySource(...).

TryCreateSource(...) can fail and return false if the variation isn't loaded yet or if the maximum number of active OpenAL sources is exceeded. This failure can happen when the game is initially starting up or when there are many players firing weapons at the same time. In the latter scenario it's likely a few missing sounds won't be noticed and it's better to silently ( ha ) fail than check for an older active sound source that can be disposed, potentially causing a lag spike.

public class SoundStorage
{
    // Helper function for generating a random number
    // between 0 and variations.Count
    int RandomVariationIndex => rand.Index(variations);


    public bool TryCreateSource(int index, out Source source)
    {
        // If a variation wasn't explicitly selected, play a random variation
        if (index == -1)
            index = RandomVariationIndex;

        return variations[index].TryCreateSource(out source);
    }
}

That's all! The next in the Raytraced Audio series explains how these sound sources are connected with in-game objects like players and projectiles.