soledad penadés

My first "ambitious" Web Audio project was porting my VST instrument Sorollet (which was written in C++) to JavaScript, as sorollet.js.

At a very simplified level, software using a VST instrument requests periodically a "slice" of audio data¹ from the instrument, and the instrument returns such a thing by writing the results of its computations into a buffer.

Thus when I wrote sorollet.js, I used Web Audio's venerable ScriptProcessor to achieve the same functionality: the script processor node onaudioprocess function would be called periodically by the audio engine so that it would return a slice of audio by writing to a buffer.

From one of the examples:

jsAudioNode.onaudioprocess = function(event) {
    var buffer = event.outputBuffer,
        outputBufferLeft = buffer.getChannelData(0),
        outputBufferRight = buffer.getChannelData(1),
        numSamples = outputBufferLeft.length,
        voiceBuffer = voice.getBuffer(numSamples);

    for(var i = 0; i < numSamples; i++) {
        outputBufferLeft[i] = voiceBuffer[i];
        outputBufferRight[i] = voiceBuffer[i];
    }

    scopeGraph.update( voiceBuffer );

    if( debug ) {
        envLastValue.innerHTML = 'Volume ' + voice.ampADSR.lastValue + '<br />' + 'Pitch ' + voice.pitchADSR.lastValue;
    }
};

You might notice that in addition to obtaining audio data from a Voice instance, it is also updating the scope graph and checking if we're in debug mode, to update a layer with debug data if that's the case. Leaving aside whether this is or not a "best practice", we could just say that it is a consequence of the fact that this code runs on the JavaScript main thread.

In that particular example the synthesiser code is really not that processor intensive. If you look at the function, you see it is roughly zeroing a buffer, then calling two functions, combining them and returning the output in a buffer. Even then I added some optimisations, such as skipping some calculations if the volume was zero, etc.

So it used to run just fine and you could have multiple voices in parallel happily chugging out their additive synthesis output... all as long as you didn't expect to interact much with the program. Because when you did that, you could hear the sound become a bit cracked and glitchy, and the user interface become sluggish. Why? Because it was all running on the main thread: sound generation, user interface input, user interface updating. So nothing really worked great; it was all a little bit wrong.

To address this type of issues, the Web Audio gods² came up with AudioWorklet nodes. They fix the issue of the audio becoming glitchy by running in what they call the "rendering thread", i.e. the thread where the audio is computed as contrasted to the "control thread" which is where the nodes are created and connected (and which most people call "the main JS thread"). And thus this decouples sound processing from everything else.

But I hadn't had the chance to use AudioWorklets yet, and I haven't found many examples that showed how to do some things I was interested in. So this is the post for people, like me, who used ScriptProcessor before and want to learn how to use AudioWorklets.

For demonstration purposes, I'll start by generating monophonic noise, although I'll show how to do more complicated things later, as that was one of the aspects I found no examples for.

Defining and instantiating AudioWorklets is quite different

With ScriptProcessor, you could simply do something like this:

let ac = new AudioContext;
let sp = ac.createScriptProcessor(2048 /* buffer size */);

sp.onaudioprocess = function(event) {
    let buffer = event.outputBuffer;
    let outputBufferLeft = buffer.getChannelData(0);
    let outputBufferRight = buffer.getChannelData(1);
    let numSamples = outputBufferLeft.length;

    for(let i = 0; i < numSamples; i++) {
        let v = 2.0 * Math.random() - 1;
        outputBufferLeft[i] = v;
        outputBufferRight[i] = v;
    }
};

sp.connect(ac.destination);

And in this case you would have everything in the same file... and even scope³!

But in the new world of AudioWorklets, processors code lives in separate files (and scope).

So to create an AudioWorklet equivalent to the ScriptProcessor above, we start by creating a separate file.

Assuming it's called noise-generator.js, its contents would be this:

class NoiseGenerator extends AudioWorkletProcessor {
    process(inputs, outputs, parameters) {
        const output = outputs[0];
        const numSamples = output[0].length;

        for (let i = 0; i < numSamples; i++) {
            let v = 2.0 * Math.random() - 1;
            output.forEach((channel) => {
                channel[i] = v;
            });
        }

        return true;
    }
}

registerProcessor("noise-generator", NoiseGenerator);

This is similar to how Web Workers code is first loaded from a separate file and then ran in a different scope, with access to special methods that aren't available to code running in the main thread, such as importScripts in the case of Web Workers.

In this case, the AudioWorkletProcessor class is not even available in the global scope (try console.log in your devtools - you won't find it there), but it is available when in the context of an AudioWorkletGlobalScope.

Also, registering the processor before we can use it is also very similar to how you have to register custom web components before using them.

And to use it, in our "main" file:

let audioContext = new AudioContext();
await audioContext.audioWorklet.addModule('noise-generator.js');

let noiseNode = new AudioWorkletNode(audioContext, 'noise-generator');
noiseNode.connect(audioContext.destination);

You can see a demo here.

Some notable differences compared to the old style of doing things:

• We have decoupled the noise generation code from everything else: the processor code now only focuses on audio data computation, whereas the main file sets up the audio context, but often does much more (in particular, it's not shown for brevity, but we also set up a click listener so that the audio context is started when the user interacts with the page, as otherwise it will be suspended to prevent audio playing automatically).
• Loading the file and getting it registered is an asynchronous operation. You really need to wait on it to finish or risk not being able to instantiate the node as the audio context doesn't have the type in its registry.
• We do not define the buffer size when creating the processor. It is up to the implementation to decide, but it seems to be 128.
• We also need to signify if a node is active, by making sure we finish the process method with return true, for as long as the worklet node is meant to be emitting audio. This is to help Web Audio collect nodes which are "done", or not active any more. Effectively if you do not return anything it is considered as if you have returned false and thus the node might be considered as "recyclable" so to speak⁴.

Things that are not entirely clear to me yet:

• What does outputs exactly mean here? I think it means each node that the Worklet instance is connected to, although the Web Audio spec is not very forthcoming about this in the AudioWorklet process docs. If I understood correctly, it means that you can't assume a worklet is connected to just one output. In this case all the outputs get the same signal by computing each value once and copying the result to every output and channel, but it could be possible to return different values if we wanted to.
• Can the length of the buffers change from call to call? This was possible in VSTi if I recall correctly. Not a big dealbreaker for me if so, but I am curious to know.
• How do you load other code? Is there anything like importScripts available? (because importScripts doesn't seem to be).

And now we'll enter the realm of "things I have not seen in any AudioWorklet examples" but I have figured out!

How do you pass parameters?

Since the processor code is instantiated in a separate scope, it has no access to any variables outside. This is fine if you only expect to generate "white noise" as in the examples I've seen, but what if you want to communicate with the processor instance from "outside"? For example, to configure it? Musical equipment generally offers some way of affecting the way it generates or processes sound.

There seem to be two options available⁵:

AudioParam parameters

The first option is via the parameters argument to the process function itself. This is based on the concept of AudioParams, and will contain a number of Float32Arrays which contain precomputed values of parameters. How many you get depends on whether and how you defined the parameters upfront when declaring the class. This is not exactly the type of data I want to send to the processor in my example, and besides, I have not used this functionality yet, so I am going to skip talking more about this option for now.

port

The second option is via the port property.

Each AudioWorklet instance has a port property, which is of type MessagePort.

A port is like a "pipe" and you can send things through it to the other side of the pipe.

So we can send "atomic" messages to the worklet, meaning smallish pieces of data. For example, imagine that you want to activate "debug" mode for a worklet, so that then it renders a particular type of signal when debug is ON. It would be nice if you could just send a simple message from the main thread to the processor if the user changes something in the UI, and the processor picks it up and does something with it.

We would send a message by running this in the main thread:

noiseNode.port.postMessage({ debug: true });

But we also should add message listening features to the noise processor by defining its constructor, which we omitted initially:

constructor(...args) {
    super(...args);

    this.port.onmessage = (e) => {
        let { debug } = e.data;
        this._debug = !!debug;
    };
}

Note how the event e we receive contains a data property with the value we sent from the main thread. From there we can pull out the debug value we're interested in, and do whatever we want with its value.

There's a number of caveats here though:

• you can't expect things to change half-way during a processing round. With that I mean that if you sent a message while a buffer "slice" is being processed, it's likely the process function needs to finish before the message is received and acknowledged. In other words, JavaScript might seem to be an "asynchronous" language but it's still "one thing after the other" at the end of the day. The length of the buffer will influence how much time you need to wait until the message is processed. This is Audio Synthesis 101; I don't make the rules.
• and the usual caveats when using ports and message passing
  1. as advised by the spec: you should call close() on either side of the port when you're done, so that the resources can be collected
  2. the messages you send are transferred, not copied, meaning that they are no longer usable on the sending side.
     • If you imagined, as I suggested, message passing as sending objects down a pipe, you'll understand why the sender cannot access the thing it sent once it's sent.
     • This could bite you in certain cases, although it's fine here since we generate the message data when we send it; it'd be different if we sent complicated objects which we intend to keep using elsewhere in the program, as we'd need to clone them before posting them.
• if you wanted to send very finely grained values (to sample level grain size), you'll probably be better off using AudioParams.
• you still need to "parse" the messages you receive and decide what happens when you receive them; Web Audio won't do this for you.

Classic sound synthesis

The examples I've shown so far only generate noise with functions that do not depend on any previous run of the process function.

But if you try to generate a function with value continuity, such as a sine wave, you need to know where you left before generating the next slice of audio. Otherwise it is going to sound really choppy and wrong (unless you intend to generate noisy things, that is).

Let me be very clear: you don't need to generate sine waves with Web Audio, because the OscillatorNode already provides you with an implementation of sine waves. But if you wanted to, this is how you would do it.

Where are we?

The first thing we'd need to achieve continuity is to know how many samples we've already processed.

I thought I would need to do as I used to do in the past and keep track of this myself, but instances of AudioWorkletProcessor have access to a currentFrame property in their global scope that is updated automatically, so we can use it in our code.

There is also another useful property called currentTime which also gets updated automatically and will only ever increase, and could be used in a similar way if you preferred.

To access either, we refer to them "neat", i.e. without using this, as they don't belong to the instance, but the scope where the instance runs. They're global.

Computing an audio block

To generate our sine wave we need to compute the value of each of the samples in the output buffer, much like we did in the noise generator above.

But we also want to be able to configure the frequency of the generated wave.

With this I mean that we would like to be able to control how many complete cycles of the Math.sin function we want to hear per second (the unit for this is a Hertz or Hz).

For example, if the frequency was 1 Hz, that would mean that it would take a whole second⁶ to vibrate from 0 to 1 down to -1 and back to 0.

Assume we pass this frequency to our generator through the port, the same way that we passed the debug value before. Then the formula for filling in the buffer would be like this:

const cst = 2.0 * Math.PI * this._freq * this._inverseSampleRate;
let pos = currentFrame;

for (let i = 0; i < numSamples; i++) {
    let v = Math.sin(cst * pos);

    output.forEach((channel) => {
        channel[i] = v;
    });

    pos++;
}

Note how it depends on another variable I haven't mentioned yet, this._inverseSampleRate. To compute this, we need to access the sampleRate of the audio context the processor is generating audio for, and it's important to get this right, as otherwise you create sound that is not in the right pitch.

Since once the context is created the sample rate is fixed and won't change, I define this in the constructor for the processor, and then we don't need to compute this each time process is called.

So with this and the parsing of frequency, this is how the constructor looks like:

constructor(...args) {
    super(...args);

    this.port.onmessage = (e) => {
        let { frequency } = e.data;

        // TODO it goes without saying, but we're not checking this is a number, which it should be 8-)
        if (frequency !== undefined) {
            this._freq = frequency;
        }
    };

    this._inverseSampleRate = 1.0 / sampleRate;
}

By checking for the presence of undefined we can send single values from the main thread, for example:

sineNode.port.postMessage({ frequency: v });

A complete sine processor

And this is how the final sine generator worklet ends up looking⁷:

class SineGenerator extends AudioWorkletProcessor {
    _freq = 220.0;

    constructor(...args) {
        super(...args);

        this.port.onmessage = (e) => {
            let { frequency } = e.data;

            // TODO it goes without saying, but we're not checking this is a number, which it should be 8-)
            if (frequency !== undefined) {
                this._freq = frequency;
            }
        };

        this._inverseSampleRate = 1.0 / sampleRate;
    }

    process(inputs, outputs, parameters) {
        const output = outputs[0];
        const numSamples = output[0].length;

        const cst = 2.0 * Math.PI * this._freq * this._inverseSampleRate;
        let pos = currentFrame;

        for (let i = 0; i < numSamples; i++) {
            let v = Math.sin(cst * pos);

            output.forEach((channel) => {
                channel[i] = v;
            });

            pos++;
        }

        return true;
    }
}

registerProcessor("sine-generator", SineGenerator);

In the sine demo I built, we can change the frequency of the sine generator with an input element. When a change event is dispatched, we send a message to the generator with the new frequency.

You might notice the ocassional audio glitch when you change the frequency. That happens when there are discontinuities in the signal, or they don't start at exactly 0.

The native OscillatorNode implementation addresses the discontinuities that changing the frequencies creates by using some clever math, and as for the signals not starting with 0 causing a "CHK!", I talked about that elsewhere already, if you are curious as to how it could be fixed.

That said, setting the frequency with the port method is not ideal, because one of the best aspects of web audio is that you can almost connect the output of everything to the input of almost anything, taking advantage of the automation feature. In other words, if you use proper AudioParams, you can get really interesting effects with very little effort from your side.

But I had to show you how to send data to the processor with something :-)

I might review this post to add an example of how to use proper AudioParams, but in the meantime you now know how to implement a processor that generates a periodic signal, or just a processor that needs to get some sort of simple parameter sent to it in order to determine its output.

You can also check out the examples in this demo repository: https://github.com/sole/audioworklet-examples

〜〜〜〜 Happy audioworkletting! 〜〜〜〜